Instrumenting an AI-Powered GitHub Analyzer with OpenTelemetry and SigNoz

A developer built GitIntel, an AI-powered GitHub repository analyzer using Gemini 2.5 Flash, and instrumented it with OpenTelemetry and SigNoz to gain end-to-end observability. The tool analyzes code across eight engineering dimensions, and the instrumentation revealed insights into latency, token usage, and execution flow that application logs alone could not provide.

This article is my submission for the Agents of SigNoz Hackathon https://www.wemakedevs.org/hackathons/signoz : Blog Track, where participants instrument real applications with OpenTelemetry and SigNoz to make AI systems observable. GitIntel is an AI-powered GitHub repository analyzer that uses Gemini 2.5 Flash to evaluate repositories and generate a comprehensive developer assessment. Given a GitHub username and selected repositories atmost 5 , it analyzes code across eight engineering dimensions: architecture, security, testing, documentation, complexity, and engineering practices to build both repository-level insights and an overall developer profile. The idea was straightforward: Combine GitHub data with an LLM to understand not just what the code does, but what it says about the developer behind it. As GitIntel evolved, every analysis became a multi-step workflow involving GitHub API requests, asynchronous file fetching, prompt construction, multiple Gemini API calls, response aggregation, and report generation. From the user's perspective, it was just one Analyze button. From the application's perspective, it was a distributed pipeline. That mismatch quickly became a problem. Some repository analyses consumed nearly 40,000 Gemini tokens, while others finished with fewer than 4,000. End-to-end latency ranged from 8 secs to 52 secs, even for repositories that appeared similar. My application logs already gave me useful information. I could see total token usage, overall analysis duration, GitHub rate-limit status, and when requests succeeded or failed. But they couldn't answer many other questions like: I had the outcomes. I didn't have the execution story. Terminal logs told me what happened. They couldn't show me how the entire pipeline behaved or where time was actually being spent. Every optimization still involved a degree of guesswork. That's when I decided to instrument GitIntel with OpenTelemetry and use SigNoz as the observability backend. To make GitIntel observable, I instrumented the application with OpenTelemetry and connected it to a self-hosted SigNoz instance. Instead of treating the application as a black box, I could now trace every request end-to-end from GitHub API calls to Gemini interactions, while correlating traces, metrics, and logs in a single place. The result was complete visibility into latency, token usage, failures, retries, and the overall execution flow. In this article, I'll share the setup, the instrumentation process, the challenges I encountered, the insights observability uncovered, and how you can reproduce the same environment on Ubuntu. Live Demo: GitIntel https://gitintel-2kh2.onrender.com Source Code: AI-powered GitHub profile analyzer. Enter a username, select up to 5 repos, and get a deep code assessment powered by Gemini 2.5 Flash — with full observability via OpenTelemetry and SigNoz. Live demo: gitintel-2kh2.onrender.com https://gitintel-2kh2.onrender.com Built for the Agents of SigNoz Hackathon — blog track. gemini.tokens.total , gemini.tokens.prompt , gemini.tokens.completion …| Component | Stack | |---|---| Backend | Python 3.12, FastAPI, Uvicorn | AI | Gemini 2.5 Flash google-genai | Data Fetching | GitHub REST API, aiohttp 10 concurrent workers | Observability | OpenTelemetry SDK + Self-hosted SigNoz | Frontend | HTML, CSS, Vanilla JavaScript | Platform | Ubuntu 24.04 LTS, Docker & Docker Compose | Before getting started, make sure you have the following ready: repo read:user gemini-2.5-flash Before continuing, verify that your environment is set up correctly: python3 --version should be 3.12.x docker --version 20+ is fine docker compose version If docker compose version fails, you're most likely using the legacy docker-compose v1 . Install Docker Compose v2 with: sudo apt-get install docker-compose-plugin 1 Start by forking the repository on GitHub, then clone your fork locally: git clone https://github.com/YOUR USERNAME/Github-Analyzer cd Github-Analyzer 2 Create a virtual environment and install the project dependencies: python3 -m venv venv source venv/bin/activate pip install -r requirements.txt 3 Next, create your environment file: cp .env.example .env nano .env 4 Update it with your GitHub token, Gemini API key, and OpenTelemetry configuration: GITHUB TOKEN=ghp yourtokenhere GEMINI API KEY=AIzaSy yourkeyhere OTEL EXPORTER OTLP ENDPOINT=http://localhost:4317 OTEL SERVICE NAME=gitintel OTEL EXPORTER OTLP ENDPOINT points to the SigNoz OpenTelemetry Collector, which we'll set up in Part 2. Don't worry if it isn't running yet—the application works perfectly fine without it. 5 Now start the application: uvicorn main:app --reload 6 Open http://localhost:8000 , enter any GitHub username, select a few repositories, and click You'll get a complete developer assessment: repository scores, developer profile, strengths, weaknesses, and the generated report. At this point, the application is fully functional. But there's one problem:- You still have no idea what happened behind the scenes. The terminal logs show you total token counts and overall duration, but not which batch cost what, not per-step timing, not which API call took 30 of those 40 secs, and not why one repo took 5x longer than another. The results are there. The why isn't. The application gives you the results, but none of the answers. That's exactly what Part 2 solves. One of the reasons I chose SigNoz is that it's Open Source, OpenTelemetry-native, and can be self-hosted in just a few mins. That meant I could keep my entire observability stack running locally while developing GitIntel, without relying on any third-party SaaS. The repo already includes a complete Docker Compose configuration under pours/deployment/compose.yaml . Spinning it up launches everything GitIntel needs to collect, process, store, and visualize telemetry: 4317 gRPC and 4318 HTTP , receiving telemetry exported from GitIntel.Start the entire stack with: cd pours/deployment docker compose up -d Verify everything is healthy docker compose ps The first startup usually takes around 60 secs, as ClickHouse needs a little time to initialize. Once everything is ready, open http://localhost:8080 . At this point, the SigNoz dashboard will load, but it will be empty. That's expected because GitIntel hasn't exported any telemetry yet. Now restart GitIntel so it can connect to the OpenTelemetry Collector: cd ../.. source venv/bin/activate uvicorn main:app --reload With both GitIntel and SigNoz running, analyze any GitHub repository. As soon as the analysis begins, GitIntel starts exporting telemetry through the OpenTelemetry Collector. Within a few seconds, a new service named gitintel automatically appears under Services in SigNoz. No manual registration, configuration, or dashboard creation is required, the service is discovered automatically from the OpenTelemetry service.name resource attribute configured in the application. Open Services → gitintel , and you'll immediately see live application health, including: With SigNoz up & running, the next step is teaching GitIntel what telemetry to collect and where to send it. Instead of scattering observability code throughout the project, I centralized everything in a single file: telemetry.py . One setup function initializes traces & metrics then exports them to the OpenTelemetry Collector using OTLP/gRPC on port 4317. telemetry.py OTLP ENDPOINT = os.getenv "OTEL EXPORTER OTLP ENDPOINT", "http://localhost:4317" SERVICE NAME = os.getenv "OTEL SERVICE NAME", "gitintel" resource = Resource.create {"service.name": SERVICE NAME} def setup telemetry app : otlp available = OTLP ENDPOINT = "http://localhost:4317" or os.getenv "OTEL ENABLED" tracer provider = TracerProvider resource=resource if otlp available: tracer provider.add span processor BatchSpanProcessor OTLPSpanExporter endpoint=OTLP ENDPOINT, insecure=True trace.set tracer provider tracer provider A few lines here do most of the heavy lifting. The Resource defines the identity of the application by assigning it the service name gitintel . Every trace, metric, & log generated by the application carries this metadata, allowing SigNoz to group everything under a single service automatically. The BatchSpanProcessor batches spans before exporting them to the OpenTelemetry Collector. Instead of sending every span individually, telemetry is exported efficiently in batches, reducing overhead during repository analysis. Perhaps my favorite part is how little manual instrumentation was actually required. FastAPIInstrumentor.instrument app app RequestsInstrumentor .instrument These two lines unlock a surprising amount of observability. One small detail ended up being far more important than I expected: otlp available = OTLP ENDPOINT = "http://localhost:4317" or os.getenv "OTEL ENABLED" During local development, GitIntel exports telemetry to the self-hosted SigNoz instance as expected. When deployed to Render, however, there's no OpenTelemetry Collector running. Without this guard, the application would repeatedly try to export telemetry and flood the logs with connection refused errors every few seconds. I only discovered that after deploying the project, so this check became a simple but an important one. Once GitIntel starts with SigNoz running, open http://localhost:8080 http://localhost:8080 → Services. Because the service name is already defined in the OpenTelemetry Resource , SigNoz automatically discovers gitintel and begins collecting telemetry immediately. Without writing a single dashboard query, you can already monitor: At this point, every request through GitIntel is visible, timed, and traceable. Auto-instrumentation gives you excellent visibility into HTTP requests, but GitIntel isn't just another web application, it's a LLM-powered analysis pipeline. Knowing that an endpoint took 12 secs is useful, but it doesn't answer the questions I actually cared about. To answer those questions, I created eight custom OpenTelemetry instruments in telemetry.py : telemetry.py github ratelimit gauge = meter.create gauge "github.ratelimit.remaining" gemini tokens counter = meter.create counter "gemini.tokens.total", unit="tokens" gemini prompt tokens counter = meter.create counter "gemini.tokens.prompt", unit="tokens" gemini completion tokens counter = meter.create counter "gemini.tokens.completion", unit="tokens" files processed counter = meter.create counter "github.files.processed" loc processed counter = meter.create counter "github.loc.processed", unit="lines" assessment duration histogram = meter.create histogram "assessment.duration", unit="s" api error counter = meter.create counter "api.errors" Each instrument captures a different aspect of GitIntel's execution, from LLM token usage and GitHub rate limits to assessment latency and processing volume. Together, they provide a much more complete picture than HTTP metrics alone. One implementation detail turned out to be particularly valuable. Every token counter includes the repository name as an attribute. python gemini client.py — track usage def track usage usage, repo: str : pt = getattr usage, "prompt token count", 0 or 0 ct = getattr usage, "candidates token count", 0 or 0 gemini prompt tokens counter.add pt, {"repo": repo} gemini completion tokens counter.add ct, {"repo": repo} gemini tokens counter.add pt + ct, {"repo": repo} That small {"repo": repo} attribute completely changes how useful the data becomes. Without it, I'd only know GitIntel consumed 50,000 tokens. With it, I can immediately identify which repository consumed 38,000 of those tokens, compare repositories side by side, and focus optimization efforts where they'll have the biggest impact. In observability, context is just as important as the metric itself. GitHub's remaining API quota is tracked in a similar way. github client.py — remaining = resp.headers.get "X-RateLimit-Remaining" if remaining is not None: github ratelimit gauge.set int remaining GitHub conveniently includes the remaining rate limit in every API response header. Updating the gauge here means it's always current without making any additional API requests. Once telemetry starts flowing, open Metrics → Query Builder in SigNoz. Here are the visualizations I found most useful: gemini.tokens.total repo assessment.duration github.ratelimit.remaining These dashboards quickly answered questions that were impossible to answer before instrumentation: 📸 SS 4:Metrics Query Builder showing gemini.tokens.total grouped by the repo attribute.📸 SS 5:Time-series chart for github.ratelimit.remaining decreasing during consecutive analyses.📸 SS 6:Histogram or time-series for assessment.duration , highlighting p50 and p95 latency. Finally, I combined these visualizations into a custom dashboard named GitIntel LLM Observability . Instead of jumping between multiple pages, I now have a single view showing token usage, GitHub rate-limit headroom, request latency, and overall application health every time I open SigNoz. Auto-instrumentation is fantastic for HTTP requests, but it doesn't know anything about your application's business logic. It can't tell you: To answer those questions, I added manual spans around the critical parts of GitIntel's analysis pipeline. For every Gemini request, gemini client.py creates a span that records both the repository being analyzed and the size of the prompt sent to the model. php gemini client.py — call gemini def call gemini prompt: str, repo: str - str: with tracer.start as current span "gemini.generate", attributes={ "gemini.repo": repo, "gemini.prompt chars": len prompt , }, : for attempt in range 5 : try: response = client.models.generate content model=MODEL, contents=prompt, track usage response.usage metadata, repo return response.text except Exception as e: api error counter.add 1, {"service": "gemini", "repo": repo} if attempt == 4: raise time.sleep 10 attempt + 1 The prompt chars attribute became surprisingly useful. While it isn't the exact token count that's tracked separately as a metric , it's an excellent indicator of prompt size and makes it much easier to correlate larger prompts with longer response times. I applied the same idea to GitHub. Rather than simply tracing the HTTP requests, GitIntel records repository-level metadata during file collection. github client.py — get all files with tracer.start as current span "github.get all files", attributes={ "github.repo": f"{owner}/{repo}", "github.branch": branch, }, : span = trace.get current span ... fetch tree ... span.set attribute "github.file count.total", len tree span.set attribute "github.file count.source", len source files By attaching these attributes directly to the span, every trace now includes useful context such as: That context makes the trace far more useful than a simple timeline of function calls. When everything runs together, one repository analysis produces a distributed trace that looks like this: api.analyze ├── github.get user github.username ├── github.get repos ├── github.get all files file count.total, file count.source ├── gemini.assess repo github.repo, file count │ ├── gemini.generate prompt chars — batch 1 │ └── gemini.generate prompt chars — batch 2, large repos only └── gemini.developer profile └── gemini.generate Instead of seeing a single request that took 42 seconds , I can now see exactly where those 42 seconds were spent . Open Traces → Filter by api.analyze and select any request. The trace expands into the full execution tree, showing every nested span, its duration, and its metadata. Click on any gemini.generate span, and the details panel immediately shows attributes such as: gemini.repo gemini.prompt chars This was the moment observability finally clicked for me. The problem was never that GitIntel was "slow." The problem was that I had no visibility into why it was slow. With distributed tracing, that mystery disappeared. At this point, GitIntel was fully instrumented. The real question wasn't whether telemetry was flowing—it was whether it could answer the questions I actually cared about. To make the differences obvious, I ran analyses on two repositories with very different characteristics: Seeing these runs side by side is where observability becomes valuable. The contrast makes latency spikes, token usage, and processing costs immediately visible instead of hidden behind a single loading spinner. The first thing that stood out was token consumption. Some repositories finished after a single Gemini request, while larger ones required multiple batches, causing token usage to increase dramatically. Instead of guessing why one analysis was expensive, I could see exactly which repository consumed the most tokens and correlate that with the trace. Before instrumentation, all I knew was that some analyses took 10 secs while others took nearly a minute. Distributed traces showed exactly where that time was spent. GitHub requests completed relatively quickly, while one or more gemini.generate spans dominated the overall request duration. Rather than treating the request as a single slow operation, I could pinpoint the specific step responsible for the delay. One assumption I had before adding telemetry was that downloading hundreds of files from GitHub would dominate the execution time. The traces showed otherwise. Repository fetching remained relatively fast, even for larger projects. Most of the end-to-end latency came from prompt generation and LLM inference. That insight completely changed where I would spend time optimizing GitIntel. Every GitHub API response updates the github.ratelimit.remaining gauge. Instead of wondering how close I was to GitHub's limit, I could watch it decrease in real time during consecutive analyses. This makes it much easier to decide when to slow requests, introduce caching, or back off before hitting the hourly quota. After creating a custom dashboard, I no longer needed to jump between traces, metrics, and logs. One screen showed: Instead of debugging reactively, I could monitor GitIntel proactively. Three things surprised me after instrumenting GitIntel: Those are the kinds of insights I simply couldn't have gained without end-to-end observability. Before adding observability, every failed analysis looked identical. Something was slow. Something failed. Something used too many tokens. I had no way of knowing where, why, or how expensive it was. After instrumenting GitIntel with OpenTelemetry and SigNoz, every request became explainable. The prompt chars span attribute immediately shows which Gemini batch caused a latency spike. The repo attribute on my custom metrics identifies exactly which repository consumed the most tokens. And because every log is correlated with its trace, I can see the complete execution path that led to a failure instead of staring at an isolated error message. Observability didn't just make GitIntel easier to debug. It made the application understandable. That's the difference between using AI APIs and owning AI infrastructure. This wouldn't be a completely honest engineering blog if everything worked perfectly the first time. It didn't. These were the two issues that cost me the most time, and the fixes that finally made the setup reliable. After deploying GitIntel to Render, my application logs were immediately flooded with this message every few seconds: WARNING opentelemetry.exporter.otlp.proto.grpc.exporter Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 0.82s. Failed to connect to remote host: Connection refused The reason was simple. Render wasn't running SigNoz, so nothing was listening on localhost:4317 . OpenTelemetry kept retrying indefinitely, filling the logs with connection errors and making it difficult to spot genuine application issues. The fix was a small guard inside telemetry.py : otlp available = OTLP ENDPOINT = "http://localhost:4317" or os.getenv "OTEL ENABLED" If the exporter endpoint is still the default localhost address and OTEL ENABLED hasn't been explicitly enabled, GitIntel simply skips creating the OTLP exporters altogether. That means: One condition removed hundreds of unnecessary log messages. The second issue was much more confusing. I started the entire SigNoz stack successfully. The application launched. Requests completed normally. But the Services page remained completely empty. The problem wasn't OpenTelemetry at all. It was startup timing. Although the containers report as running almost immediately after docker compose up -d , ClickHouse still needs roughly a minute to finish initializing before it can persist telemetry. During that window, the OpenTelemetry Collector accepts incoming spans, but the backend isn't fully ready to store them. The solution was simply to wait until every container reported a healthy state before launching GitIntel. cd pours/deployment docker compose up -d Wait until all services are healthy — takes ~60 seconds docker compose ps Look for healthy on: signoz-signoz-0 signoz-telemetrystore-clickhouse-0-0 signoz-telemetrykeeper-clickhousekeeper-0 signoz-metastore-postgres-0 cd ../.. uvicorn main:app --reload After starting the application, I ran a single repository analysis and waited another 15–20 secs. That's expected. SigNoz batches telemetry before writing it to storage. If the Services page is still empty after about half a minute, the fastest way to diagnose the issue is to inspect the Collector logs: cd pours/deployment docker compose logs ingester --tail=20 Once the first spans appear, everything behaves normally from that point onward. The delay only affects the initial startup while the telemetry backend finishes initializing. Neither of these bugs had anything to do with GitIntel itself. They were observability infrastructure problems. Ironically, building observability meant I first had to debug the observability stack. Once those issues were solved, though, every repository analysis became traceable, measurable, and far easier to reason about. And that's exactly the point of instrumenting an AI application—you spend less time guessing and more time understanding what your system is actually doing. OpenTelemetry's auto-instrumentation gave me immediate visibility into GitIntel. With just a few lines of code, FastAPIInstrumentor and RequestsInstrumentor automatically traced every FastAPI request and every outbound GitHub API call. Latency, status codes, request rate; they were all available without touching my application logic. That was a great starting point, but it wasn't enough. Auto-instrumentation could tell me that a request took 52 secs. It couldn't tell me why. The real value came from the telemetry I added myself. Attributes like prompt chars , repo , and file counts transformed traces and metrics into something I could actually reason about. Instead of seeing a slow request, I could identify which Gemini batch was responsible. Instead of seeing total token usage, I could pinpoint exactly which repository consumed the most tokens. And because every log was correlated with its trace, a single click showed the complete execution path leading to a failure. The biggest lesson I took away is this: Auto-instrumentation gives you visibility. Custom instrumentation gives you understanding. When I started GitIntel, I almost skipped observability altogether. I wanted to finish the project first and "add monitoring later." I'm glad I didn't. The unexpected 40,000-token analyses, the OpenTelemetry exporter spam on Render, the ClickHouse cold-start delay, and the huge latency differences between repositories would all have looked like random bugs without traces and metrics. Instead of guessing, I always had evidence pointing me in the right direction. Observability also changed how I think about metrics. A global token count wasn't nearly as useful as attaching a repo attribute to every measurement. A retry log became far more valuable once it was linked to the exact distributed trace that produced it. Every small piece of context made debugging dramatically easier. The biggest shift wasn't technical, it was mental. I stopped asking, "Why is my app behaving like this?" and started asking, "What does the telemetry tell me?" That's a much better question. Building GitIntel taught me that integrating an LLM is only half the problem. Understanding what the application is doing in production is the other half. With OpenTelemetry and SigNoz, every repository analysis became observable from end to end. I could follow GitHub API requests, inspect Gemini calls, measure token usage, monitor latency, correlate logs with traces, and identify failures—all from a single platform. That's what observability should do: replace assumptions with evidence. If you're building applications powered by LLMs, agents, or external APIs, don't wait until production to add telemetry. Instrument them from day one. The first time something behaves unexpectedly, you'll be glad you did. Because in the end, you can't optimize what you don't measure, you can't debug what you can't see, and you don't truly own an AI system until you can observe it.