# Instrumenting an AI-Powered GitHub Analyzer with OpenTelemetry and SigNoz > Source: > Published: 2026-07-17 12:07:31+00:00 This article is my submission for the [Agents of SigNoz Hackathon](https://www.wemakedevs.org/hackathons/signoz): Blog Track, where participants instrument real applications with OpenTelemetry and SigNoz to make AI systems observable. GitIntel is an AI-powered GitHub repository analyzer that uses `Gemini 2.5 Flash` to evaluate repositories and generate a comprehensive developer assessment. Given a GitHub username and selected repositories (atmost 5), it analyzes code across eight engineering dimensions: architecture, security, testing, documentation, complexity, and engineering practices to build both repository-level insights and an overall developer profile. The idea was straightforward: Combine GitHub data with an LLM to understand not just what the code does, but what it says about the developer behind it. As GitIntel evolved, every analysis became a multi-step workflow involving GitHub API requests, asynchronous file fetching, prompt construction, multiple Gemini API calls, response aggregation, and report generation. From the user's perspective, it was just one **Analyze** button. From the application's perspective, it was a distributed pipeline. That mismatch quickly became a problem. Some repository analyses consumed nearly 40,000 Gemini tokens, while others finished with fewer than 4,000. End-to-end latency ranged from 8 secs to 52 secs, even for repositories that appeared similar. My application logs already gave me useful information. I could see total token usage, overall analysis duration, GitHub rate-limit status, and when requests succeeded or failed. But they couldn't answer many other questions like: I had the outcomes. I didn't have the execution story. Terminal logs told me what happened. They couldn't show me how the entire pipeline behaved or where time was actually being spent. Every optimization still involved a degree of guesswork. That's when I decided to instrument GitIntel with **OpenTelemetry** and use **SigNoz** as the observability backend. To make GitIntel observable, I instrumented the application with OpenTelemetry and connected it to a self-hosted SigNoz instance. Instead of treating the application as a black box, I could now trace every request end-to-end from GitHub API calls to Gemini interactions, while correlating traces, metrics, and logs in a single place. The result was complete visibility into latency, token usage, failures, retries, and the overall execution flow. In this article, I'll share the setup, the instrumentation process, the challenges I encountered, the insights observability uncovered, and how you can reproduce the same environment on Ubuntu. **Live Demo:** [GitIntel](https://gitintel-2kh2.onrender.com) **Source Code:** AI-powered GitHub profile analyzer. Enter a username, select up to 5 repos, and get a deep code assessment powered by Gemini 2.5 Flash — with full observability via OpenTelemetry and SigNoz. **Live demo:** [gitintel-2kh2.onrender.com](https://gitintel-2kh2.onrender.com) Built for the [Agents of SigNoz Hackathon]— blog track. `gemini.tokens.total` , `gemini.tokens.prompt` , `gemini.tokens.completion` …| Component | Stack | |---|---| Backend | Python 3.12, FastAPI, Uvicorn | AI | Gemini 2.5 Flash (`google-genai` ) | Data Fetching | GitHub REST API, `aiohttp` (10 concurrent workers) | Observability | OpenTelemetry SDK + Self-hosted SigNoz | Frontend | HTML, CSS, Vanilla JavaScript | Platform | Ubuntu 24.04 LTS, Docker & Docker Compose | Before getting started, make sure you have the following ready: `repo` `read:user` `gemini-2.5-flash` Before continuing, verify that your environment is set up correctly: ``` python3 --version # should be 3.12.x docker --version # 20+ is fine docker compose version ``` If `docker compose version` fails, you're most likely using the legacy **docker-compose (v1)**. Install Docker Compose v2 with: ``` sudo apt-get install docker-compose-plugin ``` 1) Start by forking the repository on GitHub, then clone your fork locally: ``` git clone https://github.com/YOUR_USERNAME/Github-Analyzer cd Github-Analyzer ``` 2) Create a virtual environment and install the project dependencies: ``` python3 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` 3) Next, create your environment file: ``` cp .env.example .env nano .env ``` 4) Update it with your GitHub token, Gemini API key, and OpenTelemetry configuration: ``` GITHUB_TOKEN=ghp_yourtokenhere GEMINI_API_KEY=AIzaSy_yourkeyhere OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 OTEL_SERVICE_NAME=gitintel ``` `OTEL_EXPORTER_OTLP_ENDPOINT` points to the SigNoz OpenTelemetry Collector, which we'll set up in Part 2. Don't worry if it isn't running yet—the application works perfectly fine without it. 5) Now start the application: ``` uvicorn main:app --reload ``` 6) Open ** http://localhost:8000**, enter any GitHub username, select a few repositories, and click You'll get a complete developer assessment: repository scores, developer profile, strengths, weaknesses, and the generated report. At this point, the application is fully functional. But there's one problem:- You still have no idea what happened behind the scenes. The terminal logs show you total token counts and overall duration, but not which batch cost what, not per-step timing, not which API call took 30 of those 40 secs, and not why one repo took 5x longer than another. The results are there. The why isn't. The application gives you the results, but none of the answers. That's exactly what **Part 2** solves. One of the reasons I chose SigNoz is that it's Open Source, OpenTelemetry-native, and can be self-hosted in just a few mins. That meant I could keep my entire observability stack running locally while developing GitIntel, without relying on any third-party SaaS. The repo already includes a complete Docker Compose configuration under `pours/deployment/compose.yaml` . Spinning it up launches everything GitIntel needs to collect, process, store, and visualize telemetry: `4317 (gRPC)` and `4318 (HTTP)` , receiving telemetry exported from GitIntel.Start the entire stack with: ``` cd pours/deployment docker compose up -d # Verify everything is healthy docker compose ps ``` The first startup usually takes around 60 secs, as ClickHouse needs a little time to initialize. Once everything is ready, open ** http://localhost:8080**. At this point, the SigNoz dashboard will load, but it will be empty. That's expected because GitIntel hasn't exported any telemetry yet. Now restart GitIntel so it can connect to the OpenTelemetry Collector: ``` cd ../.. source venv/bin/activate uvicorn main:app --reload ``` With both GitIntel and SigNoz running, analyze any GitHub repository. As soon as the analysis begins, GitIntel starts exporting telemetry through the OpenTelemetry Collector. Within a few seconds, a new service named `gitintel` automatically appears under **Services** in SigNoz. No manual registration, configuration, or dashboard creation is required, the service is discovered automatically from the OpenTelemetry `service.name` resource attribute configured in the application. Open **Services → gitintel**, and you'll immediately see live application health, including: With SigNoz up & running, the next step is teaching GitIntel what telemetry to collect and where to send it. Instead of scattering observability code throughout the project, I centralized everything in a single file: ** telemetry.py**. One setup function initializes traces & metrics then exports them to the OpenTelemetry Collector using OTLP/gRPC on port 4317. ``` # telemetry.py OTLP_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317") SERVICE_NAME = os.getenv("OTEL_SERVICE_NAME", "gitintel") resource = Resource.create({"service.name": SERVICE_NAME}) def setup_telemetry(app): otlp_available = OTLP_ENDPOINT != "http://localhost:4317" or os.getenv("OTEL_ENABLED") tracer_provider = TracerProvider(resource=resource) if otlp_available: tracer_provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter(endpoint=OTLP_ENDPOINT, insecure=True) ) ) trace.set_tracer_provider(tracer_provider) ``` A few lines here do most of the heavy lifting. The `Resource` defines the identity of the application by assigning it the service name `gitintel` . Every trace, metric, & log generated by the application carries this metadata, allowing SigNoz to group everything under a single service automatically. The `BatchSpanProcessor` batches spans before exporting them to the OpenTelemetry Collector. Instead of sending every span individually, telemetry is exported efficiently in batches, reducing overhead during repository analysis. Perhaps my favorite part is how little manual instrumentation was actually required. ``` FastAPIInstrumentor.instrument_app(app) RequestsInstrumentor().instrument() ``` These two lines unlock a surprising amount of observability. One small detail ended up being far more important than I expected: ``` otlp_available = OTLP_ENDPOINT != "http://localhost:4317" or os.getenv("OTEL_ENABLED") ``` During local development, GitIntel exports telemetry to the self-hosted SigNoz instance as expected. When deployed to Render, however, there's no OpenTelemetry Collector running. Without this guard, the application would repeatedly try to export telemetry and flood the logs with connection refused errors every few seconds. I only discovered that after deploying the project, so this check became a simple but an important one. Once GitIntel starts with SigNoz running, open [http://localhost:8080](http://localhost:8080) → Services. Because the service name is already defined in the OpenTelemetry `Resource` , SigNoz automatically discovers ** gitintel** and begins collecting telemetry immediately. Without writing a single dashboard query, you can already monitor: At this point, every request through GitIntel is visible, timed, and traceable. Auto-instrumentation gives you excellent visibility into HTTP requests, but GitIntel isn't just another web application, it's a LLM-powered analysis pipeline. Knowing that an endpoint took 12 secs is useful, but it doesn't answer the questions I actually cared about. To answer those questions, I created eight custom OpenTelemetry instruments in `telemetry.py` : ``` # telemetry.py github_ratelimit_gauge = meter.create_gauge("github.ratelimit.remaining") gemini_tokens_counter = meter.create_counter("gemini.tokens.total", unit="tokens") gemini_prompt_tokens_counter = meter.create_counter("gemini.tokens.prompt", unit="tokens") gemini_completion_tokens_counter = meter.create_counter("gemini.tokens.completion", unit="tokens") files_processed_counter = meter.create_counter("github.files.processed") loc_processed_counter = meter.create_counter("github.loc.processed", unit="lines") assessment_duration_histogram = meter.create_histogram("assessment.duration", unit="s") api_error_counter = meter.create_counter("api.errors") ``` Each instrument captures a different aspect of GitIntel's execution, from LLM token usage and GitHub rate limits to assessment latency and processing volume. Together, they provide a much more complete picture than HTTP metrics alone. One implementation detail turned out to be particularly valuable. Every token counter includes the repository name as an attribute. ``` python # gemini_client.py — _track_usage() def _track_usage(usage, repo: str): pt = getattr(usage, "prompt_token_count", 0) or 0 ct = getattr(usage, "candidates_token_count", 0) or 0 gemini_prompt_tokens_counter.add(pt, {"repo": repo}) gemini_completion_tokens_counter.add(ct, {"repo": repo}) gemini_tokens_counter.add(pt + ct, {"repo": repo}) ``` That small `{"repo": repo}` attribute completely changes how useful the data becomes. Without it, I'd only know GitIntel consumed 50,000 tokens. With it, I can immediately identify which repository consumed 38,000 of those tokens, compare repositories side by side, and focus optimization efforts where they'll have the biggest impact. In observability, context is just as important as the metric itself. GitHub's remaining API quota is tracked in a similar way. ``` # github_client.py — remaining = resp.headers.get("X-RateLimit-Remaining") if remaining is not None: github_ratelimit_gauge.set(int(remaining)) ``` GitHub conveniently includes the remaining rate limit in every API response header. Updating the gauge here means it's always current without making any additional API requests. Once telemetry starts flowing, open **Metrics → Query Builder** in SigNoz. Here are the visualizations I found most useful: `gemini.tokens.total` `repo` `assessment.duration` `github.ratelimit.remaining` These dashboards quickly answered questions that were impossible to answer before instrumentation: 📸 SS 4:Metrics Query Builder showing`gemini.tokens.total` grouped by the`repo` attribute.📸 SS 5:Time-series chart for`github.ratelimit.remaining` decreasing during consecutive analyses.📸 SS 6:Histogram (or time-series) for`assessment.duration` , highlighting p50 and p95 latency. Finally, I combined these visualizations into a custom dashboard named **GitIntel LLM Observability**. Instead of jumping between multiple pages, I now have a single view showing token usage, GitHub rate-limit headroom, request latency, and overall application health every time I open SigNoz. Auto-instrumentation is fantastic for HTTP requests, but it doesn't know anything about your application's business logic. It can't tell you: To answer those questions, I added **manual spans** around the critical parts of GitIntel's analysis pipeline. For every Gemini request, `gemini_client.py` creates a span that records both the repository being analyzed and the size of the prompt sent to the model. ``` php # gemini_client.py — _call_gemini() def _call_gemini(prompt: str, repo: str) -> str: with tracer.start_as_current_span( "gemini.generate", attributes={ "gemini.repo": repo, "gemini.prompt_chars": len(prompt), }, ): for attempt in range(5): try: response = client.models.generate_content( model=MODEL, contents=prompt, ) _track_usage(response.usage_metadata, repo) return response.text except Exception as e: api_error_counter.add(1, {"service": "gemini", "repo": repo}) if attempt == 4: raise time.sleep(10 * (attempt + 1)) ``` The `prompt_chars` attribute became surprisingly useful. While it isn't the exact token count (that's tracked separately as a metric), it's an excellent indicator of prompt size and makes it much easier to correlate larger prompts with longer response times. I applied the same idea to GitHub. Rather than simply tracing the HTTP requests, GitIntel records **repository-level metadata** during file collection. ``` # github_client.py — get_all_files() with tracer.start_as_current_span( "github.get_all_files", attributes={ "github.repo": f"{owner}/{repo}", "github.branch": branch, }, ): span = trace.get_current_span() # ... fetch tree ... span.set_attribute("github.file_count.total", len(tree)) span.set_attribute("github.file_count.source", len(source_files)) ``` By attaching these attributes directly to the span, every trace now includes useful context such as: That context makes the trace far more useful than a simple timeline of function calls. When everything runs together, one repository analysis produces a distributed trace that looks like this: ``` api.analyze ├── github.get_user (github.username) ├── github.get_repos ├── github.get_all_files (file_count.total, file_count.source) ├── gemini.assess_repo (github.repo, file_count) │ ├── gemini.generate (prompt_chars — batch 1) │ └── gemini.generate (prompt_chars — batch 2, large repos only) └── gemini.developer_profile └── gemini.generate ``` Instead of seeing a single request that took **42 seconds**, I can now see **exactly where those 42 seconds were spent**. Open **Traces → Filter by api.analyze** and select any request. The trace expands into the full execution tree, showing every nested span, its duration, and its metadata. Click on any ** gemini.generate** span, and the details panel immediately shows attributes such as: `gemini.repo` `gemini.prompt_chars` This was the moment observability finally clicked for me. The problem was never that GitIntel was "slow." The problem was that I had no visibility into *why* it was slow. With distributed tracing, that mystery disappeared. At this point, GitIntel was fully instrumented. The real question wasn't whether telemetry was flowing—it was whether it could answer the questions I actually cared about. To make the differences obvious, I ran analyses on two repositories with very different characteristics: Seeing these runs side by side is where observability becomes valuable. The contrast makes latency spikes, token usage, and processing costs immediately visible instead of hidden behind a single loading spinner. The first thing that stood out was token consumption. Some repositories finished after a single Gemini request, while larger ones required multiple batches, causing token usage to increase dramatically. Instead of guessing why one analysis was expensive, I could see exactly which repository consumed the most tokens and correlate that with the trace. Before instrumentation, all I knew was that some analyses took 10 secs while others took nearly a minute. Distributed traces showed exactly where that time was spent. GitHub requests completed relatively quickly, while one or more `gemini.generate` spans dominated the overall request duration. Rather than treating the request as a single slow operation, I could pinpoint the specific step responsible for the delay. One assumption I had before adding telemetry was that downloading hundreds of files from GitHub would dominate the execution time. The traces showed otherwise. Repository fetching remained relatively fast, even for larger projects. Most of the end-to-end latency came from prompt generation and LLM inference. That insight completely changed where I would spend time optimizing GitIntel. Every GitHub API response updates the `github.ratelimit.remaining` gauge. Instead of wondering how close I was to GitHub's limit, I could watch it decrease in real time during consecutive analyses. This makes it much easier to decide when to slow requests, introduce caching, or back off before hitting the hourly quota. After creating a custom dashboard, I no longer needed to jump between traces, metrics, and logs. One screen showed: Instead of debugging reactively, I could monitor GitIntel proactively. Three things surprised me after instrumenting GitIntel: Those are the kinds of insights I simply couldn't have gained without end-to-end observability. Before adding observability, every failed analysis looked identical. Something was slow. Something failed. Something used too many tokens. I had no way of knowing where, why, or how expensive it was. After instrumenting GitIntel with OpenTelemetry and SigNoz, every request became explainable. The `prompt_chars` span attribute immediately shows which Gemini batch caused a latency spike. The `repo` attribute on my custom metrics identifies exactly which repository consumed the most tokens. And because every log is correlated with its trace, I can see the complete execution path that led to a failure instead of staring at an isolated error message. Observability didn't just make GitIntel easier to debug. It made the application understandable. That's the difference between using AI APIs and owning AI infrastructure. This wouldn't be a completely honest engineering blog if everything worked perfectly the first time. It didn't. These were the two issues that cost me the most time, and the fixes that finally made the setup reliable. After deploying GitIntel to Render, my application logs were immediately flooded with this message every few seconds: ``` WARNING opentelemetry.exporter.otlp.proto.grpc.exporter Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 0.82s. Failed to connect to remote host: Connection refused ``` The reason was simple. Render wasn't running SigNoz, so nothing was listening on `localhost:4317` . OpenTelemetry kept retrying indefinitely, filling the logs with connection errors and making it difficult to spot genuine application issues. The fix was a small guard inside `telemetry.py` : ``` otlp_available = OTLP_ENDPOINT != "http://localhost:4317" or os.getenv("OTEL_ENABLED") ``` If the exporter endpoint is still the default localhost address and `OTEL_ENABLED` hasn't been explicitly enabled, GitIntel simply skips creating the OTLP exporters altogether. That means: One condition removed hundreds of unnecessary log messages. The second issue was much more confusing. I started the entire SigNoz stack successfully. The application launched. Requests completed normally. But the **Services** page remained completely empty. The problem wasn't OpenTelemetry at all. It was startup timing. Although the containers report as running almost immediately after `docker compose up -d` , ClickHouse still needs roughly a minute to finish initializing before it can persist telemetry. During that window, the OpenTelemetry Collector accepts incoming spans, but the backend isn't fully ready to store them. The solution was simply to wait until every container reported a healthy state before launching GitIntel. ``` cd pours/deployment docker compose up -d # Wait until all services are healthy — takes ~60 seconds docker compose ps # Look for (healthy) on: # signoz-signoz-0 # signoz-telemetrystore-clickhouse-0-0 # signoz-telemetrykeeper-clickhousekeeper-0 # signoz-metastore-postgres-0 cd ../.. uvicorn main:app --reload ``` After starting the application, I ran a single repository analysis and waited another 15–20 secs. That's expected. SigNoz batches telemetry before writing it to storage. If the Services page is still empty after about half a minute, the fastest way to diagnose the issue is to inspect the Collector logs: ``` cd pours/deployment docker compose logs ingester --tail=20 ``` Once the first spans appear, everything behaves normally from that point onward. The delay only affects the initial startup while the telemetry backend finishes initializing. Neither of these bugs had anything to do with GitIntel itself. They were observability infrastructure problems. Ironically, building observability meant I first had to debug the observability stack. Once those issues were solved, though, every repository analysis became traceable, measurable, and far easier to reason about. And that's exactly the point of instrumenting an AI application—you spend less time guessing and more time understanding what your system is actually doing. OpenTelemetry's auto-instrumentation gave me immediate visibility into GitIntel. With just a few lines of code, `FastAPIInstrumentor` and `RequestsInstrumentor` automatically traced every FastAPI request and every outbound GitHub API call. Latency, status codes, request rate; they were all available without touching my application logic. That was a great starting point, but it wasn't enough. Auto-instrumentation could tell me that a request took 52 secs. It couldn't tell me why. The real value came from the telemetry I added myself. Attributes like `prompt_chars` , `repo` , and file counts transformed traces and metrics into something I could actually reason about. Instead of seeing a slow request, I could identify which Gemini batch was responsible. Instead of seeing total token usage, I could pinpoint exactly which repository consumed the most tokens. And because every log was correlated with its trace, a single click showed the complete execution path leading to a failure. The biggest lesson I took away is this: Auto-instrumentation gives you visibility. Custom instrumentation gives you understanding. When I started GitIntel, I almost skipped observability altogether. I wanted to finish the project first and "add monitoring later." I'm glad I didn't. The unexpected 40,000-token analyses, the OpenTelemetry exporter spam on Render, the ClickHouse cold-start delay, and the huge latency differences between repositories would all have looked like random bugs without traces and metrics. Instead of guessing, I always had evidence pointing me in the right direction. Observability also changed how I think about metrics. A global token count wasn't nearly as useful as attaching a `repo` attribute to every measurement. A retry log became far more valuable once it was linked to the exact distributed trace that produced it. Every small piece of context made debugging dramatically easier. The biggest shift wasn't technical, it was mental. I stopped asking, "Why is my app behaving like this?" and started asking, "What does the telemetry tell me?" That's a much better question. Building GitIntel taught me that integrating an LLM is only half the problem. Understanding what the application is doing in production is the other half. With OpenTelemetry and SigNoz, every repository analysis became observable from end to end. I could follow GitHub API requests, inspect Gemini calls, measure token usage, monitor latency, correlate logs with traces, and identify failures—all from a single platform. That's what observability should do: replace assumptions with evidence. If you're building applications powered by LLMs, agents, or external APIs, don't wait until production to add telemetry. Instrument them from day one. The first time something behaves unexpectedly, you'll be glad you did. Because in the end, **you can't optimize what you don't measure, you can't debug what you can't see, and you don't truly own an AI system until you can observe it.**