If you are building an LLM agent or a RAG-based search engine, the biggest hurdle isn't the retrieval—it's the observability. When users provide their own keys, you need a way to intercept those requests to monitor latency, token usage, and response quality without compromising the security of the key itself. This MIT-licensed library handles that plumbing, allowing you to implement a professional AI workflow where the infrastructure is decoupled from the API costs.
Getting started with the implementation #
To integrate this into a production environment, you generally need to wrap your LLM calls in a tracking layer. Instead of calling the OpenAI or Anthropic SDK directly, you route the request through the library's handler.
-
Installation and Setup: Ensure your environment is configured to handle environment variables for your tracking database.
-
Key Injection: Create a middleware that captures the user's API key from the request header and passes it to the library.
-
Request Wrapping: Wrap your search logic. For example, if you're using a Python-based backend:
from byok_tracker import SearchTracker
tracker = SearchTracker(project_id="ai-search-01")
def perform_ai_search(user_key, query):
with tracker.track(api_key=user_key):
response = call_llm_api(user_key, query)
return response
- Data Analysis: The library logs the metadata (tokens, time-to-first-token, and model version) into your specified storage, giving you a real-world look at how your search prompts are performing across different user keys.
Why this beats standard logging #
Using a specialized library for BYOK search tracking offers a few technical advantages over a generic logger.info()
approach:
Token Accuracy: It calculates exact token counts based on the specific model's tokenizer rather than estimating based on character count.Latency Breakdown: It separates network overhead from model inference time, which is critical for debugging slow AI search results.Cost Attribution: Since it's tied to the BYOK model, you can generate reports on which specific users are hitting the most expensive models.
For anyone doing a deep dive into prompt engineering for search, this is a practical tutorial in observability. You stop guessing if a prompt change improved the search quality and start seeing the actual delta in response times and token consumption across your entire user base.
TypeScript needs goroutines to actually compete with Go 3d ago
mcp-use v2: A Deep Dive into Stateless MCP Servers 6d ago
Next Anthropic aiming for a 2 trillion dollar IPO by October is →