ContextOps, an ESLint-like static analyzer for LLM context

ContextOps, a new open-source static analysis tool for LLM context, has been released. It acts as a deterministic linter that evaluates structural quality—such as redundancy, token waste, and source concentration—before inference, producing a Context Health Score without requiring embeddings or external API calls. The tool aims to make LLM context quality observable and testable, similar to how ESLint enforces code quality in software engineering.

Static analysis for LLM context. ContextOps is a deterministic, model-independent context linter for LLM applications. It analyzes the context sent to an LLM before inference and detects structural problems such as redundancy, token waste, context imbalance, and source concentration. It produces a reproducible Context Health Score CHS together with actionable diagnostics — without embeddings, model calls, or external services. Think of ContextOps as ESLint for LLM context . Modern software engineering has deterministic quality gates. - Compilers catch syntax errors. - Linters catch code smells. - Formatters enforce consistency. - Static analyzers detect architectural problems. LLM applications rarely have an equivalent layer for the context they send into models. Instead, prompts quietly grow over time: - duplicated retrieval chunks - bloated system prompts - runaway conversation history - excessive tool output - hidden token waste These issues increase latency and cost, make model behavior less predictable, and often go unnoticed until production. ContextOps makes context quality observable, measurable, and testable before inference. ContextOps evaluates four structural dimensions. | Dimension | Max Penalty | What It Measures | |---|---|---| Redundancy | 30 pts | Lexical duplication across context items | Density | 30 pts | Token waste from formatting and structural bloat | Structure | 20 pts | Distribution imbalance between context components | Concentration | 20 pts | Over-reliance on a single document or source | The result is a deterministic 0–100 Context Health Score together with detailed findings and suggested fixes. ContextOps intentionally does not evaluate: - prompt engineering quality - reasoning ability or hallucinations - factual correctness - retrieval relevance - LLM outputs It focuses exclusively on the structural quality of the context before inference. contextops inspect context.json Context Health Score: 81 / 100 ✓ Low structural complexity ✓ Good source diversity Warnings • 214 duplicated tokens detected • Retrieval occupies 78% of context • Two retrieval chunks are near duplicates Estimated token savings: 12% Use --roast mode for more candid diagnostics: contextops inspect context.json --roast pip install contextops Run the interactive demo: contextops demo Dependencies: Only tiktoken and click . No GPU, no network, no external API keys required. Save your LLM payload before sending it to the model. python import json messages = {"role": "system", "content": system prompt}, {"role": "user", "content": user query}, Add two lines before your API call with open "context.json", "w" as f: json.dump messages, f, indent=2 response = openai.chat.completions.create model="gpt-4o", messages=messages Or use the structured dict format for richer analysis: { "system": "You are a helpful customer support bot.", "messages": {"role": "user", "content": "How long will my refund take?"} , "chunks": {"content": "Refunds take 3-5 business days.", "source": "docs/refunds.md"} , "memory": "The user asked about a refund yesterday." , "tools": {"name": "search api", "output": "Tool response text here"} } contextops inspect context.json contextops check context.json --min-score 75 Analyse a context file and display a rich report. contextops inspect context.json contextops inspect context.json --roast contextops inspect context.json --profile rag contextops inspect context.json --explain contextops inspect context.json --json-output | Flag | Default | Description | |---|---|---| --json-output | off | Output raw JSON instead of terminal format | --model <name | gpt-4o | Model for token encoding e.g. gpt-4 , claude-3 | --profile <name | general | Archetype: rag , agent , chatbot , toolchain | --config <path | none | Path to a JSON config file with custom thresholds | --retrieval-max-ratio <f | 0.70 | Override max allowed ratio for retrieval chunks | --system-max-ratio <f | 0.50 | Override max allowed ratio for system prompt | --memory-max-ratio <f | 0.50 | Override max allowed ratio for memory entries | --tool-max-ratio <f | 0.60 | Override max allowed ratio for tool outputs | --explain | off | Show detailed "Top Score Drivers" for each penalty | --roast | off | Enable brutally honest score-band commentary | CI gate. Exits with code 0 pass or 1 fail . contextops check context.json --min-score 75 | Exit Code | Meaning | |---|---| 0 | PASS — score meets or exceeds threshold | 1 | FAIL — score is below threshold or analysis error | 2 | ERROR — invalid JSON or unreadable input | GitHub Actions example: - name: Check context quality run: contextops check context.json --min-score 75 --profile rag Compare two context snapshots to detect regressions. contextops diff before.json after.json Shows score delta, which penalties changed, and whether the change is an improvement or regression. Useful for A/B testing retrieval strategies or prompt refactors. Run a deterministic stability report to verify the scoring engine is working correctly. contextops stability contextops stability context.json Local-only, opt-in telemetry for tracking context quality trends over time. contextops telemetry status Check if telemetry is active contextops telemetry log --limit 25 Show recent events contextops telemetry trends --days 7 Show 7-day quality trends Generate a GitHub shields.io markdown badge. contextops badge Uses your last telemetry score contextops badge --score 87 Use a specific score Output: ContextOps https://img.shields.io/badge/ContextOps-87-green https://github.com/Abhijeet777/contextops python from contextops.api.inspect import inspect context result = inspect context payload, dict, list, or plain string model="gpt-4o", model for token counting archetype="rag", archetype profile optional print result.score int 0–100 print result.token breakdown.wasted tokens for rec in result.recommendations: print f" - {rec.fix}" Full result fields: result.score int 0–100 result.score breakdown redundancy, density, structure, concentration penalties result.token breakdown total tokens, by type, wasted tokens, estimated cost usd result.redundancy findings list RedundancyFinding result.structure findings list StructureFinding result.recommendations list Recommendation result.density signal DensitySignal result.archetype resolved e.g. "rag" result.roast str | None if roast enabled result.metadata item count, model, version python from contextops.api.diff import diff contexts result = diff contexts payload a, payload b python from contextops.core.config import ContextOpsConfig config = ContextOpsConfig.default profile="rag" config = ContextOpsConfig.from dict { "retrieval max ratio": 0.90, "system max ratio": 0.30, "roast enabled": True } pip install contextops langchain-core python from contextops import ContextOps Log the score default — non-blocking chain = chain.with config { "callbacks": ContextOps.auto } Block execution if context quality is too low chain = chain.with config { "callbacks": ContextOps.auto mode="block", min score=75, profile="rag" } result = chain.invoke {"question": "What is the refund policy?"} | Mode | Behaviour | |---|---| "log" | Prints score report to stdout default | "warn" | Emits Python warning if score < min score | "block" | Raises ContextOpsScoreError if score < min score — blocks LLM call | Archetypes adjust structural thresholds for your specific use case. The global 0–100 score is never affected — only which warnings fire. | Profile | When to Use | Retrieval | System | Memory | Tool | |---|---|---|---|---|---| general | Default — mixed use cases | 70% | 50% | 50% | 60% | rag | Pure document retrieval | 95% | 40% | 20% | 30% | agent | Autonomous agents with tool loops | 50% | 40% | 40% | 90% | chatbot | Conversational apps with large history | 40% | 50% | 85% | 30% | toolchain | Multi-tool pipelines | 50% | 40% | 30% | 95% | Resolution order highest priority wins : --profile CLI flag archetype= Python API argument "archetype" key inside the JSON payload config.context profile - Default: "general" ContextOps accepts three input formats. Structured dict recommended — gives the richest analysis : { "system": "...", "messages": ... , "chunks": ... , "memory": ... , "tools": ... } OpenAI message list paste your existing payload directly : {"role": "system", "content": "You are a helpful bot."}, {"role": "user", "content": "Hello"} Plain string treated as a system prompt . Score = max 0, min 100, round 100 − total penalty Where total penalty = redundancy + density + structure + concentration . | Penalty | Max | What It Detects | |---|---|---| Redundancy | 30 | Lexical duplication via Jaccard + N-gram overlap | Density | 30 | Format overhead, whitespace waste, entropy compression | Structure | 20 | Retrieval dominance, system bloat, memory explosion | Concentration | 20 | Source dominance + entropy imbalance across chunks | Lexical only.ContextOps uses N-gram overlap, Jaccard similarity, and exact string matching — not semantic similarity. This is intentional: it is a structural analyser, not a semantic one. If LeetCode is DSA for algorithms , ContextBench is DSA for context . Just as competitive programming teaches you that the algorithm matters — not just the answer — ContextBench teaches you that context architecture matters , not just whether the LLM eventually gets it right. A brute-force O N² solution that produces the correct answer still gets penalized for time complexity. A bloated, redundant context that still produces a good LLM response still gets penalized for structural waste. The leaderboard notices both. | LeetCode / DSA | ContextBench Leaderboard | |---|---| | Problem set | 1,500 pre-built context windows across 5 failure categories | | Judge | ContextOps engine — deterministic scorer, same output every time | | Your solution | optimize context ctx — takes a broken context, returns a better one | | Score metric | Quality 50% + Compression 35% + Latency 15% | | Leaderboard | Ranked by final score across all benchmark samples | | Adversarial track | ContextSecBench — 9,500 adversarial attack payloads | ContextBench contains 1,500 samples across 5 categories — think of these as your difficulty tiers: | Category | Samples | What It Tests | |---|---|---| | Optimal Architectures | 300 | Healthy pipelines — prove you don't produce false positives | | Structural Failures | 300 | System prompt bloat, retrieval flooding, memory explosion | | Redundancy Failures | 300 | Near-duplicate clusters, boilerplate explosion, paraphrase loops | | Agent Architecture Failures | 300 | Multi-agent context explosion, recursive planning loops, tool chain bloat | | Temporal Context Drift | 300 | Stale memory injection, retrieval drift, invalidated historical state | Every sample has a ground truth — just like a LeetCode problem has expected output: { "ground truth": { "failure modes": "system prompt bloat" , "expected properties": { "contains redundancy": false, "contains density bloat": true, "contains structure imbalance": true } } } Flag redundancy when contains redundancy: false ? False positive. Score drops. Miss the density bloat? False negative. Score drops. Instead of writing a sorting algorithm, you write a context optimizer: php def optimize context ctx: dict - dict: Your logic: prune, compress, deduplicate, rebalance return better ctx The harness runs your function across all 1,500 samples and scores: final score = 0.50 quality score ContextOps CHS must stay ≥ 78 + 0.35 compression Token reduction reward + 0.15 latency mult Speed penalty if you're slow 100 The Quality Floor Gate score ≥ 78 is non-negotiable — optimizers that destroy context quality to hit compression targets are disqualified. ContextSecBench is the adversarial extension — the CTF track on top of LeetCode. 9,500 attack payloads covering: Prompt injection hiding — malicious instructions buried deep inside retrieved context Truncation smuggling — payloads designed to survive chunking with harmful content intact Semantic Denial of Service SDoS — padding the context window to exhaust the attention budget Context poisoning — subtle corruption of retrieval content to bias model behavior Format corruption — structural attacks that break parser assumptions Your optimizer must handle malicious context the same way a secure sorting algorithm must handle adversarial inputs. Poor context architecture causes problems long before the model itself becomes the bottleneck: - Higher token costs and inference latency - Lost-in-the-middle failures on long contexts - Degraded reasoning quality from attention bandwidth collapse - Context window exhaustion in long-running agent loops - Hidden operational waste that compounds silently at scale Better models don't solve poor context architecture. ContextBench measures the layer that does. The same input always produces the same score, breakdown, findings, and recommendations — on any machine, at any time. No randomness. No embeddings, GPUs, inference APIs, or external AI services. The entire engine is pure Python math. Guaranteed <2s for payloads ≤ 5,000 tokens. <10s for 50,000-token payloads. Works offline. Exit-code driven. Measures context architecture — not semantic quality, not model correctness, not factual truth. | Payload Size | Max Execution Time | |---|---| | ≤ 5,000 tokens | < 2 seconds | | ≤ 20,000 tokens | < 5 seconds | | ≤ 50,000 tokens | < 10 seconds | ContextOps aims to make LLM context as observable, measurable, and testable as source code. Just as modern software uses compilers, linters, and static analysis before deployment, LLM systems should validate the structural quality of their context before inference. Retrieve │ ▼ Build Context │ ▼ ContextOps ←── structural gate │ ▼ LLM | Document | Contents | |---|---| | STABILITY.md /Abhijeet777ui/contextops/blob/main/STABILITY.md USER GUIDE.md /Abhijeet777ui/contextops/blob/main/USER GUIDE.md HOW TO GET JSON.md /Abhijeet777ui/contextops/blob/main/HOW TO GET JSON.md CONTRIBUTING.md /Abhijeet777ui/contextops/blob/main/CONTRIBUTING.md CHANGELOG.md /Abhijeet777ui/contextops/blob/main/CHANGELOG.md Contributions are welcome — bug fixes, new ContextBench samples, documentation improvements, and new analyzer signals. Read CONTRIBUTING.md for: - Development setup and project structure - How to run the test suite including the chaos and signal contract tests - Rules for adding new signals or changing the scoring engine - The PR checklist and commit style guide - What the stability contract forbids For anything non-trivial, open an issue first so we can align before you write code. Repository: github.com/Abhijeet777/contextops https://github.com/Abhijeet777/contextops Released under the Sustainable Use License /Abhijeet777ui/contextops/blob/main/LICENSE .