Static analysis for LLM context.
ContextOps is a deterministic, model-independent context linter for LLM applications.
It analyzes the context sent to an LLM before inference and detects structural problems such as redundancy, token waste, context imbalance, and source concentration. It produces a reproducible Context Health Score (CHS) together with actionable diagnostics — without embeddings, model calls, or external services.
Think of ContextOps as ESLint for LLM context.
Modern software engineering has deterministic quality gates.
- Compilers catch syntax errors.
- Linters catch code smells.
- Formatters enforce consistency.
- Static analyzers detect architectural problems.
LLM applications rarely have an equivalent layer for the context they send into models.
Instead, prompts quietly grow over time:
- duplicated retrieval chunks
- bloated system prompts
- runaway conversation history
- excessive tool output
- hidden token waste
These issues increase latency and cost, make model behavior less predictable, and often go unnoticed until production.
ContextOps makes context quality observable, measurable, and testable before inference.
ContextOps evaluates four structural dimensions.
| Dimension | Max Penalty | What It Measures |
|---|---|---|
| Redundancy | ||
| 30 pts | Lexical duplication across context items | |
| Density | ||
| 30 pts | Token waste from formatting and structural bloat | |
| Structure | ||
| 20 pts | Distribution imbalance between context components | |
| Concentration | ||
| 20 pts | Over-reliance on a single document or source |
The result is a deterministic 0–100 Context Health Score together with detailed findings and suggested fixes.
ContextOps intentionally does not evaluate:
- prompt engineering quality
- reasoning ability or hallucinations
- factual correctness
- retrieval relevance
- LLM outputs
It focuses exclusively on the structural quality of the context before inference.
contextops inspect context.json
Context Health Score: 81 / 100
✓ Low structural complexity
✓ Good source diversity
Warnings
• 214 duplicated tokens detected
• Retrieval occupies 78% of context
• Two retrieval chunks are near duplicates
Estimated token savings: 12%
Use --roast
mode for more candid diagnostics:
contextops inspect context.json --roast
pip install contextops
Run the interactive demo:
contextops demo
Dependencies: Only tiktoken
and click
. No GPU, no network, no external API keys required.
Save your LLM payload before sending it to the model.
import json
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_query},
]
with open("context.json", "w") as f:
json.dump(messages, f, indent=2)
response = openai.chat.completions.create(model="gpt-4o", messages=messages)
Or use the structured dict format for richer analysis:
{
"system": "You are a helpful customer support bot.",
"messages": [
{"role": "user", "content": "How long will my refund take?"}
],
"chunks": [
{"content": "Refunds take 3-5 business days.", "source": "docs/refunds.md"}
],
"memory": ["The user asked about a refund yesterday."],
"tools": [{"name": "search_api", "output": "Tool response text here"}]
}
contextops inspect context.json
contextops check context.json --min-score 75
Analyse a context file and display a rich report.
contextops inspect context.json
contextops inspect context.json --roast
contextops inspect context.json --profile rag
contextops inspect context.json --explain
contextops inspect context.json --json-output
| Flag | Default | Description |
|---|---|---|
--json-output |
||
| off | Output raw JSON instead of terminal format | |
--model <name> |
||
gpt-4o |
||
Model for token encoding (e.g. gpt-4 , claude-3 ) |
||
--profile <name> |
||
general |
||
Archetype: rag , agent , chatbot , toolchain |
||
--config <path> |
||
| none | Path to a JSON config file with custom thresholds | |
--retrieval-max-ratio <f> |
||
| 0.70 | Override max allowed ratio for retrieval chunks | |
--system-max-ratio <f> |
||
| 0.50 | Override max allowed ratio for system prompt | |
--memory-max-ratio <f> |
||
| 0.50 | Override max allowed ratio for memory entries | |
--tool-max-ratio <f> |
||
| 0.60 | Override max allowed ratio for tool outputs | |
--explain |
||
| off | Show detailed "Top Score Drivers" for each penalty | |
--roast |
||
| off | Enable brutally honest score-band commentary |
CI gate. Exits with code 0
(pass) or 1
(fail).
contextops check context.json --min-score 75
| Exit Code | Meaning |
|---|---|
0 |
|
| PASS — score meets or exceeds threshold | |
1 |
|
| FAIL — score is below threshold or analysis error | |
2 |
|
| ERROR — invalid JSON or unreadable input |
GitHub Actions example:
- name: Check context quality
run: contextops check context.json --min-score 75 --profile rag
Compare two context snapshots to detect regressions.
contextops diff before.json after.json
Shows score delta, which penalties changed, and whether the change is an improvement or regression. Useful for A/B testing retrieval strategies or prompt refactors.
Run a deterministic stability report to verify the scoring engine is working correctly.
contextops stability
contextops stability context.json
Local-only, opt-in telemetry for tracking context quality trends over time.
contextops telemetry status # Check if telemetry is active
contextops telemetry log --limit 25 # Show recent events
contextops telemetry trends --days 7 # Show 7-day quality trends
Generate a GitHub shields.io markdown badge.
contextops badge # Uses your last telemetry score
contextops badge --score 87 # Use a specific score
Output:
[](https://github.com/Abhijeet777/contextops)
python
from contextops.api.inspect import inspect_context
result = inspect_context(
payload, # dict, list, or plain string
model="gpt-4o", # model for token counting
archetype="rag", # archetype profile (optional)
)
print(result.score) # int 0–100
print(result.token_breakdown.wasted_tokens)
for rec in result.recommendations:
print(f" -> {rec.fix}")
Full result fields:
result.score # int 0–100
result.score_breakdown # redundancy, density, structure, concentration penalties
result.token_breakdown # total_tokens, by_type, wasted_tokens, estimated_cost_usd
result.redundancy_findings # list[RedundancyFinding]
result.structure_findings # list[StructureFinding]
result.recommendations # list[Recommendation]
result.density_signal # DensitySignal
result.archetype_resolved # e.g. "rag"
result.roast # str | None (if roast_enabled)
result.metadata # item_count, model, version
python
from contextops.api.diff import diff_contexts
result = diff_contexts(payload_a, payload_b)
python
from contextops.core.config import ContextOpsConfig
config = ContextOpsConfig.default(profile="rag")
config = ContextOpsConfig.from_dict({
"retrieval_max_ratio": 0.90,
"system_max_ratio": 0.30,
"roast_enabled": True
})
pip install contextops langchain-core
python
from contextops import ContextOps
chain = chain.with_config({
"callbacks": [ContextOps.auto()]
})
chain = chain.with_config({
"callbacks": [
ContextOps.auto(
mode="block",
min_score=75,
profile="rag"
)
]
})
result = chain.invoke({"question": "What is the refund policy?"})
| Mode | Behaviour |
|---|---|
"log" |
|
| Prints score report to stdout (default) | |
"warn" |
|
| Emits Python warning if score < min_score | |
"block" |
|
Raises ContextOpsScoreError if score < min_score — blocks LLM call |
Archetypes adjust structural thresholds for your specific use case. The global 0–100 score is never affected — only which warnings fire.
| Profile | When to Use | Retrieval | System | Memory | Tool |
|---|---|---|---|---|---|
general |
|||||
| Default — mixed use cases | 70% | 50% | 50% | 60% | |
rag |
|||||
| Pure document retrieval | 95% | ||||
| 40% | 20% | 30% | |||
agent |
|||||
| Autonomous agents with tool loops | 50% | 40% | 40% | 90% | |
chatbot |
|||||
| Conversational apps with large history | 40% | 50% | 85% | ||
| 30% | |||||
toolchain |
|||||
| Multi-tool pipelines | 50% | 40% | 30% | 95% |
Resolution order (highest priority wins):
--profile
CLI flagarchetype=
Python API argument"archetype"
key inside the JSON payloadconfig.context_profile
- Default:
"general"
ContextOps accepts three input formats.
Structured dict (recommended — gives the richest analysis):
{
"system": "...",
"messages": [...],
"chunks": [...],
"memory": [...],
"tools": [...]
}
OpenAI message list (paste your existing payload directly):
[
{"role": "system", "content": "You are a helpful bot."},
{"role": "user", "content": "Hello"}
]
Plain string (treated as a system prompt).
Score = max(0, min(100, round(100 − total_penalty)))
Where total_penalty = redundancy + density + structure + concentration
.
| Penalty | Max | What It Detects |
|---|---|---|
| Redundancy | ||
| 30 | Lexical duplication via Jaccard + N-gram overlap | |
| Density | ||
| 30 | Format overhead, whitespace waste, entropy compression | |
| Structure | ||
| 20 | Retrieval dominance, system bloat, memory explosion | |
| Concentration | ||
| 20 | Source dominance + entropy imbalance across chunks |
Lexical only.ContextOps uses N-gram overlap, Jaccard similarity, and exact string matching — not semantic similarity. This is intentional: it is a structural analyser, not a semantic one.
If LeetCode is DSA for algorithms, ContextBench is DSA for context.
Just as competitive programming teaches you that the algorithm matters — not just the answer — ContextBench teaches you that context architecture matters, not just whether the LLM eventually gets it right.
A brute-force O(N²) solution that produces the correct answer still gets penalized for time complexity. A bloated, redundant context that still produces a good LLM response still gets penalized for structural waste. The leaderboard notices both.
| LeetCode / DSA | ContextBench Leaderboard |
|---|---|
| Problem set | 1,500 pre-built context windows across 5 failure categories |
| Judge | ContextOps engine — deterministic scorer, same output every time |
| Your solution | optimize_context(ctx) — takes a broken context, returns a better one |
| Score metric | Quality (50%) + Compression (35%) + Latency (15%) |
| Leaderboard | Ranked by final_score across all benchmark samples |
| Adversarial track | ContextSecBench — 9,500 adversarial attack payloads |
ContextBench contains 1,500 samples across 5 categories — think of these as your difficulty tiers:
| Category | Samples | What It Tests |
|---|---|---|
| Optimal Architectures | 300 | Healthy pipelines — prove you don't produce false positives |
| Structural Failures | 300 | System prompt bloat, retrieval flooding, memory explosion |
| Redundancy Failures | 300 | Near-duplicate clusters, boilerplate explosion, paraphrase loops |
| Agent Architecture Failures | 300 | Multi-agent context explosion, recursive planning loops, tool chain bloat |
| Temporal Context Drift | 300 | Stale memory injection, retrieval drift, invalidated historical state |
Every sample has a ground truth — just like a LeetCode problem has expected output:
{
"ground_truth": {
"failure_modes": ["system_prompt_bloat"],
"expected_properties": {
"contains_redundancy": false,
"contains_density_bloat": true,
"contains_structure_imbalance": true
}
}
}
Flag redundancy when contains_redundancy: false
? False positive. Score drops. Miss the density bloat? False negative. Score drops.
Instead of writing a sorting algorithm, you write a context optimizer:
def optimize_context(ctx: dict) -> dict:
return better_ctx
The harness runs your function across all 1,500 samples and scores:
final_score = (
0.50 * quality_score # ContextOps CHS must stay ≥ 78
+ 0.35 * compression # Token reduction reward
+ 0.15 * latency_mult # Speed penalty if you're slow
) * 100
The Quality Floor Gate (score ≥ 78) is non-negotiable — optimizers that destroy context quality to hit compression targets are disqualified.
ContextSecBench is the adversarial extension — the CTF track on top of LeetCode.
9,500 attack payloads covering:
Prompt injection hiding— malicious instructions buried deep inside retrieved context** Truncation smuggling**— payloads designed to survive chunking with harmful content intact** Semantic Denial of Service (SDoS)— padding the context window to exhaust the attention budget Context poisoning**— subtle corruption of retrieval content to bias model behavior** Format corruption**— structural attacks that break parser assumptions
Your optimizer must handle malicious context the same way a secure sorting algorithm must handle adversarial inputs.
Poor context architecture causes problems long before the model itself becomes the bottleneck:
- Higher token costs and inference latency
- Lost-in-the-middle failures on long contexts
- Degraded reasoning quality from attention bandwidth collapse
- Context window exhaustion in long-running agent loops
- Hidden operational waste that compounds silently at scale
Better models don't solve poor context architecture. ContextBench measures the layer that does.
The same input always produces the same score, breakdown, findings, and recommendations — on any machine, at any time. No randomness.
No embeddings, GPUs, inference APIs, or external AI services. The entire engine is pure Python math.
Guaranteed <2s
for payloads ≤ 5,000 tokens. <10s
for 50,000-token payloads. Works offline. Exit-code driven.
Measures context architecture — not semantic quality, not model correctness, not factual truth.
| Payload Size | Max Execution Time |
|---|---|
| ≤ 5,000 tokens | < 2 seconds |
| ≤ 20,000 tokens | < 5 seconds |
| ≤ 50,000 tokens | < 10 seconds |
ContextOps aims to make LLM context as observable, measurable, and testable as source code.
Just as modern software uses compilers, linters, and static analysis before deployment, LLM systems should validate the structural quality of their context before inference.
Retrieve
│
▼
Build Context
│
▼
ContextOps ←── structural gate
│
▼
LLM
| Document | Contents |
|---|---|
STABILITY.mdUSER_GUIDE.mdHOW_TO_GET_JSON.mdCONTRIBUTING.mdCHANGELOG.mdContributions are welcome — bug fixes, new ContextBench samples, documentation improvements, and new analyzer signals.
Read ** CONTRIBUTING.md** for:
- Development setup and project structure
- How to run the test suite (including the chaos and signal contract tests)
- Rules for adding new signals or changing the scoring engine
- The PR checklist and commit style guide
- What the stability contract forbids
For anything non-trivial, open an issue first so we can align before you write code.
Repository: github.com/Abhijeet777/contextops
Released under the Sustainable Use License.