Nexa-gauge – LLM evaluation framework with per-node scoring controls

Overview #

nexa-gauge is a graph-based evaluation system for LLM and LVLM application outputs. It replaces ad-hoc manual checks with a repeatable pipeline that can be run on local datasets or hosted datasets.

At a high level, nexa-gauge:

• Normalizes raw records into a typed evaluation state.
• Executes only the nodes required for the selected target.
• Reuses prior node outputs through deterministic caching.
• Produces a consistent per-case report for downstream tooling.

This architecture supports day-to-day prompt iteration, benchmark runs, and release gating with measurable quality and safety signals.

Why LLM-As-A-Judge Is Necessary #

Exact-match metrics are useful but limited for modern generative systems. In many real tasks, multiple answers can be valid, quality depends on context use, and failure modes are semantic rather than lexical.

LLM-as-a-judge provides scalable semantic evaluation by scoring outputs against explicit criteria. In nexa-gauge, this capability is combined with targeted metrics so teams can evaluate quality from multiple angles:

relevance

for input-output alignment.grounding

for support in provided context.redteam

for safety and risk behavior.geval

for rubric-based judgment.reference

for overlap with known reference answers.

Execution Model And Caching #

nexa-gauge provides two operational modes:

run

executes the selected branch and returns final artifacts.estimate

computes uncached eligible cost before execution.

Both modes follow the same branch-planning logic, which makes cost estimates actionable before you run full evaluations.

Caching is route-aware and deterministic. Reuse occurs only when input content and routing semantics are unchanged. Changes to inputs, prompts, or model routing intentionally invalidate affected steps.

Practical outcome:

• Teams can estimate budget before execution.
• Iterative runs avoid recomputing stable nodes.
• Results remain reproducible under fixed inputs and model routes.

Architecture #

Node Summary #

Input And Orchestration

Utility Nodes

Metric Essentials

Metric Nodes

Typical Workflow #

nexagauge estimate eval --input sample.json --limit 100

nexagauge run eval --input sample.json --limit 100 --output-dir ./report

For dataset fields, accepted aliases, and metric activation rules, see the Data Schema.

For iterative development, repeated runs on unchanged inputs and routing should show high cache reuse and lower incremental latency.