{"slug": "how-evalport-s-grader-system-works-11-types-for-llm-evaluation", "title": "How EvalPort's Grader System Works: 11 Types for LLM Evaluation", "summary": "EvalPort introduces a grader system with 11 types for LLM evaluation, including exact_match, semantic_similarity, llm_judge, and custom, designed to be framework-agnostic and self-describing. The system allows multiple graders per test case and records scores separately, aiming to cover 90% of real-world eval needs.", "body_md": "When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it.\n\nThe result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing.\n\nexact_match — Compare output to expected output, optionally ignoring case.\n\ncontains — Check if the output contains a substring.\n\nregex — Match against a regular expression.\n\nsemantic_similarity — Embed output and expected output, compare cosine similarity against a threshold.\n\nllm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader.\n\njson_schema — Validate that the output is valid JSON matching a JSON Schema.\n\njson_path — Extract a value from JSON output using a JSONPath expression, then compare it.\n\ncode — Run a function to evaluate the output.\n\nhuman — Defer to human review.\n\nmodel_graded — Compare the output to a reference answer using a model.\n\ncustom — Escape hatch for graders not covered by built-in types.\n\nA test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately.\n\nSelf-describing: An eval suite carries everything a framework needs to execute it.\n\nFramework-agnostic: Any framework can implement any subset of grader types.\n\nExtensible: The custom type lets frameworks bring their own graders.\n\nComparable: Results from different frameworks use the same grader IDs.\n\npip install evalport-sdk\n\nnpm install evalport-sdk\n\nSpec: [https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md](https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md)", "url": "https://wpnews.pro/news/how-evalport-s-grader-system-works-11-types-for-llm-evaluation", "canonical_source": "https://dev.to/adha_ak_d60b39fbb66769fd1/how-evalports-grader-system-works-11-types-for-llm-evaluation-1m06", "published_at": "2026-08-04 20:59:13+00:00", "updated_at": "2026-08-04 21:47:34.946747+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": ["EvalPort", "DeepEval", "Promptfoo", "Inspect AI"], "alternates": {"html": "https://wpnews.pro/news/how-evalport-s-grader-system-works-11-types-for-llm-evaluation", "markdown": "https://wpnews.pro/news/how-evalport-s-grader-system-works-11-types-for-llm-evaluation.md", "text": "https://wpnews.pro/news/how-evalport-s-grader-system-works-11-types-for-llm-evaluation.txt", "jsonld": "https://wpnews.pro/news/how-evalport-s-grader-system-works-11-types-for-llm-evaluation.jsonld"}}