Developers often rely on manual, ad-hoc testing for LLM features, but a more systematic approach using automated red teaming tools like Giskard and Promptfoo can uncover critical vulnerabilities quick…
EvalPort introduces a grader system with 11 types for LLM evaluation, including exact_match, semantic_similarity, llm_judge, and custom, designed to be framework-agnostic and self-describing. The syst…
OpenEval, a new open-source project, aims to standardize LLM evaluation by defining a portable JSON Schema for test cases, graders, and results. The project provides SDKs, a CLI, and converters for po…
Pydantic Logfire ranks as the best AI agent optimization platform in 2026, according to a Pydantic analysis that evaluated tools on their ability to close the loop by proposing and shipping production…
AI red teaming tools like Giskard, DeepEval, and Promptfoo automate adversarial testing to systematically find edge-case failures in model logic, moving beyond manual 'vibe checks' that risk PR disast…
A developer gated CI on six open-source LLM evaluation frameworks in a real GitHub Actions merge queue for eight months, finding that only Promptfoo and DeepEval survived due to their deterministic ch…
A new comparison of LLM evaluation frameworks RAGAS, DeepEval, and Promptfoo reveals that the LLM-as-a-judge mechanism they all rely on has measurable biases—position bias, self-preference bias, and v…
A new defensive technique called a 'context bomb' — a short string hidden in decoy resources that triggers safety guardrails in offensive AI agents — reduced autonomous cyberattack success by roughly …
Promptfoo, an open-source evaluation and red-teaming framework, addresses the challenge of testing non-deterministic LLM outputs by treating prompts as versioned, testable code. The framework enables …
A developer evaluated six LLM-as-judge tools—DeepEval, Confident AI, Evidently, Braintrust, Promptfoo, and Future AGI—and found that none of them prioritize validating judge outputs against human labe…
A developer built a C# LLM event parser called EventParser and tested it using Promptfoo's LLM-as-a-judge evaluation. The project separates the prompt file from the code, allowing Promptfoo to test th…
OpenAI released SchemaFlow, an AI-assisted database change workflow using the OpenAI Agents SDK, which converts natural-language requests into structured JSON, performs impact analysis, generates SQL,…
OpenAI acquired cloud startup Ona on 11 June 2026 to integrate its secure, persistent agent workspace into Codex, the company's coding assistant. The deal aims to support Codex's growing enterprise us…
The open-source tool Promptfoo, acquired by OpenAI in March 2026, maps its 155 attack plugins to the OWASP LLM Top 10 2025 list for structured red teaming of LLM-powered products. It details the 2025 …
As engineering teams iterate on prompts for large language models, they face challenges similar to managing configuration files, requiring dedicated storage, versioning, sharing, and security. It revi…