A developer testing the two most popular LLM-as-judge faithfulness metrics, RAGAS and DeepEval, found that they gave opposite scores for the same fabricated answer: RAGAS scored it 0.0 while DeepEval …
A senior engineering manager warns that basic PDF chatbots with LangChain and Streamlit will no longer impress hiring managers in 2026, as tech companies now demand engineers who understand system lat…
A developer demonstrates using DeepEval to evaluate an enterprise agent, contrasting its test-case-first paradigm with RAGAS's batch evaluation. The implementation uses a custom judge LLM (glm-4-flash…
A developer building ARIA, a free AI tutor for 1.6 billion children across 35 languages, discovered a severe gap between test and real-world performance: the system scored 94% on automated Socratic co…
A developer's LLM-based Socratic teaching system, ARIA, scored 94% on automated compliance benchmarks but failed 77.8% of real student interactions, a phenomenon the developer calls the 'Watermelon Ef…
A developer found that an LLM judge gave opposite results for the same eval run on consecutive days due to position bias, one of three systematic biases documented in the 2023 paper "Judging LLM-as-a-…
A developer's practical guide details the engineering challenges and solutions for deploying production-ready Retrieval-Augmented Generation (RAG) applications, covering data indexing, vector stores, …
Atlas, an enterprise RAG copilot project, treats retrieval quality like unit tests by blocking merges on quality or cost regressions. The pipeline uses deterministic, GPU-free eval gates with committe…
A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble inclu…
A developer building FetchGate warns that AI agents often answer questions based on pages they never actually read, because an HTTP 200 status code only confirms bytes arrived, not that the intended c…
Researchers fine-tuned LLaMA 3 (8B) as a drop-in reranker for RAG pipelines, achieving gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctne…
A new comparison of LLM evaluation frameworks RAGAS, DeepEval, and Promptfoo reveals that the LLM-as-a-judge mechanism they all rely on has measurable biases—position bias, self-preference bias, and v…
Amazon proposes the Unified RAG Evaluation Schema (URES) to standardize cross-supplier quality measurement for RAG and agentic workloads, enabling enterprises to compare evaluation results from differ…
Researchers proposed DysLexLens, a low-resource LLM framework to analyze dyslexic learners' experiences with AI tools using Reddit forum data. The framework filters noisy posts, integrates knowledge-g…
A study benchmarking six RAG architectures on a financial PDF reveals that Vision RAG achieves the highest answer correctness (0.823) but low faithfulness (0.538), while Graph RAG is the most faithful…
A developer built a groundedness-checking layer for a local RAG system to catch hallucinations, running on Ollama with an 8B model. The layer decomposes answers into claims and verifies each against s…
A developer built an evaluation pipeline for LLM-based RAG systems that runs on every deploy to detect drift and hallucinations. The pipeline uses RAGAS with LLM-as-judge to measure faithfulness and a…
Jeroen Herczeg has launched a Leanpub book titled 'Retrieval-Augmented Generation: An Engineer's Guide to Building RAG Systems with Your Own Data', aimed at engineers building production RAG systems. …