Eight analyzers, thirteen languages, and not one clean sweep
Reliable agentic code generation needs solid quality signals, so we created DataFlowBench and UsageBench to help evaluate Bifrost, our static analysis guardrail for keeping AI-generated code in check.
These benchmarks includes other analyzers as well as Bifrost. In this post, we'll look at DataFlowBench’s taint analysis tests, the analyzers, and how each run turns into a new set of results.
What is Taint Analysis? #
In the security space, we primarily care about sources that are potentially-attacker-controlled data vectors such as HTTP request parameters; sinks are places that use that data in ways that could allow the attacker to inject unintended behavior into our system: SQL queries, shell commands, filesystem writes, etc.
Accurately tracking the flow of this "tainted" data means modeling how the program behaves — its semantics. DataFlowBench measures how well each analyzer does this by testing small "toy" programs with known source-to-sink expectations, "frozen" before the tools run. Tool-specific adapters provide working configurations and build information to give each analyzer its best shot.
In total, the cases range from dead simple flows to fairly complex ones. DataFlowBench v0.7.1 records 3,480 analyzer-case results across 852 frozen cases in thirteen programming languages. In addition to our very own Bifrost, we have seven other analyzers with mixed levels of language coverage. Finally, all releases present a web-based result explorer generated from the frozen set of results.
This is a living benchmark. A perfect score for a language triggers a review of the cases and new challenges for a future version.
Note: The benchmark tests the behaviors represented by these cases, so the scores are not necessarily a guarantee of performance on a production application.
What is a kernel? #
A kernel is a small program that isolates one data-flow behavior while still being executable by the real analyzer.
The kernels range from local propagation to branches, aliases, calls, reflection, and recursion. Matched positive and negative cases test whether an analyzer can find a real flow and reject its near miss.
For example, here is a simple positive kernel in Java:
static void run() {
int value = dfb_source();
value = value; // taint remains
dfb_sink(value); // expected flow
}
A positive asks whether the analyzer can follow the flow, and in the above example, the data flow from dfb_source to dfb_sink is preserved. A closely related negative case is:
static void run() {
int value = dfb_source();
value = 0; // taint is killed
dfb_sink(value); // expected no flow
}
In this example, we kill the flow by overwriting value, so an analyzer should report that no flow exists between dfb_source and dfb_sink in this method.
Between these cases, only one expression changes. Finding the real flow tests recall; rejecting the near miss tests precision.
Each language has the following kernel families in v0.7.1:
How a run becomes a result #
Each benchmark case begins as a frozen fixture with a preregistered expected outcome. An analyzer-specific adapter runs the pinned tool and retains its raw output, which DataFlowBench then converts into a common evidence model: reached, not-reached, inconclusive, unsupported, or runner-error.
Only then do we compare the result with the expected outcome and score it as a true or false positive, a true or false negative, or a non-decision.
Each run ends in one of five states:
- A finding matched to the expected source-to-sink flow is
reached; - a completed run without one is
not-reached; - partial analysis is
inconclusive; - explicitly unavailable capabilities are
unsupported, and - execution failures are
runner-error.
Multiple paths do not increase the score, although the raw evidence is retained.
Making unlike tools comparable #
We ask every analyzer the same semantic question using its strongest documented configuration. Raw evidence stays available so maintainers can reproduce the results, debug, or benchmaxx their tool.
The field #
The eight analyzers in this comparison approach taint analysis from different angles and target different users. All eight can run locally, although their licenses deserve a careful read.
Bifrost, CodeQL, Joern, and Semgrep CE span a broad range of languages, while OpenTaint and FlowDroid are JVM specialists. Among Meta’s tools, Infer covers C, C++, and Java, whereas Pysa focuses on Python.
Note: Semgrep Community Edition is evaluated within its documented single-function, per-file data-flow scope. Cross-function and cross-file analysis require Semgrep Pro Engine.
Before comparing results, we should separate the number of languages are supported from correctness.
We can only score languages with both a working adapter and benchmark cases, so the tested footprint may be smaller than a tool’s documented support. The chart below shows both. Wider bars mean broader reach, not better accuracy. And yes, as a Brokk product, DataFlowBench has prioritized matching Bifrost's coverage first.
Responsiveness and latency #
Correct answers are only useful if they arrive in time. A delay that's fine in a nightly CI job can be maddening inside an editor or agent loop.
Timing is tricky: some tools pay an indexing or build cost up front, while others repeat most of their work on every invocation. Cold-start timings include process startup; warm query latency and estimated per-invocation overhead are measured separately.
These timings measure cold, per-case analyzer invocations. Tools cover different languages and case populations, so their workloads are not identical. They tell us how quickly each tool handles these small benchmark cases, not how its runtime or memory use grows with repository size. Scalability needs a separate benchmark.
Decisiveness versus accuracy #
Precision is the share of reported flows that are real; recall is the share of real flows the analyzer finds. Neither tells us how often it reaches a decision. For each analyzer/language core, DataFlowBench therefore reports decisiveness, or decision coverage: (TP + TN + FP + FN) / all selected scored assertions.
Once a case enters that population,
inconclusive, unsupported, and runner-error remain in the denominator; a tool cannot improve its decisiveness merely by declining difficult cases. Alongside this, we report conclusively correct coverage: (TP + TN) / all selected scored assertions. An analyzer is never rewarded for being confidently wrong. The complete result shows both, making it possible to distinguish a cautious analyzer from a noisy one, or broad reliable coverage from a small island of correct answers.
The following plots show why that distinction matters:
Bifrost sits furthest right, with the broadest documented language support, but its accuracy across covered kernels is 61% because inconclusive results remain in the denominator. Infer leads that vertical axis.
When we look only at conclusive answers, Bifrost has 100% accuracy in this corpus. It reaches a decision on 61% of the full corpus; CodeQL leads that decision-coverage measure at 83.6%.
The results at a glance #
No analyzer wins every dimension. Bifrost leads responsiveness, precision, and language coverage; CodeQL leads full-corpus decisiveness; OpenTaint leads recall. The scorecard keeps these trade-offs separate rather than collapsing them into a misleading overall rank.
What the scorecard means for your codebase #
If false positives are your main headache, Bifrost’s precision stands out. If finding more real flows matters most, OpenTaint’s recall deserves a look if you're only on the JVM. CodeQL reaches decisions on the largest share of the full corpus - so arguably the best "generalist" for recall at the cost of being last for reponsiveness.
Those are useful starting points, but each comes with a catch: precision needs decision coverage beside it, and high recall can mean more false alarms to sort through.
Language support might be a good first filter. OpenTaint’s recall is relevant to JVM code; Infer’s precision may appeal to a C, C++, or Java project. FlowDroid is another JVM option, while Pysa focuses on Python. A specialist’s narrower (and often more precise) reach may be perfectly fine if it covers the code you actually ship.
Then there’s the wait. Bifrost’s responsiveness makes it interesting for editor and agent loops, where every adds up. A slower analyzer may still fit a background CI job or a deeper security review. Check the timings for your language and workflow: a specialist can change the shortlist, and being fast within one group does not mean being fast enough for an interactive loop.
Reading the ranking responsibly #
There is no best taint analyzer, only the best one for the job. Query languages and day-to-day usability deserve a discussion of their own.
We’ll turn these trade-offs into explicit use-case rankings in a follow-up post. For now, the scorecard shows where each tool shines and what you might have to give up.
The benchmark is a starting point. Check language support, maintenance, licensing, and integration options. Include checking MCP support if you want an analyzer in an agent loop. If false positives are a deal-breaker, look at precision alongside decision coverage. A tool can be confidently wrong.
Our bet is that OpenTaint and Bifrost (we’re working on that bit) will see the biggest shifts between benchmark releases. The others are evolving too, though platform activity does not always translate into benchmark gains. Joern is moving quickly (e.g., more language coverage), but its open-source data-flow engine is changing more slowly. Pysa’s Pyrefly migration could shake things up, while CodeQL and Infer look more like mature systems under steady refinement. Semgrep is highly active, but much of its interprocedural investment is in Pro, rather than the Community Edition tested here.
Stay tuned for the next update!