Spend the same money on a 5x-cheaper model and you get five tries for every one. On Ramp’s published data that buys Kimi K2.6 a small win over GPT-5.5: 4.06 bugs found on average to 3.64, out of 7. The win only lasts while the budget is small. Below is the script that computes it, and how to run it on your own pass/fail counts before you pick a model for an agent swarm.
On November 4 the AI Engineer YouTube channel posted “Our Smaller Model Beat Opus on Finance Tasks”, a 14-minute talk by Alexander Shevchenko of Ramp Labs. He covers three projects. The third is a swarm of 10,000 parallel coding agents that pen-tested Ramp’s backend and found seven high-severity vulnerabilities.
Paraphrasing the auto-generated captions: open-weight models up to five times cheaper than the frontier reasoning model did not detect five times less, because the scan runs as a swarm, so you can buy more of the cheaper tokens and get almost as good a detection rate.
That is a claim you can check, because Ramp published the data behind it.
The write-up is “Finding high-severity security issues with publicly available models” by Alex Levinson (May 27, 2026). The production scan used GPT-5.5 at high reasoning effort. The post says a scan that size “can easily exceed $20,000 USD” at full price, and calls Kimi K2.6 and DeepSeek V4 Pro “roughly five times cheaper per token”.
They then re-ran the same prompt and harness against the 7 confirmed high-severity bugs, 25 attempts per model per bug, with DeepSeek at high reasoning effort and Kimi with thinking enabled. The site serves the chart data as JSON at article.json. Converted from percentages to hits out of 25:
+--------------------+---------+-----------+----------+| Bug | GPT-5.5 | Kimi K2.6 | DeepSeek |+--------------------+---------+-----------+----------+| ISS-1 Config | 11 | 25 | 25 || ISS-2 API logic | 25 | 22 | 9 || ISS-3 Auth | 24 | 10 | 2 || ISS-4 User mgmt | 14 | 0 | 0 || ISS-5 Payments | 6 | 3 | 0 || ISS-6 Integrations | 5 | 4 | 0 || ISS-7 Access ctrl | 6 | 0 | 0 || Total of 175 | 91 | 64 | 36 |+--------------------+---------+-----------+----------+
The totals match the post’s own summary (52.0%, 36.6%, 20.6%), and the script checks that before it prints anything.
Per try, GPT-5.5 is clearly better: 52% against Kimi’s 36.6%. But a security scan is not graded like a coding task. Ramp’s post makes the point itself: the scan only needs one trajectory to land a real hit, and a failed pass costs nothing but tokens.
So the useful question is not “which model is better per try?” It is: if one GPT-5.5 try costs the same as five Kimi tries, which pile of tries finds more bugs?
For each bug you have c hits out of n tries. The chance that at least one of k fresh tries hits is the pass@k estimator from OpenAI's 2021 Codex paper:
1 - comb(n - c, k) / comb(n, k)
It uses only the 25 real tries, with no curve fitted. Summed over the 7 bugs it gives the number of bugs you should expect to find. The catch: it only works up to k = n, so the script never claims anything past 25 tries.
Download the data, save the script below as scancheck.py, and run it. It needs Python 3.8+ and nothing else. Medium drops blank lines inside code blocks; the script runs the same without them.
curl -sO https://labs.ramp.com/research/security-scanning-public-models/article.jsonpython3 scancheck.py article.json
"""scancheck.py - one GPT-5.5 try, or N cheaper tries, for the same money?Reads Ramp Labs' published detection data (25 tries per model per bug)and asks: at equal spend, how many of the 7 bugs does each model find?Your own results work too: a CSV with model,bug,hits,tries per line,expensive model first.usage: python3 scancheck.py ramp.json|yours.csv [price_ratio]"""import csv, json, re, sysfrom math import combRATE = "security-scanning-public-models/detection-rate"SUMMARY = "security-scanning-public-models/model-summary"def load_ramp(path): figs = json.load(open(path))["figures"] rows = figs[RATE]["spec"]["data"] bugs = sorted({r["issue"] for r in rows}) data = {} for s in figs[SUMMARY]["spec"]["stats"]: # "91 / 175 attempts" = 175 tries spread over the 7 bugs hits, total = map(int, re.findall(r"\d+", s["delta"])) n = total // len(bugs) got = {} for r in rows: if r["model"] == s["label"]: c = r["rate"] * n / 100 assert c == int(c), f"{r['issue']}: rate is not k/{n}" got[r["issue"]] = int(c) assert sum(got.values()) == hits, f"{s['label']}: != {hits} hits" data[s["label"]] = (n, got) return bugs, datadef load_csv(path): data, bugs = {}, [] for row in csv.reader(open(path)): if len(row) < 3 or not row[2].strip().lstrip("-").isdigit(): continue # skip blank lines and a header model, bug, hits, n = row prev_n, got = data.setdefault(model, (int(n), {})) assert int(n) == prev_n, f"{model}: tries must match per bug" assert 0 <= int(hits) <= int(n), f"{model},{bug}: hits > tries" assert bug not in got, f"{model},{bug}: duplicate line" got[bug] = int(hits) bugs += [bug] if bug not in bugs else [] for m, (n, got) in data.items(): assert len(got) == len(bugs), f"{m}: missing some bugs" return bugs, datadef p_found(n, c, k): """chance at least one of k tries hits, from c hits in n tries""" k = min(k, n) # pass@k estimator only works up to k = n return 1 - comb(n - c, k) / comb(n, k)def expected(n, got, k): return sum(p_found(n, c, k) for c in got.values())def main(): if len(sys.argv) < 2: sys.exit("usage: python3 scancheck.py ramp.json|yours.csv [ratio]") path = sys.argv[1] try: ratio = float(sys.argv[2]) if len(sys.argv) > 2 else 5.0 except ValueError: ratio = 0 if not 1 <= ratio < 1e6: sys.exit("error: price ratio must be between 1 and 1e6") try: = load_csv if path.endswith(".csv") else load_ramp bugs, data = (path) except (OSError, KeyError, ValueError, AssertionError) as e: print(f"error: cannot read {path}: {e}", file=sys.stderr) sys.exit(1) models = list(data) if len(models) < 2: sys.exit(f"error: {path} needs at least 2 models") base, cheap = models[0], models[1:] for m in models: n, got = data[m] raw = " ".join(f"{got.get(b, 0):>2}" for b in bugs) print(f"{m[:15]:<15} hits out of {n}, per bug: {raw}") print(f"\nprice ratio {ratio:g}x: each {base} try buys " f"{ratio:g} cheap tries (rounded down)") print(f"{base[:12] + ' tries':>18} | " + " | ".join(f"{m[:15]:>15}" for m in models)) for b in range(1, 6): k = int(b * ratio) # rounds down: never gives the cheap model extra cells = [(expected(*data[base], b), b > data[base][0])] cells += [(expected(*data[m], k), k > data[m][0]) for m in cheap] line = f"{b:>18} | " + " | ".join( f"{x:>15.2f}{'*' if cap else ''}" for x, cap in cells) print(line) print(f"(expected bugs found out of {len(bugs)}, from real tries;") print(" * = budget exceeds the tries in the data, value held at n)") one = expected(*data[base], 1) for m in cheap: n, got = data[m] tie = next((k for k in range(1, n + 1) if expected(n, got, k) >= one), None) never = [b for b in bugs if got.get(b, 0) == 0] print(f"\n{m}: needs {tie or f'more than {n}'} tries per bug") print(f" to match ONE {base} try ({one:.2f} bugs)") print(f" never found in {n} tries: {', '.join(never) or 'none'}") if never: print(f" (0 of {n}: 95% upper bound ~{3 / n:.0%} per try)") cap = expected(n, got, n) past = next((b for b in range(1, data[base][0] + 1) if expected(*data[base], b) > cap), None) word = "try" if past == 1 else "tries" when = f"passes it at {past} {word}" if past else "never passes it" print(f" ceiling with all {n} tries: {cap:.2f} bugs; {base} {when}")if __name__ == "__main__": main()
The output:
GPT-5.5 hits out of 25, per bug: 11 25 24 14 6 5 6Kimi K2.6 hits out of 25, per bug: 25 22 10 0 3 4 0DeepSeek V4 Pro hits out of 25, per bug: 25 9 2 0 0 0 0price ratio 5x: each GPT-5.5 try buys 5 cheap tries (rounded down) GPT-5.5 tries | GPT-5.5 | Kimi K2.6 | DeepSeek V4 Pro 1 | 3.64 | 4.06 | 2.28 2 | 4.74 | 4.69 | 2.65 3 | 5.43 | 4.93 | 2.85 4 | 5.90 | 5.00 | 2.97 5 | 6.22 | 5.00 | 3.00(expected bugs found out of 7, from real tries; * = budget exceeds the tries in the data, value held at n)Kimi K2.6: needs 4 tries per bug to match ONE GPT-5.5 try (3.64 bugs) never found in 25 tries: ISS-4, ISS-7 (0 of 25: 95% upper bound ~12% per try) ceiling with all 25 tries: 5.00 bugs; GPT-5.5 passes it at 3 triesDeepSeek V4 Pro: needs more than 25 tries per bug to match ONE GPT-5.5 try (3.64 bugs) never found in 25 tries: ISS-4, ISS-5, ISS-6, ISS-7 (0 of 25: 95% upper bound ~12% per try) ceiling with all 25 tries: 3.00 bugs; GPT-5.5 passes it at 1 try
Read each row as one budget. Row 1 is “one GPT-5.5 try per bug, or the same money in cheap tries”.
The talk said cheap tries get “almost as good” a detection rate. At this budget the data is a little better than that: five Kimi tries find 4.06 bugs on average, one GPT-5.5 try finds 3.64. Kimi wins by about 0.4 of a bug.
One row later it is gone. Two GPT-5.5 tries find 4.74, ten Kimi tries find 4.69. From row 3 on, GPT-5.5 pulls away: 5.43, 5.90, 6.22, while Kimi tops out at 5.00.
DeepSeek V4 Pro never comes close. Even 25 tries per bug, all the data there is, find 3.00 bugs on average, less than one GPT-5.5 try.
Kimi’s ceiling is 5.00 because it went 0 for 25 on two bugs: ISS-4 (user management) and ISS-7 (access control). GPT-5.5 found those in 14 and 6 of 25 tries. The post does not name them, but it says GPT-5.5 was “the only model to identify a couple of the more subtle bugs”. In the data, the only two bugs no other model found are ISS-4 and ISS-7.
Zero out of 25 does not prove Kimi can never find them. By the rule of three, 0 of 25 gives a 95% upper bound of about 12% per try, so 50 or 100 Kimi tries might land one. But inside the data, no number of extra Kimi tries closes that gap, and once you can afford 3 GPT-5.5 tries per bug, GPT-5.5 is ahead of Kimi’s best case.
It cuts the other way too. On ISS-1 (configuration) Kimi went 25 for 25 against GPT-5.5’s 11, the one issue where the post says Kimi “meaningfully outperforms” it.
That is the pattern to expect in your own swarms. More cheap tries win on the easy bugs, which they find most of the time anyway. Only the stronger model reaches the hard ones.
The second argument is the price ratio. It defaults to 5, Ramp’s “roughly five times cheaper per token”. Change it to whatever your own bill says:
python3 scancheck.py article.json 3 | sed -n 5,8p
price ratio 3x: each GPT-5.5 try buys 3 cheap tries (rounded down) GPT-5.5 tries | GPT-5.5 | Kimi K2.6 | DeepSeek V4 Pro 1 | 3.64 | 3.55 | 1.99 2 | 4.74 | 4.24 | 2.38
At 3x Kimi already loses row 1, 3.55 to 3.64. The line that does not depend on the ratio is the break-even: Kimi needs 4 tries per bug to match one GPT-5.5 try. So Kimi only pays off if a Kimi try costs a quarter of a GPT-5.5 try or less.
Note the word “try”. Ramp’s 5x is per token. A try that thinks longer or loops more costs more tokens, so measure cost per finished session from your own logs, not from the price page. Prices also move. GPT-5.5 is no longer on OpenAI’s pricing page, which now lists GPT-5.6 and the GPT-6 models. Fireworks’ model page says serverless Kimi K2.6 is deprecated from September 25, 2026, with on-demand deployments continuing.
The pass@k math works for any task where one hit is enough: security scans, test generation, search for a failing input. Run each model a fixed number of times on a few known targets, count the hits, and write one line per model and target, with the expensive model first:
big-model,auth,9,10big-model,billing,2,10small-model,auth,6,10small-model,billing,1,10
python3 scancheck.py mine.csv 4 | sed -n 4,8p
price ratio 4x: each big-model try buys 4 cheap tries (rounded down) big-model tries | big-model | small-model 1 | 1.10 | 1.40 2 | 1.38 | 1.80 3 | 1.53 | 2.00*
The star marks a budget bigger than the tries you recorded: 3 big-model tries buy 12 small ones, but there are only 10 in the data, so the value is held at 10. Below the table the script prints the break-even and the targets a model never hit. A header line is skipped; a missing target, a duplicate line or hits above tries stop it with an error and exit 1, as does a wrong path. No arguments prints the usage line.
It treats the 25 tries per bug as a fair sample and each new try as a fresh draw. It does not model the 10,000-session scan itself, where agents were pointed at endpoints, not at known bugs. It does not price the dedup and validation passes that turned 6,000 raw tickets into seven confirmed highs. And 7 bugs is a small sample: one bug changing hands moves the averages a lot.
Reproduce it: scancheck.py prints every number in the output blocks of sections 05, 08 and 09, from Ramp's article.json and the four-line CSV. The table in section 02 is the same counts. The quotes, the $20,000 figure and the 6,000 tickets come from Ramp's post; the 10,000 agents and seven bugs come from the talk and the post. Dates, prices and model availability come from the linked pages.
On Ramp’s own data the cheap swarm wins, but only at a small budget: five Kimi tries beat one GPT-5.5 try, 4.06 bugs to 3.64 on average. Kimi needs to be at least 4x cheaper per try to match or beat it, and it never found 2 of the 7 bugs, so from 3 GPT-5.5 tries per bug the frontier model wins at any price the 25-try data can test. Use the cheap swarm to cover the easy surface, and keep some frontier tries for the hard bugs.
How do you split budget between cheap and frontier models in your agent swarms?
If this saves you a bill, a clap helps others find it.
Follow me for daily LLM deep-dives.
Why Cheap Kimi Agents Only Beat GPT-5.5 on a Small Budget was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.