Jev matched 72.5% of Kepler labels after a prompt rewrite

Pavel Rabtsevich reported on September 23rd that his Jev model matched NASA Exoplanet Archive dispositions on 5,840 of 8,054 Kepler Objects of Interest, a 72.51% match rate, after a prompt rewrite that followed a first run scoring 54.2% against a fixed three-rule baseline of 64.4%. The linked technical report states the run used Jev 1.13, made 8,054 calls with a median latency of 322 milliseconds and an estimated cost of $0.34, and that Jev correctly labeled 1,999 of 2,731 archive-confirmed planets (73.2% recall, 73.4% precision), while the report notes the gist lacks the full dataset snapshot, preprocessing code, leakage-audit implementation and per-object results needed for independent rescoring. The result is a retrospective label-prediction test, not a new planet discovery, and the report says there was no held-out prompt-development set.

Jev matched 72.5% of Kepler labels after a prompt rewrite Pavel Rabtsevich's linked test classified 8,054 historical signals, but its stronger result came after the first run informed a redesigned prompt. By Ryan Merket https://runtimewire.com/author/ryan-merket · Published Primary source: X https://x.com/p rabtsevich/status/2102733624147865770 Why it matters The 1,999 figure is a class-specific recall result from a retrospective label-prediction test. Jev beat the fixed rules baseline only after a prompt redesign informed by an earlier run, so the result shows how much task framing can move an AI benchmark - and why it should not be described as discovering planets. Pavel Rabtsevich https://x.com/p rabtsevich said on September 23rd that he had used Jev to classify NASA Kepler signals, reporting 1,999 correct matches among 2,731 archive-labeled confirmed planets. The technical report linked with the experiment https://gist.github.com/ipaulsmith/e5c3ae3a492a455435d5bfc161404312 puts that result in context: it was the second of two runs, after the first attempt scored below a simple rules-based benchmark. The final run classified 8,054 Kepler Objects of Interest, or KOIs, into three categories: confirmed planet, candidate, or false positive. It matched the NASA Exoplanet Archive's recorded disposition on 5,840 cases, or 72.51%, according to the report. A fixed three-rule classifier matched 64.4% on the same records. Jev correctly labeled 1,999 of 2,731 archive-confirmed planets, a 73.2% recall rate for that category. That is a test of whether a model can predict an archive label from supplied measurements. It is not a new discovery of planets. The report says Jev received measurements about the signals and their host stars, while the archive classifications were hidden until predictions had been saved. NASA's archive distinguishes confirmed planets from candidates that remain unconfirmed and false positives; matching those labels is not the same as independently establishing what caused each signal. The model's first pass scored 54.2%, below the same 64.4% baseline. In that run, Jev received 21 fields, mostly raw measurements, and selected "confirmed" only eight times. The second pass changed the representation to eight named fields and gave the model more structured criteria for each category. It followed inspection of the first result, and the report says there was no held-out prompt-development set. The final score therefore measures a revised setup informed by an earlier attempt, not a prompt fixed before testing began. The report provides a useful accounting of that revision, along with its limits. It says the run used Jev 1.13, made 8,054 calls, had a median latency of 322 milliseconds, and cost an estimated $0.34 at the cited input-token price. But it also says the gist does not include the full dataset snapshot, preprocessing code, leakage-audit implementation, or per-object results needed to rerun and independently rescore every prediction. The reported measurements are inspectable; a complete reproduction is not available in the attached material. The category-level scores show why the overall match rate needs care. Jev correctly classified 81.7% of archive false positives, while correctly identifying 45% of candidates. The confirmed-planet figure is stronger: 73.2% recall, with 73.4% precision among Jev's confirmed predictions. Those results describe one retrospective dataset and one prompt design. They do not establish how the model would perform on new telescope data or on a different collection of signals. Rabtsevich's professional profile points to a product-minded builder rather than an astronomy research group: it describes him as working on Veche, a grand-strategy game, and lists a degree in international relations from Belarusian State University. In his post, the Kepler exercise is presented as a capability test for Jev, a model that makes typed choices among supplied options rather than producing open-ended prose. TypeSafe AI's documentation itself advises users to move numeric comparisons into code and notes that Jev 1.13 can struggle with numerical precision. That design choice shaped the experiment. The second run translated measurements into named, human-readable fields and supplied explicit criteria for each label. The result suggests the model can make useful bounded judgments when the inputs and task are carefully framed. The first run and the prompt revision also show the distance between a headline recovery count and a benchmark that can support claims about general scientific discovery.