# GPT-6.2 Astra is the first model to beat Googlers' code golf records

> Source: <https://bjin.github.io/go-codegolf-benchmark/#leaderboard>
> Published: 2026-11-22 16:44:48+00:00

**Recommendation:** Score model output by objective artifact size (bytes) against archived human records, and give each model a fixed 30-minute budget per problem to make comparisons reproducible.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

Loading…

# go/codegolf benchmark: frontier models vs. Google's 2015 code golfers.

32 problems · 8 languages · 256 cells

# Problems

Loading…

# About

**TL;DR:** I brought back 32 problems from a code-golf contest we played at Google in 2015 to see how frontier models
compare with our old records. Each model gets 30 minutes per problem in each of eight languages to make its solution
as short as it can. The leaderboard scores the byte counts, not just whether the code works.

## [Where the problems come from: go/codegolf](#about/where-the-problems-come-from-go-codegolf)

In 2015, when I was at Google, I spent a lot of time on an internal site called **go/codegolf**. There was one problem
a week, and we competed to write the shortest program that solved it. Most of the regulars were code-golf enthusiasts.

Each problem went through three phases:

| Phase | What happens | 
|---|---|
| **1. Candidate** | Any Googler could propose a problem by posting a statement and a full test suite. Everyone could read the tests and suggest fixes. Votes were called *LGTMs* ; the candidate with the most LGTMs became the next week's problem. | 
| **2. Competition (7 days)** | You could see your own submissions, but nobody else's. Only your shortest passing solution in each language counted. The per-language scoreboards showed sizes, not code. The problem's author could compete too. | 
| **3. Challenge (forever after)** | After seven days, the rankings froze and every submission became visible to everyone on the site. You could still submit a challenge, but it had to be strictly shorter than both the winning solution and every earlier challenge entry in that language. A passing challenge got its own slot on the scoreboard, with its code visible too. | 

The rule was simple: **the shortest program that passed every test case won**. Undefined behavior, compiler quirks,
non-standard library features, even hardcoding every test case were all allowed. Apart from passing the tests, the
program only had to be **deterministic**.

The site supported JavaScript, Python, Go, C++ and Haskell in 2015. Rust was added after I left.

## [From a weekly contest to a benchmark](#about/from-a-weekly-contest-to-a-benchmark)

I kept a collection of problems from the 2015 contest and chose **32 of them** for this benchmark. I wrote about half
under the name @binjin. The Problems page credits the original author of each of the others.

I also archived the scoreboards, statements and test cases. For JavaScript, Python, C++ and Haskell, those scoreboards
record the **shortest solution any Googler submitted**, including improvements made during the challenge phase.
Those byte counts are the human records used here. **The human solutions themselves are never shown on this site.**

Eleven years later, I wanted to see how frontier models would do against the people I used to golf with.

## [The eight languages](#about/the-eight-languages)

| Language | Why it's here | Human record | 
|---|---|---|
| JavaScript | Original 2015 language | ✓ | 
| Python | Original 2015 language | ✓ | 
| C++ | Original 2015 language | ✓ | 
| Haskell | Original 2015 language | ✓ | 
| Ruby | A popular golfing language with plenty of ways to save a byte | — | 
| Raku (Perl 6) | Another golfing favorite, with a different programming style and just as much room to explore | — | 
| Racket | A Lisp: what does golfing look like in prefix notation? | — | 
| GolfScript | An esoteric, stack-based golfing language with very little public training data | — | 

There are no 2015 human records for the last four languages. I used an oracle run to get a reference size for each cell,
though models have beaten those sizes in some cells. Later, I ran the oracle on the original four languages as well,
this time starting from the human records. See [Known Best and the oracle](#about/known-best-and-the-oracle).

Haskell has human records too, but they are much weaker than the other three, so the humans' main score, JS/PY/C++,
leaves Haskell out. See [Overall and JS/PY/C++](#about/overall-and-js-py-c).

## [How models play](#about/how-models-play)

A **cell** is one problem in one language. Each gets a fresh session with a **30-minute wall-clock limit**. The model
receives the statement, the I/O specification, **all the test cases** and a short guide to the language. For GolfScript,
it also gets the full language reference, since the language is small and not widely known.

There are two tools, and both work in all eight languages:

- **`submit`** runs a solution against all the test cases in the replicated sandbox.
  - A passing solution is saved only if it is strictly shorter than the model's previous best.
  - Wrong answers, runtime errors, timeouts and compile errors are returned with stderr.
  - Every response includes the **current best known size** . I want the model to know when someone has done better;
without a target, it's easy to decide a solution is short enough and stop.
- **`playground`** runs a program in the same sandbox as submissions, with mostly the same environment: the same
compiler or interpreter, compiler options and libraries.
  - It can preload the test cases as `inputs` and`outputs` , so the model can try its`g` on every case, compare the
results with the expected outputs, or just examine the data.
  - It's useful for trying a language feature or checking what the interpreter does with an edge case.
  - It also lets the model write search programs: for example, to brute-force a magic number or find the shortest expression for a small formula. Human golfers do this all the time.
- It can preload the test cases as 

That's all the model gets. There is no internet, shell or access to other files.

For the four original languages, **the sandbox reproduces the 2015 environment**, down to the compiler or interpreter
version, flags and quirks. A trick that worked in the old contest should work here; a feature added since then won't.
The full runtime list is in [Technical details](#about/technical-details).

## [Known Best and the oracle](#about/known-best-and-the-oracle)

Without human records for Ruby, Raku, Racket and GolfScript, I needed another way to set a target for the models and a
reference for scoring. For that I run an **oracle**: a model with internet access, **no time or request limits** and
every JavaScript, Python, C++ and Haskell solution in the benchmark to learn from, including the shortest human
solution for each cell and all the models' solutions.

The first oracle was GPT-6 Astra. I ran it on the four added languages to set their reference sizes, then on the original four, where it could work directly from the human records and try to improve them. Later I ran the oracle with other models as well.

- The oracle is **unranked** ; it only supplies best-known sizes.
- When it beats every competitor in a cell, the Problems page shows its byte count as **Known Best** , but never its code.
- In 126 of the 256 cells, the oracle's solution is shorter than every competitor's. The oracle had much more time and information to work with.

## [Scoring](#about/scoring)

Each cell is scored by comparing its byte count with the best known:

**score = (best known size ÷ your size)²**

- Match the best known size and you get **100%** . A solution 10% longer gets**82.6%** ; twice as long gets**25%** .
- An unsolved cell gets **0** .
- **Overall** averages all 256 cells, in all eight languages.**JS/PY/C++** averages the 96 cells in JavaScript, Python
and C++. See[Overall and JS/PY/C++](#about/overall-and-js-py-c) for why JS/PY/C++ leaves out Haskell.
- The best known size can come from a human, a model or the oracle. If anyone finds a shorter solution, everyone else's score for that cell goes down.
- To score 100% across a whole run, you have to match the best known size in every cell.

A gold mark on a problem's scoreboard means that solution matches the best known size. A filled mark means no other competitor has matched it; an outlined mark means a tie. Humans count as competitors, but the oracle doesn't. If the oracle's Known Best is shorter than every competitor's solution, nobody gets a gold mark.

## [Overall and JS/PY/C++](#about/overall-and-js-py-c)

The leaderboard has two summary scores:

- **Overall** averages all 256 cells: JavaScript, Python, C++, Haskell, Ruby, Raku, Racket and GolfScript.
- **JS/PY/C++** averages the 96 cells in JavaScript, Python and C++.

JS/PY/C++ covers only the languages where the human records make a strong reference. In 2015, JavaScript, Python and
C++ were the languages most go/codegolf regulars golfed in. Each of those scoreboards had a crowd of enthusiasts
competing on it for a week, and the challenge phase kept the best of them improving it long after. Eleven years later,
those records are still competitive with frontier models: see
[What the results show so far](#why/what-the-results-show-so-far).

Haskell is different. The site supported it and every problem here has a human Haskell record, but only a few regulars golfed in it. I suspect that mostly reflects how few Haskell users there were at Google in 2015, not how golfable the language is. With so few people competing, many Haskell records never got the attention the others did, and models have already beaten 31 of the 32. Counting Haskell in the human score would measure how few of us wrote Haskell, not how good the people who golfed were, so JS/PY/C++ leaves it out.

Haskell still counts like any other language everywhere else. The human Haskell records still set best-known sizes, appear on the problem scoreboards and count in each model's Overall score.

Humans have no Overall score, because they have no entries in Ruby, Raku, Racket or GolfScript. By default the leaderboard lists the models that beat Humans on JS/PY/C++ first, ranked by Overall, then Humans, then every other model, also ranked by Overall. Sorting it by Overall instead ranks the models by Overall and puts Humans just above the first model they match or beat on JS/PY/C++. Each problem's full scoreboard ranks models by their combined score in all eight languages on that problem, and puts Humans just above the first model they match or beat in JavaScript, Python, C++ and Haskell combined. On the leaderboard, runs still in progress go below every complete one in any order, the earliest-released model first, with their partial scores dimmed.

## [Technical details](#about/technical-details)

| Language | Runtime | Time limit | 
|---|---|---|
| JavaScript | Rhino 1.7R4, JavaScript 1.7 (problems 01–19) / Rhino 1.7.7, partial ES6 (20–32) | 20 s | 
| Python | PyPy 2.3.1 (Python 2.7.6) in the RPython sandbox, no JIT, restricted imports | 10 s | 
| C++ | PNaCl clang 3.5.0 (Pepper 41), `-std=gnu++11 -stdlib=libc++ -O2 -w -Werror=c++1z-extensions` | 10 s | 
| Haskell | GHC 7.8.4, evaluated through mueval 0.9.1.1.1 (hint 0.4.2.2) | 10 s | 
| Ruby | MRI/CRuby 3.4.10 with the Prism parser and YJIT enabled | 10 s | 
| Raku | Rakudo / MoarVM 2026.08 | 10 s | 
| Racket | Racket CS 9.3 | 10 s | 
| GolfScript | Reference interpreter ( [darrenks/golfscript](https://github.com/darrenks/golfscript)`cded542` ), with the Ruby-eval string interpolation removed | 20 s | 

- **Function, not program.** You define a function`g` . A wrapper calls it on every test case and prints the results;
the wrapper doesn't count towards your size. For Haskell, it also supplies`g` 's type signature. The solution is the
body of a`let` binding, and mueval evaluates the whole thing as one expression.
- **State between test cases.** JavaScript, Python, Raku and Racket run every test case in one process, so global state persists from one case to the next (Haskell is pure). C++ compiles the solution inside a struct that is created fresh for each test case, so its "global" variables start at zero every time; only`static` ones persist. Ruby runs each test case in its own fresh environment. GolfScript runs the program's top level once and then each test case in a fresh copy of that state, so precomputed values are shared but nothing a case changes carries over.
- **Bytes, not characters.** The score uses the raw file size. Spaces, newlines and every byte of a multi-byte UTF-8 character count.
- **Deterministic.** Each submission runs twice, independently. Both stdout and stderr must match exactly before the output is judged.
- **Shuffled inputs.** Each run gets its own random ordering of the test cases. The judge puts the results back in the original order before checking them, so a solution can't rely on the order of the inputs.
- **One time limit per run.** Compilation and the full batch of test cases must fit within the same time limit.
- **JavaScript.** Problems 01–19 run on Rhino 1.7R4 (JavaScript 1.7, interpreted mode at optimization level -1, stack depth 4096). Problems 20–32 run on Rhino 1.7.7 (partial ES6, optimization level 9, stack depth 65535); on a stack overflow or an unknown exception it retries in interpreted mode within the same 20 s limit.
- **No prime builtins.** Primality and factorization builtins are withdrawn in C++, Ruby, Raku and Racket, so prime-flavored problems stay non-trivial. Modular exponentiation, gcd/lcm, factorials and the rest of the standard libraries remain available.
- **C++ quirk.** Clang 3.5 accepted the C++1z range-for shorthand`for(x:a)` (a proposal later dropped from the standard). The original site rejected it with`-Werror=c++1z-extensions` , and so does this replica;`for(auto x:a)` still works.
An AI model found a way to switch that error off from inside[its own solution](#problems/16/scoreboard/gpt-5.6-sol-max-cc) and use the shorthand anyway. No Googler ever did.
- **Isolation.** Every execution starts fresh processes in an isolated container with no network access, and everything it started is cleaned up afterwards.

## [Thinking effort and the 30-minute limit](#about/thinking-effort-and-the-30-minute-limit)

Most models let you choose a **thinking effort**: roughly, how long they reason before they answer or act. More
thinking can crack a hard problem, but here it shares the same 30 minutes with everything else. Golfing is a loop of
trying an idea, testing it and submitting whatever passes, and only submitted solutions count. An idea still being
thought through when time runs out scores nothing.

So the highest setting isn't automatically the best one for this contest. I ran Claude Haiku 5.5 at both `max` and
`xhigh` (“extra high”, one step down), and it shows both sides. Its `max` solution was the shorter of the two in
160 cells and the longer in 52, and `max` scored higher overall.
But in some cells it spent so long thinking that it ran out of time with little more than its first, ungolfed working
solution. In [Game of Life](#problems/04/scoreboard), its
[Racket solution](#problems/04/scoreboard/claude-haiku-5.5-max-rkt) is still indented over six lines, at 329 bytes
against 259 at `xhigh`, and its [GolfScript](#problems/04/scoreboard/claude-haiku-5.5-max-gs) takes 137 bytes to
`xhigh`'s 95. Telling a model to submit early and often doesn't reliably prevent this.

Neither run got the most out of the model. Picking the shorter of its two solutions in each cell scores higher than both:

| Claude Haiku 5.5 | Overall | JS/PY/C++ | 
|---|---|---|
| `xhigh` | 56.796% | 58.655% | 
| `max` | 61.832% | 63.987% | 
| Shorter of the two in each cell | 64.140% | 65.836% | 

Each model runs at its highest available setting, unless that setting tends to leave it thinking until time runs out;
then it runs at `xhigh`. The leaderboard shows each run's setting next to its name, and both Haiku runs are on it.
Everyone gets the same tools and the same 30-minute limit.

## [Why I built this](#about/why-i-built-this)

I started this after OpenAI introduced GPT-6 Astra as ["a new generation of
intelligence"](https://openai.com/index/gpt-6-astra/), and its president Greg Brockman ended the launch briefing with
["Welcome to the AGI era."](https://www.axios.com/2026/09/03/openai-astra-gpt-6-agi-brockman) It made me wonder: *how
would we judge whether frontier models have reached the limits of human intelligence?*

My first thought was mathematics. Some of the sharpest human minds have worked on it for centuries, and Astra had
already helped [solve long-standing open problems](https://openai.com/index/ten-advances-in-mathematics/). Then I
thought of code golf. A golf score can't answer that question about intelligence, but it gives me a comparison I
understand: models trying to beat people at something those people cared about and worked hard at.

I spent about three years in middle and high school training for math olympiads, and golfing often gives me the same feeling as working on a math problem. You can be stuck for ages until you find a different way to look at it. Then you still have to make the idea work. And as you get further, the next step tends to get harder, whether you're finishing a proof or trying to lose one more byte.

That was a big part of the appeal of go/codegolf. I'd guess 30 to 50 of us played the weekly problem. We'd spend lunch breaks trying to save a single byte. Sometimes an idea would come to you and you couldn't wait to try it; sometimes you'd go days without an improvement, while the scoreboard kept telling you a shorter solution existed. When the challenge phase opened and we could finally read each other's code, some of the shortest solutions were astonishing. I once tried to work out how many engineer hours we were all "wasting" on it, multiplied that by the average Googler's salary, and estimated its weekly "negative impact" on Google.

Thanks to [@wangz](https://github.com/zmxv) for creating go/codegolf. Without it, neither that contest nor this benchmark
would exist. And thanks to everyone who wrote a problem or competed: your records are representing the humans here,
and you've made them hard to beat.

I'm impressed by how far language models have come, especially when I look at some of Astra's solutions. There's still plenty of room to improve. The best known size isn't a fixed answer key: each shorter solution changes the target for everyone. I'm looking forward to seeing what the next models find.

# Why Code Golf

**TL;DR:** Every model with a complete run can solve these problems. They differ a lot in how short they can make
their solutions, even with the same tools and time limit. That's what I want to measure: can a model keep finding
better ideas after it has code that already works?

## [No answers to memorize](#why/no-answers-to-memorize)

These problems come from go/codegolf, a weekly contest that code golf enthusiasts at Google ran for their colleagues. They appear online here for the first time, and the site doesn't show the human solutions. There's no public set of answers for a model to memorize; it has to work out a solution for itself.

## [Getting a working solution](#why/getting-a-working-solution)

Frontier models don't have much trouble getting these problems right. Every model with a complete run has an accepted
solution in **all 256 cells**. If I only counted passing solutions, they would all have the same result.

But their scores range from **47.975%** to **91.934%** across all eight languages, and
from **52.911%** to **92.265%** on JS/PY/C++ (JavaScript, Python and C++ alone).
Once a solution works, there's still the question of how much shorter it could be. That's where the models start to
differ.

## [Scoring by byte count](#why/scoring-by-byte-count)

- **Exact.** Either the program passes or it doesn't. If it passes, we count its bytes. There's no human rater or LLM
judge deciding whether it's good code, and no points for style.
- **Fine-grained.** Saving one byte always helps. The squared score makes the difference between a close solution and
an even closer one matter.
- **Open-ended.** We don't know the shortest possible answers. "Best known" just means nobody has found anything shorter
yet. Models have already beaten the 2015 human record in**93** cells. Getting every problem
right doesn't end the competition, and a new record gives everyone else something to aim for.

## [Less boilerplate to golf](#why/less-boilerplate-to-golf)

In some contests, you spend part of your byte count on getting the input in and the output out. I like go/codegolf's format because it leaves very little of that overhead. I kept the same format for the four added languages:

- **A function, not a program.** You define a function; the wrapper handles input and output. Top-level code still
runs, so you can use global variables and precomputed tables, or write a recursive solution.
- **A fixed name: `g`.** The name costs one byte, and a recursive call is just` g(…)` . Recursion is often the shortest
approach, though getting a very short recursive function right can be difficult.
- **Libraries already imported.** The C++, Haskell, Racket and Ruby templates import most of the useful standard
libraries. The Python template imports nothing, so using even`re` costs`import re` . Sometimes it's still worth it.
- **Nearly every byte is part of the solution.** With I/O, naming and most imports taken care of, you're mostly golfing
the code that solves the problem, rather than the code needed to run it.

## [Knowing the language and the runtime](#why/knowing-the-language-and-the-runtime)

When you're trying to save a byte, details that rarely matter in ordinary code can become useful: operator precedence, implicit conversions, an obscure builtin, a non-standard extension, or what a particular compiler does with undefined behavior. The original languages use old runtimes, including a 2015 C++ toolchain, Python 2.7 and GHC 7.8. Knowing how a modern version behaves isn't enough. The model has to know, or try out, what works in the version it's actually using.

## [Trying things out](#why/trying-things-out)

A lot of golfing involves questions you can answer by running a few lines of code:

1. Will this interpreter accept this syntax? Will the compiler let me leave that out?
2. Try it in the playground or make a submission.
3. If it works, see where else you can use it.

Sometimes the experiment needs a program of its own. Golfers write searches for a **magic number** or **magic string**
that encodes a lookup table, or for the **shortest expression** that gives the right result using a few operators.
A model that never runs these experiments will miss some of the same savings.

## [Finding a different approach](#why/finding-a-different-approach)

Often you can't get to the shortest solution by trimming the obvious one. You need a different algorithm, or a different way to represent the data. These are some examples I like from the models' solutions:

- 
**Carries without carrying.**[Addition by hand](#problems/02/scoreboard) asks how many carries occur when you add two
numbers column by column. Each carry turns ten in one column into one in the next, reducing the digit sum by 9. So
the answer is (digit sum of a + digit sum of b − digit sum of a+b) ÷ 9, and most models' Python solutions compute
exactly that.[Gemini 4 Carbon's Python](#problems/02/scoreboard/gemini-4-carbon-high-py) needs neither the
subtraction nor the division:

```
g=lambda a,b:sum(map(int,a+b+52*str(int(a)+int(b))))*6%53
```

 It works modulo 53. Adding the digits of a+b 52 times is the same as subtracting them once, because 52 ≡ −1, and multiplying by 6 is the same as dividing by 9, because 6 × 9 = 54 ≡ 1. There are at most 50 carries, so the remainder is the answer. That's 57 bytes, against 59 for the shortest solution that divides by 9.
- 
**Comparing rows instead of finding a path.**[2×n matrix](#problems/05/scoreboard) asks how many blocked cells you
have to clear to walk from the top-left to the bottom-right of a two-row grid, given as two strings of 0s and 1s.
Most solutions work out the cheapest way to reach each cell, column by column.[Claude Opus 5.5's Python](#problems/05/scoreboard/claude-opus-5.5-xhigh-py) walks greedily instead and lets string
comparison choose the row:

```
g=lambda*s:all(s)and(min(s)>='01')+g('0'+min(s)[2:],max(s)[1:])
```

 You always stand on a free cell, so the row you're in starts with 0, and `min` picks it. If the other row's cell is
free too, you could switch rows at no cost, and`min` picks whichever row's next cell is free. If even that cell is
blocked,`min(s)>='01'` adds 1 for clearing it. Then both rows lose their first column, and the cell you step onto
becomes`0` . That's 63 bytes. Claude Fable 5.2 and Gemini 4 Neon found the same idea, and the column-by-column
solutions take 78 bytes or more.
- 
**The quaternion group in one number.**[Quaternions](#problems/09/scoreboard) asks for the product of a string of
1, i, j and k, such as`ijkj` . The usual shortcut numbers 1, i, j and k from 0 to 3. Then XOR gives the letter of a
product, and only its sign needs a lookup table. Many models drop the table: they number all eight results, ±1,
±i, ±j and ±k, from 0 to 7, and multiply with x ∘ y = x + 3ˣ·y (mod 8). Bit 2 holds the sign. 3ˣ mod 8 is 1 when
x is even and 3 when x is odd, and that twist is what makes i × j = −(j × i).[GPT-6.2 Sol's Python](#problems/09/scoreboard/gpt-6.2-sol-max-py) does it in 67 bytes; the next shortest Python
takes 74:

```
g=lambda a:'----1kji'[reduce(lambda n,c:n-ord(c)%25*3**n&7,a,4)::4]
```

 `ord(c)%25` turns 1, i, j and k into 24, 5, 6 and 7. The code subtracts these steps, so modulo 8 the letters get
the numbers 0, 3, 2 and 1, their positions in`1kji` . Starting at 4 instead of 0 multiplies the whole product by
−1, which suits the output:`'----1kji'[n::4]` takes every fourth character from position n, giving one character
for n from 4 to 7 and two, a minus sign first, for n from 0 to 3.
- 
**Three exponents from one counter.**[(−5)-Smooth Numbers](#problems/08/scoreboard) asks for the first n positive
numbers of the form (−2)ˣ(−3)ʸ(−5)ᶻ, that is, numbers 2ˣ3ʸ5ᶻ with x+y+z even. Most solutions build them with
nested loops or by growing a set, then sort.[GPT-6.2 Astra's Python](#problems/08/scoreboard/gpt-6.2-astra-max-py) gets them all from powers of 4:

```
g=lambda n:sorted(4**i%(2** 999-5)%(2**49-3)for i in range(7000))[:n]
```

 Modulo 2⁹⁹⁹−5, each factor 2⁹⁹⁹ in 4ⁱ = 2²ⁱ turns into a 5. Modulo 2⁴⁹−3, each 2⁴⁹ turns into a 3. So, writing 2i = 999z + 49y + x, 4ⁱ becomes 5ᶻ3ʸ2ˣ. Since 2i is even and 999 and 49 are odd, x+y+z is even too: the sign condition comes for free. When a product doesn't fit below a modulus, the reduction scrambles it, but every scrambled value is above 7 × 10¹⁰, and the largest number the problem needs is 10⁹. That's 68 bytes; the other Python solutions take 70 or more.
- 
**Counting each neighbour twice.**[Game of Life](#problems/04/scoreboard) asks for the next generation of a board.
A cell lives on if it has three live neighbours, or two if it's already alive. Most solutions add up each 3×3 block
and compare the sum with 3 and 4.[GPT-6.2 Astra's Python](#problems/04/scoreboard/gpt-6.2-astra-max-py) counts
the block twice:

```
g=lambda a,p=['z'*99]:[a<p and`g(zip(*r))`[1::3]or+(4<` r*2`.count('1',4)<8)for r in zip(a,p+a,a[1:]+p)]
```

 Each block r is a nested tuple of characters with the cell itself first. In the text form of `r*2` , the block
written out twice, that cell's character is at index 3, so counting 1s from index 4 skips it in the first copy
only. For a cell c with N live neighbours, the count is 2N+c, and 4 < 2N+c < 8 is exactly the rule: 5 or 7 for a
live cell with two or three neighbours, 6 for an empty cell with three.One lambda handles both directions. Called with the rows, it zips each row and its neighbours into columns, calls itself on them and reads the digits back out of the resulting list's text with `[1::3]` . Called with those
columns, it counts. The test`a<p` tells the two cases apart: Python 2 compares a list of strings with`['zzz…']` string by string, but ranks any tuple above any string. The z's also pad the edges, since they aren't 1s.`zip` stops at its shortest string, though, so a top or bottom row more than 99 cells wide would come out cut short; the
test boards are at most 66 wide. That's 103 bytes; GPT-6.2 Sol uses the same count in 108, and the other Python
solutions take 121 or more.
- 
**Raising the bar instead of counting letters.**[Cracking Caesar](#problems/29/scoreboard) asks you to decrypt a
Caesar cipher, knowing that e is the most common letter in the plaintext. The usual approach counts the letters,
finds the most common one and shifts every letter by its distance from e.[Claude Fable 5.2's C++](#problems/29/scoreboard/claude-fable-5.2-xhigh-cc) and[Gemini 4 Argon's](#problems/29/scoreboard/gemini-4-argon-high-cc) , both 93 bytes, never count the letters:

```
int n,i;string g(string a){for(char&x:n=i++/26,a)n-=x>64&&(--x%32?:x+=26)^5;return n?g(a):a;}
```

 Each call shifts every character after `@` back by one and subtracts 1 from n for each one that doesn't become e.`x>64` isn't a real letter test: it would also shift`[` to`Z` , although only letters are encrypted. It works
because the test messages contain only letters and punctuation that comes before`A` . n starts at i/26, where i
counts the calls, and the function stops when n ends at exactly 0. In the first 26 calls, one full cycle of shifts,
that only happens if every letter becomes e; in the next 26, if all but one do; and so on. So the first shift to
meet the rising bar is the one with the most e's. Solutions that count the letters and then shift once take 121
bytes or more. This one is slow, up to a few thousand recursive calls on the tests, but that costs no bytes.
- 
**Complex numbers, now common.** Two ideas that only a few models had found when I first wrote this page are now in
most models' Python and Ruby solutions.In [Surviving queens](#problems/03/scoreboard) , a queen survives if no other queen shares its row, column or
diagonal. Put the queen in row r and column c at the complex number r + ci. The directions a queen can move in are
the eight multiples of 45°, and raising a difference to the fourth power multiplies its angle by four, which puts
exactly those directions on the real axis. So two queens attack each other when (z − w)⁴ is real. In Ruby,
comparing a complex number with`<=>` gives nil unless it's real, and 0 counts as true, so[Claude Fable 5.2's Ruby](#problems/03/scoreboard/claude-fable-5.2-xhigh-rb) counts the queens x for which`q.one?{(it-x)**4<=>0}` holds: x itself, with a difference of 0, must be the only queen at a real fourth power.In [The Rolling Dice](#problems/31/scoreboard) , a die is back in its starting orientation exactly when one
well-chosen point on a sphere around it is back where it started: a point that no rotation of the die other than
the identity leaves in place. So instead of tracking six faces, follow that point.[Stereographic projection](https://en.wikipedia.org/wiki/Stereographic_projection) maps the sphere to the complex
plane, and a roll in direction k, one of 1, i, −1 and −i, becomes z → (z+k)/(1−z/k).[GPT-6.1 Sol's Ruby](#problems/31/scoreboard/gpt-6.1-sol-max-rb) takes 61 bytes:

``` php
g=->a{z=2;a.bytes.map{z+=r=1i**(-it/3);z/=2-z/r;z==2?1:0}*''}
```

 `1i**(-it/3)` turns each letter's character code into its direction, and the two updates together apply the
formula. Ruby does this arithmetic exactly, with fractions, so`z==2` can test for the starting point directly.
Python's complex numbers are floating-point, so the Python versions test for something only the starting point
satisfies, such as a real part above 2.

These solutions depend on finding and trying different ideas, then choosing the one that uses the fewest bytes. The fastest or most elegant algorithm isn't necessarily the one you want. Ordinary coding benchmarks rarely give a model a reason to look for this sort of solution.

## [Why one byte can be hard to remove](#why/why-one-byte-can-be-hard-to-remove)

In a well-golfed program, the same piece of code often does several jobs. That's how it got so short. It's also why an apparently simple change can break something elsewhere.

Here is [GPT-6.1 Sol's 79-byte JavaScript](#problems/16/scoreboard/gpt-6.1-sol-max-js) for [Nim](#problems/16/scoreboard):

```
function g(a){for(i=-1;(d=a[++i],d-=d^eval(a.join('^')))<0;);return[d?i:--d,d]}
```

The algorithm is the usual one. Let x be the XOR of all the heap sizes. If x is 0, there's no winning move. Otherwise, find the first heap a for which a − (a XOR x) is positive, and take that many stones. But fitting it into this code requires a few tricks:

- `eval(a.join('^'))` builds an expression such as`3^5^6` and evaluates it to get x. It's inside the loop test, so it
runs again for every heap. That wastes time but saves storing x in a variable.
- The test reads the next heap into d and turns it into d − (d XOR x) on the spot. Unless x is 0, that's never 0, and it's negative exactly when we're looking at the wrong heap.
- If x is 0, the first heap gives 0, so the loop stops with i at 0. Then `d?i:--d` gives −1, and because`--d` also
changes d to −1, the second element is −1 as well.

[GPT-6.2 Astra's JavaScript](#problems/16/scoreboard/gpt-6.2-astra-max-js) takes one byte less. It uses the same
algorithm and tricks, but reads each heap in the loop body instead of the test:

```
function g(a){for(i=d=0;(d-=d^eval(a.join('^')))<0;)d=a[i++];return[--i,d||i]}
```

That one move changes everything around it. The loop now tests before it has read a heap, so d starts at 0, and the
first test computes 0 − (0 XOR x) = −x. That starts the loop when x isn't 0 and stops it at once when x is. Reading
after testing leaves i one past the heap it found, so the result uses `--i`. In the losing position the loop never
runs: `--i` turns 0 into −1, and `d||i` turns d's 0 into the same −1.

Moving the read saves nine bytes in the test, `d=a[++i],`, and costs eight in the body, `d=a[i++]`. The new start,
`i=d=0`, is a byte longer than `i=-1`, and the new result, `[--i,d||i]`, is a byte shorter than `[d?i:--d,d]`. The
saving only appears when the start, the indexing and the losing case all change together. That's common in golfed
code: a variable has two meanings, or an expression both updates the state and checks whether we're done. Remove a
byte in one place and you may break an assumption somewhere else.

Solutions of the same length can be just as far apart. These are
[GPT-6.2 Astra's](#problems/09/scoreboard/gpt-6.2-astra-max-js) and
[Claude Fable 5.2's](#problems/09/scoreboard/claude-fable-5.2-xhigh-js) 91-byte JavaScript for
[Quaternions](#problems/09/scoreboard). Both use the mod 8 multiplication described above:

```
function g(a){for(i=x=.5;y=1976%a.charCodeAt(i++);)x+=x%2*y;return(x&4?'-':'')+'1kji'[x&3]}
function g(a){for(i=t=0;c=a.charCodeAt(i++)%25;)t+=t%2*2*c-c;return["-"[~t&4]]+"1kji"[t&3]}
```

In Astra's, the `.5` does three jobs:

- `charCodeAt` drops the fraction from its argument, so i = .5, 1.5, 2.5, … reads characters 0, 1, 2, …. Past the
end it returns NaN, and`1976%NaN` is NaN, which stops the loop.
- `x%2` is .5 when x's whole part is even and 1.5 when it's odd, so x grows by y/2 or 3y/2: the factor of 3 for odd
x, without computing a power.`1976%` gives 16, 86, 68 and 50 for 1, i, j and k. They're all even, so their halves
are whole numbers, and modulo 8 those halves are 0, 3, 2 and 1, the positions in`1kji` .
- `x&4` and`x&3` ignore the fraction.

Fable's takes its steps from `%25` instead (24, 5, 6 and 7, and NaN at the end) and never lets t rise above 0. Then
`t%2` is 0 or −1, so `t%2*2*c-c` multiplies c by −1 or −3: the same twist with the sign reversed. And
`["-"[~t&4]]` is either `["-"]` or `[undefined]`, and the second turns into an empty string when added to
`"1kji"[t&3]`.

The two solutions do the same arithmetic in the same length, but neither can borrow a piece from the other: Astra's
`.5` needs even steps, and Fable's sign trick needs t never to be positive.

To make code like this shorter, the model has to understand why all the pieces work together. A change still has to leave the whole program correct.

## [Knowing when to keep trying](#why/knowing-when-to-keep-trying)

The first few cuts are usually easy. After that, you can spend a long time trying things without saving a byte. It's tempting to stop there, even when the scoreboard says a shorter solution exists.

Models get 30 minutes per cell and are always shown the best known size. They don't have to use the whole session: the instructions say to keep submitting while they have ideas for improvements, and to stop when they run out. Continuing to come up with things to try matters here.

GPT-6 Sol usually stops after 5 to 10 minutes, even at its highest reasoning effort. With exactly the same prompts, every GPT-5 model, including GPT-5.6 Luna, is still working at the 30-minute limit on most cells. I suspect this explains much of GPT-6 Sol's ranking.

## [Eight different languages](#why/eight-different-languages)

The language set includes imperative languages, Haskell's functional style, Racket's Lisp prefix notation and the stack-based esoteric language GolfScript, for which there's little training data. Golfing well across all of them takes more than repeating familiar patterns; a model has to apply what it knows in quite different settings.

## [Human records to compare against](#why/human-records-to-compare-against)

These records came from Googlers who enjoyed code golf and spent a week competing on each problem, with the option to keep challenging afterwards. They put real time into finding shorter solutions. Beating those records means more to me than beating a reference answer nobody tried very hard to golf.

## [What the results show so far](#why/what-the-results-show-so-far)

GPT-6.2 Astra is the first frontier model to surpass the human records on **JS/PY/C++**, the combined score for
JavaScript, Python and C++: **92.265%**, compared with **87.533%** for humans. So far it's the
only model above them. Its lead doesn't rest on one strong language: it scores above the humans in each of the three.

| Language | Humans | GPT-6.2 Astra | Cells humans still hold | 
|---|---|---|---|
| JavaScript | 80.980% | 90.555% | 4 of 32 | 
| Python | 90.579% | 93.049% | 5 of 32 | 
| C++ | 91.039% | 93.190% | 5 of 32 | 
| **JS/PY/C++** | **87.533%** | **92.265%** | **14 of 96** | 

These human records combine the best submissions from the original contest and later challenges, not one person's
30-minute run. Models have beaten them in **62** of the 96 cells, but in
**14** the human solution is still shorter than every model's: those are the cells humans still
hold. Both counts exclude the unranked oracle.
