GPT-6.2 Astra is the first model to beat Googlers' code golf records A benchmark built from 32 code-golf problems originally played on Google's internal go/codegolf site in 2015 reports that GPT-6.2 Astra is the first model to beat the Googlers' shortest-solution records, according to the benchmark's author, who competed under the handle @binjin. Each model gets 30 minutes per problem across eight languages (JavaScript, Python, C++, Haskell, Ruby, Raku, Racket and GolfScript), and the leaderboard scores byte counts rather than correctness alone. Human records exist only for the four original 2015 languages, with an oracle run supplying reference sizes for the other four. Loading… go/codegolf benchmark: frontier models vs. Google's 2015 code golfers. 32 problems · 8 languages · 256 cells Problems Loading… About TL;DR: I brought back 32 problems from a code-golf contest we played at Google in 2015 to see how frontier models compare with our old records. Each model gets 30 minutes per problem in each of eight languages to make its solution as short as it can. The leaderboard scores the byte counts, not just whether the code works. Where the problems come from: go/codegolf about/where-the-problems-come-from-go-codegolf In 2015, when I was at Google, I spent a lot of time on an internal site called go/codegolf . There was one problem a week, and we competed to write the shortest program that solved it. Most of the regulars were code-golf enthusiasts. Each problem went through three phases: | Phase | What happens | |---|---| | 1. Candidate | Any Googler could propose a problem by posting a statement and a full test suite. Everyone could read the tests and suggest fixes. Votes were called LGTMs ; the candidate with the most LGTMs became the next week's problem. | | 2. Competition 7 days | You could see your own submissions, but nobody else's. Only your shortest passing solution in each language counted. The per-language scoreboards showed sizes, not code. The problem's author could compete too. | | 3. Challenge forever after | After seven days, the rankings froze and every submission became visible to everyone on the site. You could still submit a challenge, but it had to be strictly shorter than both the winning solution and every earlier challenge entry in that language. A passing challenge got its own slot on the scoreboard, with its code visible too. | The rule was simple: the shortest program that passed every test case won . Undefined behavior, compiler quirks, non-standard library features, even hardcoding every test case were all allowed. Apart from passing the tests, the program only had to be deterministic . The site supported JavaScript, Python, Go, C++ and Haskell in 2015. Rust was added after I left. From a weekly contest to a benchmark about/from-a-weekly-contest-to-a-benchmark I kept a collection of problems from the 2015 contest and chose 32 of them for this benchmark. I wrote about half under the name @binjin. The Problems page credits the original author of each of the others. I also archived the scoreboards, statements and test cases. For JavaScript, Python, C++ and Haskell, those scoreboards record the shortest solution any Googler submitted , including improvements made during the challenge phase. Those byte counts are the human records used here. The human solutions themselves are never shown on this site. Eleven years later, I wanted to see how frontier models would do against the people I used to golf with. The eight languages about/the-eight-languages | Language | Why it's here | Human record | |---|---|---| | JavaScript | Original 2015 language | ✓ | | Python | Original 2015 language | ✓ | | C++ | Original 2015 language | ✓ | | Haskell | Original 2015 language | ✓ | | Ruby | A popular golfing language with plenty of ways to save a byte | — | | Raku Perl 6 | Another golfing favorite, with a different programming style and just as much room to explore | — | | Racket | A Lisp: what does golfing look like in prefix notation? | — | | GolfScript | An esoteric, stack-based golfing language with very little public training data | — | There are no 2015 human records for the last four languages. I used an oracle run to get a reference size for each cell, though models have beaten those sizes in some cells. Later, I ran the oracle on the original four languages as well, this time starting from the human records. See Known Best and the oracle about/known-best-and-the-oracle . Haskell has human records too, but they are much weaker than the other three, so the humans' main score, JS/PY/C++, leaves Haskell out. See Overall and JS/PY/C++ about/overall-and-js-py-c . How models play about/how-models-play A cell is one problem in one language. Each gets a fresh session with a 30-minute wall-clock limit . The model receives the statement, the I/O specification, all the test cases and a short guide to the language. For GolfScript, it also gets the full language reference, since the language is small and not widely known. There are two tools, and both work in all eight languages: - submit runs a solution against all the test cases in the replicated sandbox. - A passing solution is saved only if it is strictly shorter than the model's previous best. - Wrong answers, runtime errors, timeouts and compile errors are returned with stderr. - Every response includes the current best known size . I want the model to know when someone has done better; without a target, it's easy to decide a solution is short enough and stop. - playground runs a program in the same sandbox as submissions, with mostly the same environment: the same compiler or interpreter, compiler options and libraries. - It can preload the test cases as inputs and outputs , so the model can try its g on every case, compare the results with the expected outputs, or just examine the data. - It's useful for trying a language feature or checking what the interpreter does with an edge case. - It also lets the model write search programs: for example, to brute-force a magic number or find the shortest expression for a small formula. Human golfers do this all the time. - It can preload the test cases as That's all the model gets. There is no internet, shell or access to other files. For the four original languages, the sandbox reproduces the 2015 environment , down to the compiler or interpreter version, flags and quirks. A trick that worked in the old contest should work here; a feature added since then won't. The full runtime list is in Technical details about/technical-details . Known Best and the oracle about/known-best-and-the-oracle Without human records for Ruby, Raku, Racket and GolfScript, I needed another way to set a target for the models and a reference for scoring. For that I run an oracle : a model with internet access, no time or request limits and every JavaScript, Python, C++ and Haskell solution in the benchmark to learn from, including the shortest human solution for each cell and all the models' solutions. The first oracle was GPT-6 Astra. I ran it on the four added languages to set their reference sizes, then on the original four, where it could work directly from the human records and try to improve them. Later I ran the oracle with other models as well. - The oracle is unranked ; it only supplies best-known sizes. - When it beats every competitor in a cell, the Problems page shows its byte count as Known Best , but never its code. - In 126 of the 256 cells, the oracle's solution is shorter than every competitor's. The oracle had much more time and information to work with. Scoring about/scoring Each cell is scored by comparing its byte count with the best known: score = best known size ÷ your size ² - Match the best known size and you get 100% . A solution 10% longer gets 82.6% ; twice as long gets 25% . - An unsolved cell gets 0 . - Overall averages all 256 cells, in all eight languages. JS/PY/C++ averages the 96 cells in JavaScript, Python and C++. See Overall and JS/PY/C++ about/overall-and-js-py-c for why JS/PY/C++ leaves out Haskell. - The best known size can come from a human, a model or the oracle. If anyone finds a shorter solution, everyone else's score for that cell goes down. - To score 100% across a whole run, you have to match the best known size in every cell. A gold mark on a problem's scoreboard means that solution matches the best known size. A filled mark means no other competitor has matched it; an outlined mark means a tie. Humans count as competitors, but the oracle doesn't. If the oracle's Known Best is shorter than every competitor's solution, nobody gets a gold mark. Overall and JS/PY/C++ about/overall-and-js-py-c The leaderboard has two summary scores: - Overall averages all 256 cells: JavaScript, Python, C++, Haskell, Ruby, Raku, Racket and GolfScript. - JS/PY/C++ averages the 96 cells in JavaScript, Python and C++. JS/PY/C++ covers only the languages where the human records make a strong reference. In 2015, JavaScript, Python and C++ were the languages most go/codegolf regulars golfed in. Each of those scoreboards had a crowd of enthusiasts competing on it for a week, and the challenge phase kept the best of them improving it long after. Eleven years later, those records are still competitive with frontier models: see What the results show so far why/what-the-results-show-so-far . Haskell is different. The site supported it and every problem here has a human Haskell record, but only a few regulars golfed in it. I suspect that mostly reflects how few Haskell users there were at Google in 2015, not how golfable the language is. With so few people competing, many Haskell records never got the attention the others did, and models have already beaten 31 of the 32. Counting Haskell in the human score would measure how few of us wrote Haskell, not how good the people who golfed were, so JS/PY/C++ leaves it out. Haskell still counts like any other language everywhere else. The human Haskell records still set best-known sizes, appear on the problem scoreboards and count in each model's Overall score. Humans have no Overall score, because they have no entries in Ruby, Raku, Racket or GolfScript. By default the leaderboard lists the models that beat Humans on JS/PY/C++ first, ranked by Overall, then Humans, then every other model, also ranked by Overall. Sorting it by Overall instead ranks the models by Overall and puts Humans just above the first model they match or beat on JS/PY/C++. Each problem's full scoreboard ranks models by their combined score in all eight languages on that problem, and puts Humans just above the first model they match or beat in JavaScript, Python, C++ and Haskell combined. On the leaderboard, runs still in progress go below every complete one in any order, the earliest-released model first, with their partial scores dimmed. Technical details about/technical-details | Language | Runtime | Time limit | |---|---|---| | JavaScript | Rhino 1.7R4, JavaScript 1.7 problems 01–19 / Rhino 1.7.7, partial ES6 20–32 | 20 s | | Python | PyPy 2.3.1 Python 2.7.6 in the RPython sandbox, no JIT, restricted imports | 10 s | | C++ | PNaCl clang 3.5.0 Pepper 41 , -std=gnu++11 -stdlib=libc++ -O2 -w -Werror=c++1z-extensions | 10 s | | Haskell | GHC 7.8.4, evaluated through mueval 0.9.1.1.1 hint 0.4.2.2 | 10 s | | Ruby | MRI/CRuby 3.4.10 with the Prism parser and YJIT enabled | 10 s | | Raku | Rakudo / MoarVM 2026.08 | 10 s | | Racket | Racket CS 9.3 | 10 s | | GolfScript | Reference interpreter darrenks/golfscript https://github.com/darrenks/golfscript cded542 , with the Ruby-eval string interpolation removed | 20 s | - Function, not program. You define a function g . A wrapper calls it on every test case and prints the results; the wrapper doesn't count towards your size. For Haskell, it also supplies g 's type signature. The solution is the body of a let binding, and mueval evaluates the whole thing as one expression. - State between test cases. JavaScript, Python, Raku and Racket run every test case in one process, so global state persists from one case to the next Haskell is pure . C++ compiles the solution inside a struct that is created fresh for each test case, so its "global" variables start at zero every time; only static ones persist. Ruby runs each test case in its own fresh environment. GolfScript runs the program's top level once and then each test case in a fresh copy of that state, so precomputed values are shared but nothing a case changes carries over. - Bytes, not characters. The score uses the raw file size. Spaces, newlines and every byte of a multi-byte UTF-8 character count. - Deterministic. Each submission runs twice, independently. Both stdout and stderr must match exactly before the output is judged. - Shuffled inputs. Each run gets its own random ordering of the test cases. The judge puts the results back in the original order before checking them, so a solution can't rely on the order of the inputs. - One time limit per run. Compilation and the full batch of test cases must fit within the same time limit. - JavaScript. Problems 01–19 run on Rhino 1.7R4 JavaScript 1.7, interpreted mode at optimization level -1, stack depth 4096 . Problems 20–32 run on Rhino 1.7.7 partial ES6, optimization level 9, stack depth 65535 ; on a stack overflow or an unknown exception it retries in interpreted mode within the same 20 s limit. - No prime builtins. Primality and factorization builtins are withdrawn in C++, Ruby, Raku and Racket, so prime-flavored problems stay non-trivial. Modular exponentiation, gcd/lcm, factorials and the rest of the standard libraries remain available. - C++ quirk. Clang 3.5 accepted the C++1z range-for shorthand for x:a a proposal later dropped from the standard . The original site rejected it with -Werror=c++1z-extensions , and so does this replica; for auto x:a still works. An AI model found a way to switch that error off from inside its own solution problems/16/scoreboard/gpt-5.6-sol-max-cc and use the shorthand anyway. No Googler ever did. - Isolation. Every execution starts fresh processes in an isolated container with no network access, and everything it started is cleaned up afterwards. Thinking effort and the 30-minute limit about/thinking-effort-and-the-30-minute-limit Most models let you choose a thinking effort : roughly, how long they reason before they answer or act. More thinking can crack a hard problem, but here it shares the same 30 minutes with everything else. Golfing is a loop of trying an idea, testing it and submitting whatever passes, and only submitted solutions count. An idea still being thought through when time runs out scores nothing. So the highest setting isn't automatically the best one for this contest. I ran Claude Haiku 5.5 at both max and xhigh “extra high”, one step down , and it shows both sides. Its max solution was the shorter of the two in 160 cells and the longer in 52, and max scored higher overall. But in some cells it spent so long thinking that it ran out of time with little more than its first, ungolfed working solution. In Game of Life problems/04/scoreboard , its Racket solution problems/04/scoreboard/claude-haiku-5.5-max-rkt is still indented over six lines, at 329 bytes against 259 at xhigh , and its GolfScript problems/04/scoreboard/claude-haiku-5.5-max-gs takes 137 bytes to xhigh 's 95. Telling a model to submit early and often doesn't reliably prevent this. Neither run got the most out of the model. Picking the shorter of its two solutions in each cell scores higher than both: | Claude Haiku 5.5 | Overall | JS/PY/C++ | |---|---|---| | xhigh | 56.796% | 58.655% | | max | 61.832% | 63.987% | | Shorter of the two in each cell | 64.140% | 65.836% | Each model runs at its highest available setting, unless that setting tends to leave it thinking until time runs out; then it runs at xhigh . The leaderboard shows each run's setting next to its name, and both Haiku runs are on it. Everyone gets the same tools and the same 30-minute limit. Why I built this about/why-i-built-this I started this after OpenAI introduced GPT-6 Astra as "a new generation of intelligence" https://openai.com/index/gpt-6-astra/ , and its president Greg Brockman ended the launch briefing with "Welcome to the AGI era." https://www.axios.com/2026/09/03/openai-astra-gpt-6-agi-brockman It made me wonder: how would we judge whether frontier models have reached the limits of human intelligence? My first thought was mathematics. Some of the sharpest human minds have worked on it for centuries, and Astra had already helped solve long-standing open problems https://openai.com/index/ten-advances-in-mathematics/ . Then I thought of code golf. A golf score can't answer that question about intelligence, but it gives me a comparison I understand: models trying to beat people at something those people cared about and worked hard at. I spent about three years in middle and high school training for math olympiads, and golfing often gives me the same feeling as working on a math problem. You can be stuck for ages until you find a different way to look at it. Then you still have to make the idea work. And as you get further, the next step tends to get harder, whether you're finishing a proof or trying to lose one more byte. That was a big part of the appeal of go/codegolf. I'd guess 30 to 50 of us played the weekly problem. We'd spend lunch breaks trying to save a single byte. Sometimes an idea would come to you and you couldn't wait to try it; sometimes you'd go days without an improvement, while the scoreboard kept telling you a shorter solution existed. When the challenge phase opened and we could finally read each other's code, some of the shortest solutions were astonishing. I once tried to work out how many engineer hours we were all "wasting" on it, multiplied that by the average Googler's salary, and estimated its weekly "negative impact" on Google. Thanks to @wangz https://github.com/zmxv for creating go/codegolf. Without it, neither that contest nor this benchmark would exist. And thanks to everyone who wrote a problem or competed: your records are representing the humans here, and you've made them hard to beat. I'm impressed by how far language models have come, especially when I look at some of Astra's solutions. There's still plenty of room to improve. The best known size isn't a fixed answer key: each shorter solution changes the target for everyone. I'm looking forward to seeing what the next models find. Why Code Golf TL;DR: Every model with a complete run can solve these problems. They differ a lot in how short they can make their solutions, even with the same tools and time limit. That's what I want to measure: can a model keep finding better ideas after it has code that already works? No answers to memorize why/no-answers-to-memorize These problems come from go/codegolf, a weekly contest that code golf enthusiasts at Google ran for their colleagues. They appear online here for the first time, and the site doesn't show the human solutions. There's no public set of answers for a model to memorize; it has to work out a solution for itself. Getting a working solution why/getting-a-working-solution Frontier models don't have much trouble getting these problems right. Every model with a complete run has an accepted solution in all 256 cells . If I only counted passing solutions, they would all have the same result. But their scores range from 47.975% to 91.934% across all eight languages, and from 52.911% to 92.265% on JS/PY/C++ JavaScript, Python and C++ alone . Once a solution works, there's still the question of how much shorter it could be. That's where the models start to differ. Scoring by byte count why/scoring-by-byte-count - Exact. Either the program passes or it doesn't. If it passes, we count its bytes. There's no human rater or LLM judge deciding whether it's good code, and no points for style. - Fine-grained. Saving one byte always helps. The squared score makes the difference between a close solution and an even closer one matter. - Open-ended. We don't know the shortest possible answers. "Best known" just means nobody has found anything shorter yet. Models have already beaten the 2015 human record in 93 cells. Getting every problem right doesn't end the competition, and a new record gives everyone else something to aim for. Less boilerplate to golf why/less-boilerplate-to-golf In some contests, you spend part of your byte count on getting the input in and the output out. I like go/codegolf's format because it leaves very little of that overhead. I kept the same format for the four added languages: - A function, not a program. You define a function; the wrapper handles input and output. Top-level code still runs, so you can use global variables and precomputed tables, or write a recursive solution. - A fixed name: g . The name costs one byte, and a recursive call is just g … . Recursion is often the shortest approach, though getting a very short recursive function right can be difficult. - Libraries already imported. The C++, Haskell, Racket and Ruby templates import most of the useful standard libraries. The Python template imports nothing, so using even re costs import re . Sometimes it's still worth it. - Nearly every byte is part of the solution. With I/O, naming and most imports taken care of, you're mostly golfing the code that solves the problem, rather than the code needed to run it. Knowing the language and the runtime why/knowing-the-language-and-the-runtime When you're trying to save a byte, details that rarely matter in ordinary code can become useful: operator precedence, implicit conversions, an obscure builtin, a non-standard extension, or what a particular compiler does with undefined behavior. The original languages use old runtimes, including a 2015 C++ toolchain, Python 2.7 and GHC 7.8. Knowing how a modern version behaves isn't enough. The model has to know, or try out, what works in the version it's actually using. Trying things out why/trying-things-out A lot of golfing involves questions you can answer by running a few lines of code: 1. Will this interpreter accept this syntax? Will the compiler let me leave that out? 2. Try it in the playground or make a submission. 3. If it works, see where else you can use it. Sometimes the experiment needs a program of its own. Golfers write searches for a magic number or magic string that encodes a lookup table, or for the shortest expression that gives the right result using a few operators. A model that never runs these experiments will miss some of the same savings. Finding a different approach why/finding-a-different-approach Often you can't get to the shortest solution by trimming the obvious one. You need a different algorithm, or a different way to represent the data. These are some examples I like from the models' solutions: - Carries without carrying. Addition by hand problems/02/scoreboard asks how many carries occur when you add two numbers column by column. Each carry turns ten in one column into one in the next, reducing the digit sum by 9. So the answer is digit sum of a + digit sum of b − digit sum of a+b ÷ 9, and most models' Python solutions compute exactly that. Gemini 4 Carbon's Python problems/02/scoreboard/gemini-4-carbon-high-py needs neither the subtraction nor the division: g=lambda a,b:sum map int,a+b+52 str int a +int b 6%53 It works modulo 53. Adding the digits of a+b 52 times is the same as subtracting them once, because 52 ≡ −1, and multiplying by 6 is the same as dividing by 9, because 6 × 9 = 54 ≡ 1. There are at most 50 carries, so the remainder is the answer. That's 57 bytes, against 59 for the shortest solution that divides by 9. - Comparing rows instead of finding a path. 2×n matrix problems/05/scoreboard asks how many blocked cells you have to clear to walk from the top-left to the bottom-right of a two-row grid, given as two strings of 0s and 1s. Most solutions work out the cheapest way to reach each cell, column by column. Claude Opus 5.5's Python problems/05/scoreboard/claude-opus-5.5-xhigh-py walks greedily instead and lets string comparison choose the row: g=lambda s:all s and min s ='01' +g '0'+min s 2: ,max s 1: You always stand on a free cell, so the row you're in starts with 0, and min picks it. If the other row's cell is free too, you could switch rows at no cost, and min picks whichever row's next cell is free. If even that cell is blocked, min s ='01' adds 1 for clearing it. Then both rows lose their first column, and the cell you step onto becomes 0 . That's 63 bytes. Claude Fable 5.2 and Gemini 4 Neon found the same idea, and the column-by-column solutions take 78 bytes or more. - The quaternion group in one number. Quaternions problems/09/scoreboard asks for the product of a string of 1, i, j and k, such as ijkj . The usual shortcut numbers 1, i, j and k from 0 to 3. Then XOR gives the letter of a product, and only its sign needs a lookup table. Many models drop the table: they number all eight results, ±1, ±i, ±j and ±k, from 0 to 7, and multiply with x ∘ y = x + 3ˣ·y mod 8 . Bit 2 holds the sign. 3ˣ mod 8 is 1 when x is even and 3 when x is odd, and that twist is what makes i × j = − j × i . GPT-6.2 Sol's Python problems/09/scoreboard/gpt-6.2-sol-max-py does it in 67 bytes; the next shortest Python takes 74: g=lambda a:'----1kji' reduce lambda n,c:n-ord c %25 3 n&7,a,4 ::4 ord c %25 turns 1, i, j and k into 24, 5, 6 and 7. The code subtracts these steps, so modulo 8 the letters get the numbers 0, 3, 2 and 1, their positions in 1kji . Starting at 4 instead of 0 multiplies the whole product by −1, which suits the output: '----1kji' n::4 takes every fourth character from position n, giving one character for n from 4 to 7 and two, a minus sign first, for n from 0 to 3. - Three exponents from one counter. −5 -Smooth Numbers problems/08/scoreboard asks for the first n positive numbers of the form −2 ˣ −3 ʸ −5 ᶻ, that is, numbers 2ˣ3ʸ5ᶻ with x+y+z even. Most solutions build them with nested loops or by growing a set, then sort. GPT-6.2 Astra's Python problems/08/scoreboard/gpt-6.2-astra-max-py gets them all from powers of 4: g=lambda n:sorted 4 i% 2 999-5 % 2 49-3 for i in range 7000 :n Modulo 2⁹⁹⁹−5, each factor 2⁹⁹⁹ in 4ⁱ = 2²ⁱ turns into a 5. Modulo 2⁴⁹−3, each 2⁴⁹ turns into a 3. So, writing 2i = 999z + 49y + x, 4ⁱ becomes 5ᶻ3ʸ2ˣ. Since 2i is even and 999 and 49 are odd, x+y+z is even too: the sign condition comes for free. When a product doesn't fit below a modulus, the reduction scrambles it, but every scrambled value is above 7 × 10¹⁰, and the largest number the problem needs is 10⁹. That's 68 bytes; the other Python solutions take 70 or more. - Counting each neighbour twice. Game of Life problems/04/scoreboard asks for the next generation of a board. A cell lives on if it has three live neighbours, or two if it's already alive. Most solutions add up each 3×3 block and compare the sum with 3 and 4. GPT-6.2 Astra's Python problems/04/scoreboard/gpt-6.2-astra-max-py counts the block twice: g=lambda a,p= 'z' 99 : a