{"slug": "generating-coding-puzzles-with-llms", "title": "Generating coding puzzles with LLMs", "summary": "The developer of dailyprog, a daily coding puzzle site, built an LLM-based puzzle generator to automate content creation after manual puzzles proved unscalable. The generator uses a structured vocabulary and code validation to produce original puzzles, avoiding legal risks by sourcing seed problems from MIT-licensed Exercism instead of restrictive platforms like Codeforces or Project Euler.", "body_md": "I have been working on this daily coding puzzle site\n[dailyprog](https://dailyprog.club). It’s like Wordle for programming problems.\nEach day you get one puzzle with a short narrative, a function stub, and a set\nof visible test cases. You write a solution, run it locally against the visible\ntests, then submit to verify against hidden tests on the server. There are four\nlanguages atm (JavaScript, Python, C, Go), visual output for some puzzles, and a\nshare link so you can signal your *competency* to your *friends*.\n\nThe site needs a new puzzle every day. When I first started, I wrote the first few by hand. Pick a problem, design test cases, write a reference solution, verify it in the sandbox. Each puzzle took 30 to 45 minutes. That doesn’t scale past a week.\n\nThe obvious solution is to have an LLM write them. And the obvious problem is that LLMs write broken puzzles. They inline 20,000-element arrays in JSON, they produce solutions that don’t pass their own tests, they cycle through the same three algorithm patterns, and their “brute force” attempts are sometimes correct on every hidden test.\n\nI built a generator that fixes this. Not by making the model smarter. By giving it a vocabulary for what a puzzle is, and then running its output through real code until it stops being wrong.\n\n## Picking a corpus\n\nThe first problem wasn’t technical. It was legal.\n\nAn LLM trained on the open web has seen every puzzle site. LeetCode, HackerRank, Codeforces, Project Euler. If you ask it to generate a coding puzzle, it will regurgitate something it memorized. The only way to constrain it toward original output is to give it a structured vocabulary and a set of seed problems it can remix. But the seed problems have to be legally safe to ingest into a product.\n\nI surveyed nine sources. The table wasn’t encouraging:\n\n| Source | Problems | License | Usable? |\n|---|---|---|---|\n|\n\n[Codeforces](https://codeforces.com)[CP-Algorithms](https://cp-algorithms.com)[freeCodeCamp](https://www.freecodecamp.org)[USACO Guide](https://usaco.guide)[CSES](https://cses.fi)[Project Euler](https://projecteuler.net)[Rosetta Code](https://rosettacode.org)[OEIS](https://oeis.org)Three problems run through the whole list.\n\n**NonCommercial kills half of them.** USACO Guide, CSES, and Project Euler are\nall CC BY-NC-SA. The NC clause restricts use, not the user. A for-profit product\ncan’t serve NC-derived content even if the product itself is free.\nReference-only, probably fine for private calibration, legally risky as\ningestion into a generator that ships to users. Didn’t use it anyway.\n\n**ShareAlike is an infection risk.** CC BY-SA and GFDL require derivative works\nto carry the same license. If you condition an LLM on BY-SA text and it produces\na puzzle, is the output a derivative that inherits BY-SA? Nobody has litigated\nthis. I didn’t want to be the test case. CP-Algorithms and freeCodeCamp are\nBY-SA, so I took only technique names (facts, not copyrightable) and left the\nprose alone. Rosetta Code’s GFDL is even worse: copyleft designed for\ndocumentation, awkward in a product.\n\n**Codeforces has the best data and the worst terms.** 11,255 problems with 38\ncontrolled topic tags and a calibrated Elo difficulty scale derived from actual\nsolve data. But the ToS explicitly forbid republishing problem statements in\nanything with automatic testing (which is exactly what dailyprog is) and\nexplicitly forbid publishing tests and checkers. I only kept the tag vocabulary\nand the difficulty scale, just metadata. I skipped the problem statements.\n\nThat left Exercism. MIT licensed, 151 cross-track exercises, each with a structured problem statement, a separate framing story, and canonical test data in a machine-readable JSON format with a published schema. No restrictions. No ShareAlike infection. No attribution requirement (I attribute anyway). It’s the only source on the list where you can take everything and use it without thinking twice.\n\nI ingested all 142 solvable exercises into a normalized authoring corpus: title, statement, interface signature, structured test cases with named inputs, and metadata. Then I had the LLM classify each one.\n\n## Building the ontology\n\nClassification was a single deliberate pass. Read every problem’s full statement, its test cases, its canonical solution. Classify by topic, mechanics, difficulty, and the single algorithmic technique it teaches. Report confidence. Justify each judgment.\n\nThis took about an hour of LLM time. Not because 142 problems is a lot of text. Because each judgment needed reasoning, and reasoning tokens are the point. The output isn’t a set of labels. It’s an ontology.\n\nThe topic distribution that emerged:\n\n```\npie showData\n    title Puzzle topic distribution (142 problems)\n    \"strings & text\" : 29\n    \"logic & simulation\" : 21\n    \"math & number theory\" : 21\n    \"arrays & sequences\" : 20\n    \"encoding & ciphers\" : 19\n    \"games & puzzles\" : 17\n    \"parsing & evaluation\" : 6\n    \"graphs & trees\" : 5\n    \"geometry\" : 3\n    \"dynamic programming\" : 1\n```\n\nThe technique enumeration: two pointers, sliding window, prefix sum, frequency map, binary search, monotonic stack, greedy, backtracking, BFS, DFS, dynamic programming, heap / top-k, union-find, tree traversal, linked list manipulation, matrix operations.\n\nTwo things surprised me about the classification. First, the LLM saw structure I wouldn’t have named. The “encoding & ciphers” cluster, for instance. I would have called those “string manipulation” and lost the distinction between text-processing puzzles that are about pattern matching and text-processing puzzles that are about translation between representations. The model caught it because it had to reason about each case individually.\n\nSecond, one problem failed: a pure refactoring exercise with no algorithmic core. The taxonomy has no “code-quality” category. The LLM gave it confidence 0.65 and classified it as “logic-simulation,” which is wrong. The problem wasn’t the classification. The problem was that the corpus included something that isn’t a puzzle, and the ontology correctly flagged it as an edge case.\n\n## The repair loop\n\nWith the ontology in place, the generator is a state machine. A model proposes a puzzle. A validator runs it through every gate. On failure, the issues go back to the model with specific fix instructions. On success, the puzzle is written to disk.\n\n``` php\nstateDiagram-v2\n    [*] --> generate: prompt + diversity constraints\n    generate --> validate: JSON proposal\n    validate --> write: all 4 gates pass\n    validate --> repair: any gate fails\n    repair --> generate: fix instructions\n    write --> [*]: puzzle.json + solution.ts\n```\n\nThe validator runs four checks:\n\n**Schema and quality.** Does the JSON parse? Right types? Is the id a kebab-case\nslug? Are there distinct test names? Early versions returned raw Zod errors, and\nthe model burned attempts on things like “id must be a string.” Now each failure\nclass gets its own diagnosis: “id must be a kebab-case slug starting with a\nletter, here’s what you wrote, fix it.” The difference between “ZodError:\ninvalid string” and a targeted fix instruction is the difference between a\n4-attempt failure and a 2-attempt pass.\n\n**Reference solution.** Run the model’s proposed answer through the production\nsandbox against every visible and hidden test case. If it fails, the model’s own\npuzzle defeats it. This catches most problems immediately.\n\n**Collapse.** The model also proposes a brute-force attempt (the obvious naive\nsolution). The hidden tests must defeat it. Either wrong answer on an edge case,\nor timeout on a large adversarial input. If the brute force passes everything,\nthere’s no puzzle. A correct solution to a trivial problem is busywork.\n\n**Technique diversity.** The model records which technique each puzzle teaches.\nBefore generating, the constraint scans the last 7 puzzles’ tags and blocks\nrepeats. A separate gate rejects any candidate whose technique matches a recent\npuzzle.\n\nThe loop catches things the model doesn’t know it’s doing wrong. The biggest one is array inlining. When a puzzle needs a large adversarial test case (10,000 integers to force an O(n) vs O(n squared) collapse), the model writes all 10,000 values as a literal JSON array. The payload hits 300 KB. The model spent its token budget on array elements instead of puzzle design.\n\nThe fix is compact array notation. Three sentinels expanded before the tests\nrun: `[\"__RPT__\", n, value]`\n\nfor repeats, `[\"__RNG__\", start, end]`\n\nfor ranges.\nA pre-expansion gate rejects any literal array over 50 elements. The model gets\ntold “you inlined a 10,000-element array, use **RNG** instead” and tries again.\n\nThis sounds like a trivial formatting rule. It is. The model won’t follow it unless it’s enforced. LLMs are pattern-completers, and the pattern they’ve seen in training is “test cases have concrete values.” The gate makes that pattern fail until the model cooperates.\n\nThe second thing the loop catches is catalogue-level technique drift. When I first ran the generator without diversity constraints, it produced three binary-search puzzles in a row. Different titles, different prompts, identical underlying algorithm. The model has no memory of the last puzzle it generated. The technique tags, recorded per puzzle and checked before each generation, give it that memory.\n\nMost puzzles close in 2 attempts. About 1 in 5 fails entirely and I run the\ngenerator again with a different pattern hint. The ones that ship are the ones\nthat survive the loop.\n[Hidden Stash](https://beta.dailyprog.club/en/puzzle/2026-06-29) is a prefix-sum\nproblem wrapped in a story about a pirate stashing their booty.\n[Garden Plots](https://beta.dailyprog.club/en/puzzle/2026-06-27) is a dynamic\nprogramming problem wrapped in a story about planting vegetables. The titles don’t\nname the technique. The prompts don’t either. The classification exists in the\nauthoring metadata, stripped before the puzzle reaches the client.\n\nThe site doesn’t know any of this exists. It just reads JSON files from\n`content/puzzles/`\n\n. The generator is a script that’s never imported by the app.\nIt produces artifacts for the real system to consume.", "url": "https://wpnews.pro/news/generating-coding-puzzles-with-llms", "canonical_source": "https://blog.lvmbdv.dev/posts/generating-coding-puzzles-with-llms/", "published_at": "2026-06-29 00:00:00+00:00", "updated_at": "2026-07-16 03:27:08.540394+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["dailyprog", "Exercism", "Codeforces", "Project Euler", "LeetCode", "HackerRank", "CP-Algorithms", "freeCodeCamp"], "alternates": {"html": "https://wpnews.pro/news/generating-coding-puzzles-with-llms", "markdown": "https://wpnews.pro/news/generating-coding-puzzles-with-llms.md", "text": "https://wpnews.pro/news/generating-coding-puzzles-with-llms.txt", "jsonld": "https://wpnews.pro/news/generating-coding-puzzles-with-llms.jsonld"}}