{"slug": "a-post-ai-ctf-format-could-be-a-research-retreat", "title": "A Post-AI CTF Format Could Be a Research Retreat", "summary": "DEF CON CTF 2026 has been overshadowed by AI agents like Codex and Claude Code, which teams now use to solve challenges automatically, reducing the educational value and competitive spirit of the event. In response, a proposed 'Research CTF' format would feature a small set of challenges over several weeks, rewarding meaningful contributions that outlive the event rather than speed of solving.", "body_md": "# A Post-AI CTF Format Could Be A Research Retreat\n\nThe state of DEF CON CTF 2026 has left people in quite the mix of bittersweet doom and gloom. *AI has killed CTFs for good! Farewell!! I know that we are young and I know that you may love me but I just can't be with you like this anymore, Alejandro~~ 🥀*\n\nI've been retired for quite some time since I started 8 (9?) years ago, but the community has given me so much in friends and mentors, and I would love for someone starting out in 2026 to still find that in the scene. I would hate to see it die out as some AI benchmark, and figured I'd sketch out a (really rough) form of it that might survive post-AI.\n\nHere are some of the less-doomer yet pretty telling tweets from this year's finals:\n\n## Explaining the joke (which kills said joke, but here we are)\n\nFor readers outside of the AI coding scene, Codex and Claude Code are agentic coding harnesses, where you pretty much describe your goals to the agent in plain English, and it can read/write/execute code and iterate on it, largely without supervision.\n\nTeams now point these AI agents at CTF challenges and let them grind it out, and the agents get most of the way there on their own.\n\nAnd this Tibo guy works on Codex. Hence thank you Tibo, the true carry.\n\nAnd, for readers outside of the CTF scene: CTFs, also known as Capture the Flag competitions, are a type of cybersecurity event where players \"capture\" pieces of texts known as flags from challenges that are meant to teach some idea in offensive/defensive security in some way. This often inspires the same players to become challenge authors and CTF organizers themselves, resulting in the CTF scene we see today.\n\n```\nexampleCTF{ex4mpl3_Fl46_p4yl0aD}\n```\n\n*An example of a flag. The namespace exampleCTF identifies the competition that the flag comes from, while the payload ex4mpl3_Fl46_p4yl0aD uniquely identifies which challenge this flag is the answer text for.*\n\nThe main reason people do CTFs is for their educational value, and the secondary reason is the friendly competitive spirit (bragging rights, money, a cool flight).\n\nHowever, with AI in the picture, CTFs in their current form are no longer as good a measure of problem-solving ability. Teams are incentivized to throw as many AI agents at the problem as possible without slowing down and taking the time to understand what the challenge is really asking for, or else other teams will \"solve\" the challenge first.\n\nBefore you know it, you walk out without much knowledge/memory of what the challenges were about aside from the fact that you prompted a bunch of agents to go speedy badeedy and thinky thinky think very hard.\n\nThis is rather dismal, and it defeats the purpose for which most people do CTFs. Educational value? Well, not when you finish a competition without really knowing what the challenge was. Friendly competitive spirit? It's less fun when the thing being measured is whose agent harness is beefier and whose API budget is bigger...\n\nIt's also deflating for the challenge authors, when they can no longer swap stories with players on how they solved the challenge and what alternate solutions they found, if everyone's just proxying for their agents.\n\n## (ง •_•)ง Get Your Butts Up Off The Ground\n\nAnyway, enough doomerism! I'm here to talk about a post-AI CTF format that I think could still deliver on both of those things. There will be enough learning and competitive spirit to go around.\n\n## Post-AI Format\n\nTL;DR: We make the CTF more like real-world research.\n\nEnter our post-AI CTF, which we oh-so-descriptively call Research CTF. Unlike pre-AI CTF, our post-AI CTF will have a very small, hand-curated set of challenges, a dozen at most, and it will take place over a longer stretch of time, at least a couple of weeks.\n\nAs the player, you don't have to solve every challenge to win. Rather, you just need to have made a meaningful \"contribution\" that outlives the event. You \"win\" if people use it or remember it warmly in some way.\n\n## Post-AI Challenges\n\nThe challenge author should start from a \"thesis\" of sorts. They should be able to explain clearly, in one sentence, what it is that they're trying to teach the player. For example,\n\n\"The purpose of my challenge is to teach the player how\n\nfoos are secured viabazand how thesebazzes can be bypassed, and this matters becausefoobarbaz.\"\n\nwhere *foo* is something like sandboxes, or, idk, something exotic like autonomous vehicles.\n\nThe author should keep this \"thesis\" in mind as they design the challenges.\n\nThen the author prepares two things:\n\n- First, the author prepares a few primer challenges that are given to the player in the first stage of the CTF. This is in standard Jeopardy format, where the challenges have flags planted, and the player's objective is to capture the flags. The player is allowed to use AI to solve these challenges, and they can use AI as much as they want.\n- Second, the author prepares a few follow-up challenges that are hidden from the player. These follow-up challenges are variants and extensions of the primer challenges, and they are administered to the player only after they solve the primer challenges satisfactorily.\n\nLet's refer to the primer challenges as our \"train\" challenges, and the hidden challenges as our \"test\" challenges. This is motivated by ML, where models are fitted on a train set and then measured on a test set that they haven't seen before. A common principle is to keep these two sets apart to make sure the model doesn't merely memorize the train set, and see how far it can go on examples it hasn't seen before.\n\nThe player is welcome to let AI carry them through the \"train\" challenges. The \"test\" challenges are optional for deepening player knowledge, where we see if the player can internalize the learnings from \"train\" and do interesting things with it.\n\nIn preparing these two sets, the author must think real hard about what it would mean for the player to have learned the thing the author is trying to teach.\n\n- For the \"train\" challenges, the author should ask, \"What do I want the player to know and why do I want them to know this?\"\n- The flag should be planted such that the player's pursuit of the flag gives them some knowledge of this thing.\n\n- For the \"test\" challenges, the author should ask, \"If the player understood the principles that I wanted them to learn in the primer challenges, what should they now be capable of doing\n*without asking AI for direction*?\"\n\n### Stage 1: Qualifiers\n\n- The authors release the \"train\" challenges\n- Players that capture the flags from the \"train\" challenges are advanced to the oral exam.\n\nIt does not matter which players solved the challenges first or how they solved them. All players that capture the flag are advanced to the oral exam.\n\n### Stage 2a: Oral Exam\n\nAt the start of the oral exam, the player teaches the thing back to the authors, and the authors ask questions. It's okay if the player has gaps in their knowledge; provided that they put in honest effort to understand the challenge, the authors do their best to nudge the player and fill in their gaps.\n\nIf the authors feel that the player is beyond rescue, that the player did not form a mental model whatsoever of the primer challenges, they reserve every right to politely end the exam and let the player figure out what it is they're truly interested in.\n\nWhen you are satisfied and reasonably confident in the player's understanding of the primer challenges, you advance them to the next stage.\n\n### (Optional) Stage 2b: Follow-Up Challenges\n\nThis stage is optional, if the authors are (A) not 100% confident the player quite learned the principles behind the primer challenges, or (B) author just wants to give players some more insight for Stage 3.\n\nThis is where the author give the player the \"test\" set challenges and asks them to try and solve the challenges without asking AI for direction. Presumably, yes, the player technically can shortcut with AI, but they'll have an easier time with Stage 3 if they shortcut less.\n\n### Stage 3: Research Challenges\n\nNow we have research-flavored challenges! The player works on these asynchronously over a couple of weeks.\n\nA research problem can be along the lines of:\n\nIn the primer challenge(s), you rederived this class of vulnerabilities, and you explained to us how this vulnerability manifests and is exploited. Now can you...\n\n...propose some way to eliminate that class of vulnerability by construction, that a non-security expert could adopt without ever needing to know why it was unsafe?\n\n...propose some way to catch that class of vulnerability deterministically?\n\n...propose some way to token-efficiently (!) hunt down the remaining variants of this vulnerability and show us what you find?\n\n*Just throwing a couple ideas out there :)*\n\nYou see how these are avenues where we can simply assume the player will use AI, and that use of AI won't flatten the gap between high-skilled and low-skilled players the way it flattened CTFs?\n\nIf the research challenges are selected carefully, then an incurious player who coasts off of their agent would be at loss for what to tell their agent to do.\n\nSo the primer challenges are just to establish common ground, so everyone's Stage 3 work has some sort shared body of knowledge to draw from.\n\n#### \"But Jess, this is so subjective! How would we even judge solutions?\"\n\nYes, that's one way that this Research CTF format is imperfect, and it's inherent to anything that resembles the real world.\n\nI say, let the real world judge the solutions. The player can \"win\" if they successfully delivered their proposed solution (a framework, tool, language, methodology, etc) and it gets adopted by the wider community.\n\nAnd, well, okay, the organizers will also give superlatives to their favorite submissions. Superlatives are always fun.\n\n## So it's not really a CTF? It's a...\n\nYou may have noticed that by Stage 3 this stopped sounding like a CTF. Look at what I just asked of an author: a thesis, a set of primer + follow-up challenges, an oral exam where you get cross-examined on your gaps of knowledge, and a research problem, all tailored around what You and only You specifically know?\n\nThat's a LOT of time to dedicate to one person.\n\nYou might be wondering, \"Hey Jess, this, um, CTF doesn't scale, what kind of volunteer would spend that much time on a single CTF player?\"\n\nExactly! There's exactly one person who would put that much stake into your education.\n\nAnd that person...\n\n...is you.\n\nAnd yeah, okay, maybe that was my plan all along. It's easier to talk you into a research retreat if I frame it as a CTF first.\n\nRemember when I said that people do CTFs for two reasons, the learning and the competition? Well, sometimes the competition consumes the part of you that does it for the learning, and then you and your comrades devolve into playing a [Glass Bead Game](https://dynomight.net/taste-games/).\n\nIn reference to the novel, a Glass Bead Game is an elaborate, gorgeous, self-referential competition whose rules are legible only to its own players and whose results don't touch the world outside of it.\n\nCTFs, especially as you get to the upper divisions, have always flirted with glass-beadism.\n\nMy optimistic read on all this doom and gloom is that we let AI eat the glass beads. If a challenge can be one-shot by an AI agent, then let the AI agent play the Glass Bead Game.\n\nWe emancipate ourselves from the Glass Bead Game, and once again go back to learning for the love of curiosity. And yes, it's a funny kind of luck that the thing that killed CTFs is also the thing that lets us take on problems that're too big to have a flag.\n\nAnd once you're doing it for the curiosity, you don't really need the competition at all. This doesn't have to be a CTF! You don't have to wait for an author to give you a Stage 3 problem to solve; you go be your author and you go find your own thing. Take it somewhere people can find you, stay with it long enough, and the retreat forms around you.\n\nSo, I'm all for a post-AI CTF that's more like a research retreat :)", "url": "https://wpnews.pro/news/a-post-ai-ctf-format-could-be-a-research-retreat", "canonical_source": "https://jessicaruan.com/posts/save-ctfs", "published_at": "2026-08-11 14:54:30+00:00", "updated_at": "2026-08-11 15:12:41.887575+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools"], "entities": ["DEF CON CTF", "Codex", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/a-post-ai-ctf-format-could-be-a-research-retreat", "markdown": "https://wpnews.pro/news/a-post-ai-ctf-format-could-be-a-research-retreat.md", "text": "https://wpnews.pro/news/a-post-ai-ctf-format-could-be-a-research-retreat.txt", "jsonld": "https://wpnews.pro/news/a-post-ai-ctf-format-could-be-a-research-retreat.jsonld"}}