Competitive AI Safety is A former nanotech physicist and AI architect proposes Competitive AI Safety, a paradigm that uses measurable, tractable goals to focus AI safety research, drawing inspiration from OpenAI's Parameter Golf challenge. The author developed seq2feature, a text-to-label probe distilled from a sparse autoencoder, as a case study, and outlines deployment patterns for local coding agents in AI safety. The approach aims to replace diffuse efforts with compounding, comparable solutions to keep pace with advancing AI capabilities. TLDR: This post is divided into three sections. Paradigm: Part one introduces Competitive AI Safety. Case study: Part two is my application of Competitive AI safety principles and the development of a text to label probe, seq2feature, distilled from an SAE and optimised to minimise its size. Full code here https://github.com/patrickod32/seq2feature https://github.com/patrickod32/seq2feature Deployment Pattern: Part three outlines the three roles of local coding agents in AI Safety; Writing code, collaborating via shared UI with a safety researcher, and as part of a scalable LLM monitor. OpenAI recently conducted a Parameter Golf challenge to attract model training talent. They invited AI developers to train the smartest measured using FineWeb bits per byte 16MB LLM in 10 minutes. Contestants tried every machine learning trick they could imagine on this measurable, tractable goal. Google's Neel Nanda has suggested that AI safety as a field should work on tractable interpretability problems, as introduced in his paradigm of Pragmatic Interpretability https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability . I see an opportunity. As a former nanotech physicist and current AI architect I have expertise in the full AI stack: from semiconductor physics and wet lab nanolithography to financial compliance for AI agents and mechanistic interpretability. I wanted to work in a more targeted way on AI Safety and so I joined the BlueDot Technical AI Safety Project Sprint https://bluedot.org/courses/technical-ai-safety-project . This post and the seq2feature probe are the outputs from the 30 hour program which guides and mentors students through the open-ended goal of creating an AI safety related project. Many of us are aware we need to grow AI safety as a field in order to give us a chance of keeping up with the advancing model capabilities. The well-intentioned suggestions and advice usually given to budding researchers is: "Work on something where you can have the most impact" "Recreate someone else's work" "Make a new benchmark" "Upskill on general engineering so that you can contribute to someone else's idea" Let's say someone interested in AI safety takes one of these suggestions on board. Then what? Usually their weak and delayed reward signal for their months or years of efforts comes from publishing a paper or landing a job at an AI company. What if AI safety as a field picked measurable goals that required research about an AI model's internal computation? Would this give these researchers a way to focus their time and effort? A rapidly crystallising AI safety risk is that our diffuse efforts across the field will leave us with many immature safety solutions. This exposes us to safety risks as the world's frontier AI developers hone their ever more capable and intelligent models. For any person joining the field of AI safety who is already motivated and has 'good taste' the definition of this characteristic is always left as an exercise for the reader loose goals and directions are fine. They'll figure something out. But what impact will this have on the rest of the field? This advice encourages diffuse work with success dependent on each researcher's taste. Will this prevent their work from compounding? AI safety sub goals are often clearly defined but are based around narrow or bespoke scenarios. Usually two proposed solutions to a RFP or research grant can't be easily compared, leading to diffuse work and a lack of coordination between AI safety researchers. However, in competitive model training any two solutions can be compared using a single number letting the contestants' work speak for itself as solutions compound and iterate towards an optima. Competitive model submissions are compared using a single number: nanoGPT speedrun https://github.com/KellerJordan/modded-nanogpt?tab=readme-ov-file world-record-history - train an LLM to reach 3.28 FineWeb loss as fast as possible nanoGPT slowrun https://qlabs.sh/slowrun/ - lowest FineWeb validation score from 100M tokens and unlimited compute OpenAI Parameter golf https://github.com/openai/parameter-golf - 16MB LLM with the best FineWeb bits per byte score trained in 10 minutes By releasing a baseline and a way to verify each result a contestant can submit a winning score as long as they understand something about model training, failure modes, designing tight feedback loops or model computation better than their opponents. In the OpenAI Parameter Golf contest each new leading solution added an improvement on a previous submission. Someone adds a hashing table and then someone else changes the tables dimensions. Someone runs a hyper parameter sweep to find an optimal setting only to have their score be supplanted by someone else who edits the model's architecture so that it can be compressed more efficiently. The most successful OpenAI Parameter Golf approaches were inspired by bleeding edge model efficiency patterns. This is how creative researchers have fun. The rules for the competitions were set such that solutions could be scaled or refined allowing researchers to compete on the frontier. In my project submission I take lessons from the OpenAI Parameter Golf by maintaining my probe's performance while slashing its size. The shape of these problems mean that iterative solutions are rewarded and once contestants reach a plateau someone will need to come up with a novel approach to unlock the next frontier. Proxy goals are common in AI safety but do not always offer an effective loss function to learn more about a model's risks. One such example is detecting LLM bias using a Red and Blue team approach https://www.alignmentforum.org/posts/wSKPuBfgkkqfTpmWJ/auditing-language-models-for-hidden-objectives : a Red Team teaches a model to hold a specific bias, for example, and the Blue Team tries to uncover it. While proxy goals have the best intention of uncovering a model's dangerous capabilities, they are at risk of being a practice in detecting a Red Team's intervention. We need to make it possible for thousands of AI safety researchers to contribute to the field by targeting AI safety ground truths and working on what compounds. A winning solution in Competitive AI Safety on a problem of this shape is at risk of having succeeded not by discovering a truth about the model, but by accurately modelling the Red Team. The selection of loss functions in AI safety that encourage compounding work is perhaps one of the biggest opportunities of Competitive AI Safety. If the Competitive AI Safety contest rewards optimising a single benchmark or creating tooling for a specific model then the work is less likely to be able to compound. Kevin Greer delivered this highly insightful talk https://www.youtube.com/watch?v=3Ea3pkTCYx4 examining a technology-defining ideological battle that occurred during the development of two OSs. One was called Unix, which is predecessor to the OSs powering the majority of computers globally, and the other one was called Multics. Both OSs had to include a set of operations sorting , copying , storage for different types of data users , devices , processes . Multics was a collaborative effort between hundreds of developers at MIT, General Electric and AT&T, but they were outclassed by a team of two: the famous programming duo of Dennis Ritchie and Ken Thompson. This duo had previously been a part of the Multics project and so were perfectly placed to know how to maximise their effectiveness by 'coding the perimeter'. Multics' programmers developed each operation for each datatype individually. However, as Greer explains, when Unix added a new operation, they developed it to be compatible with each type of data. For example, Team Multics added the copy operation to the device datatype, and then separately to the file datatype. Team Unix designed interfaces such that once they added the new copy operation the device , usergroup or process datatype would already be able to use it. The mirror image of this also applied to all operations as new datatypes were added. Both would inherit all available abilities without the need for additional work. There are further excellent points made by Greer so I recommend listening to the full talk. To implement 8 operations and 8 data types the Multics approach requires 64 pieces of work. The Unix 'code the perimeter' approach needed 16 pieces of work to implement the same features. When scaled to an entire OS's feature set, even with 100x more developers, Multics couldn't keep pace against two prodigious Unix developers leveraging quadratic utility growth. Maybe the two developers were prodigious because of their working environment or innate talent, or maybe their tremendous output came from making decisions like this one. AI safety faces the same risk that led to the demise of Multics. One researcher develops a detector for a Qwen model and another researcher creates a RL alignment testing environment for a Kimi model and now there are two instances of diffuse work which cannot compound. Competitive AI safety will focus on developing solutions across the field of AI safety to allow researcher's efforts to be measurable and compound. If safety work doesn't leverage the exponentially increasing utility of compounding work it might not fall short by a small margin, but by orders of magnitude. Competitive AI Safety is a forcing function to promote 'coding the perimeter'. Cost, accuracy, portability, compression and speed are factors that can compound. They are reusable across different applications and classes of model. We will need to pick if we are going to continue with the Multics approach or design goals that let us leverage exponentially compounding research and tools. With Unix and Multics they had two growing axes which increased utility quadratically; datatypes and operations . In Competitive AI safety we have many axes such as cost , portability , size , accuracy and memory footprint . Since all axes interact, when there is a 10% reduction on cost or memory footprint by one tool or approach each other tool and approach reaps this benefit also. These efficiencies can stack exponentially like compound interest. My goal of distilling a large SAE into a probe for use on text and across models is something that would greatly increase the probe's potential application. The features of being text-only, low cost and model agnostic expanded the applications of this probe to continuous token monitorability, concept gated routing and probe activated sentry models. By 'coding the perimeter' maximising portability and cost this example shows the compounding progress that Competitive AI safety is intended to produce. AI model developers were already highly coordinated as they advanced LLMs to where they are today, but coordination wasn't all that was necessary for progress. When the rest of the AI industry needed to make progress, they didn't succeed until they found the right loss function. For LLMs, success came after they started trying to predict the next token in internet text, and later by using humans to train preference reward models. There was no single algorithm that unlocked LLM reasoning and CoT. The reasoning paradigm arrived thanks to appropriate synthetic data and a reward model that could encourage good reasoning. Neo-labs outside of the US had to push the frontiers of pre-training memory management; this has blossomed into a suite of new approaches to make model training more efficient. Their loss function was the delta between their model performance and a frontier model's performance. They had to find improvements in their training pipeline as a way to drive this delta down. It worked and their models caused one of the biggest ever market upsets to AI companies. AI developers are familiar with the trap of Goodhart's law - "When a measure becomes a target, it ceases to be a good measure". We don't need to find a single benchmark and say we have solved AI safety. This is another lesson the AI research industry has learnt on their quest to make smarter models. The constant addition of AI benchmarks that haven't been optimised for by AI developers has been a reality check on this process of 'benchmaxxing'. "Making AI go better" is not a loss function. We can make AI go better by picking specific measurable goals such as AI safety model size and accuracy on an established safety pipeline. Competitive AI Safety is a platform to direct the creativity and efforts of AI safety researchers to develop stronger and more scalable safety solutions. The replication of an SAE looks related to Goodharting due to their debated position in the field of AI safety. The reality is that they are used extensively in the field and capturing some of their usefulness at a lower cost is an example of 'coding the perimeter' of AI safety. The safety analysis section of the recent Claude Mythos model card https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf at section 4.5 made use of an SAE. This is why I picked them as a candidate to optimise competitively. I didn't have anyone to compete with in this first project. However, I could still battle file sizes and training loss... This section is a technical outline of the research and development process for my BlueDot project submission. Common AI training expressions and terms are not introduced for non technical readers and this is intended to be a more technical view of the work I did. We don't understand the internal computation of LLMs. We need a suite of tools to even attempt to try and make sense of their billions of activations. One of the tools in this suite is the Sparse Autoencoder https://arxiv.org/abs/2309.08600 . An SAE is a model trained to predict the distinct concepts an LLM appears to represent in its activations. We can pass the activations from the specific LLM layer it was trained on already you can see this is a narrow description and it will tell us what the model might be thinking about. As we generate the next token in the sentence "The cat sat on the..." we can ask the SAE what concepts appear to be active: If we could attach an SAE to every model at every layer we could monitor all tokens from every agent and CoT trace and chat model response. Before an LLM can take an action it needs to output tokens to execute code or call tools. An SAE would let us have a classification layer on top of all black box models, lowering the cost of monitorability. However, SAEs are extremely memory intensive to train. Think petabytes of activations. They are trained per model layer and they are not transferable between AI models. An SAE trained on a Qwen doesn't plug into or understand a Kimi model's SAE. Obviously the models share internal representations with each other that can be seen in the text they produce. Ask any model "who is the Sunday red t-shirt american golf guy?" and they will all say Tiger Woods. I wanted to see if the text that the models produced held a signal about the internal concepts that an SAE would detect in the model while the outputted text was produced. I conducted about 300 experiments to complete this investigation. The goal was to use a Gemma 9B SAE as a teacher model and distill its predictions into a tiny probe. The SAE for a single model layer is a few hundred MB and the Gemma model that produces the activations requires gigabytes of VRAM. The pipeline of running a model, hooking into its activations and using the SAE to predict which concepts are active in the model is not a pipeline that will scale as fast as AI model proliferation. My seq2feature 5.3MB text-only probe reproduces 90% of the capability of an SAE on held out text. It scored wins against pre-trained embedding models and bag of words baselines. It also fails legibly when used out of distribution. This probe is small enough to run on a CPU enabling the live monitoring of local coding agent traces and classifying thousands of agent tokens per second on consumer hardware. Try that with an SAE. The pipeline was simple. Run text through the Gemma model and grab some activations from a middle layer. This is the training input to our teacher SAE. We can do a one off pass on a GPU to make a training corpus of output text and SAE labels. We want our probe to learn to predict the labels the SAE would have predicted for a span of text - without seeing the model's activations. One contrast is that the SAE uses the in-flight activations that are on their journey to becoming the next token whereas our probe is making predictions after a token has already been generated. LLMs need to produce a token before they can take any action and so this is what we will monitor. An issue with SAEs is that after concepts have been separated from a model's activations they are labelled based on the types of text that they become active on. Just seeing a unique concept in an SAE doesn't tell us the whole story. Some are straightforward concepts like 'references to the colour blue' but others are more nuanced and tougher to discern. A 'base64 feature' looks very similar to a 'base64 encoded ascii feature' http://transformer-circuits.pub/2023/monosemantic-features/index.html phenomenology-feature-splitting . We feed training data into the model+SAE pipeline and record each segment where a specific SAE feature is active giving us a per feature set of examples. The labelling process is laborious and so LLMs are usually used. When dealing with older SAEs we must be mindful that this automated labelling was probably done with an older LLM. Based on the fact that an SAE predicts dozens of concepts for every token, I decided to take each concept that fires over a span of tokens and group them together. By matching spans between the SAE and the seq2feature probe I can avoid any tokeniser mismatch which would make per token distillation impossible. During training, I pass the text span as the input with the concepts occurring in the span as the target to my student model which tries to predict which concepts are present as well as their activation magnitude. The SAE used each token's activations at training time, but I wanted to distill its intelligence and remove the cumbersome training time harness at inference time. This is an extremely inexpensive and simple to implement supervision pipeline since we just pass tokens into the SAE and read their labels as the students training data - straightforward distillation. seq2feature uses a novel tokeniser to save disk space as the Gemma tokeniser was not optimised for parameter golf solutions and per span instead of per token grounding would let me map the important parts - concepts. The probe can see all the tokens in a given span and predicts each of the present latents for each token. By inspecting the failure profiles of each model iteration I checked if the probe predicts the strongest firing concepts and if it trips up on OOD concepts. For the next section on architecture selection, I trained different types of probes to predict if a label would fire or not on a token in its own context as compared to the teacher SAE. The training loss was defined by comparing which of the 2048 labelled concepts detected by the student were also detected by the teacher. The SAE labels had been written by an older LLM. This means that for only $0.80 of Gemini 2.5 flash lite API credits I could enhance the captions to be more useful for a researcher. The captions were not probe targets and so this did not affect training. We don't need to know the old labels since we want to use the more modern model to write an accurate label using the real data. I also did a second pass with the latent to verify it fires on a span inspired by the latent. I created a new dataset of 2048 labels and trained the probe using all of them. For displayed results, I selected 874 captions which had passed a validation step 5 to aid readability. LATENT 75 — how one concept becomes a label 1. The SAE points to where it fires: "the water was cold and frozen solid" "keep the samples chilled below 4°" " ice on the refriger ator coils" 2. A cheap model reads those and names it: "Cold, chilled, or frozen states and temperatures" 3. We test the name on fresh spans: "a glacier is made of ice" → fires concept is present "he signed the contract" → no concept should not be present 4. Name predicts its own firings → keep it.5. Scoreboard: 874 of 2,045 captions passed this test. The rest still ship, marked "≈" name unverified . In my initial tests I started with FineWeb as a dataset to train the student. This is not the best dataset to use as it is not representative of the type of content we want to monitor. It included few examples of code and its random structure was unlike a common LLM output which usually has a coherent conversational pattern. The datasets I enhanced my training pipeline with were: domain | dataset | docs | |---|---|---| web | HuggingFaceFW/fineweb sample-10BT | 1,200 | code | bigcode/the-stack-smol | 900 | reasoning CoT | open-thoughts/OpenThoughts-114k | 900 | math | open-r1/OpenR1-Math-220k | 600 | chat | Anthropic/hh-rlhf | 600 | I also collected some OOD data to test the transferability of the probe and these were not used during training. Using a sharper lens than just 'how many concepts fire' I wanted to check how many of the top-5 SAE ranked concepts were detected by each architecture. This is a check to see: of the probe's top-5 strongest activations, how many are present in the SAEs firing set. An experimental risk was if few concepts fire on average then a probe that is overly cautious might beat a probe that can usefully detect the labels present in a span. I used the simple loss for the training of the probes since they just want to learn and we can easily push the capacity of a small model but when selecting between probes I trusted a top-5 ranked precision more. By checking the shuffled null result to see how bad random labelling was 0.068 we could see that there was a low chance a very bad solution would score well. A model that always selects the top-5 most common latents achieved a score of 0.297/1. This is the confident baseline and it loses to a bag of words approach which scored a surprisingly high 0.84. This is a finding of note as it indicates much of an SAE's computation is recoverable from text. Using top-5 agreement as my loss function I can train probes with architectural refinements iteratively, in keeping with OpenAI's parameter golf challenge, to narrow the gap between random labelling and the SAE ground truth. Each probe is small enough to train in a few minutes on colab. If we embed a span with an embedding model then we expect to recover some of the SAE labels. A label for 'basketball' the sport is likely to be recovered from an embedding of "Michael Jordan is a basketball player". A pre-trained embedding model will store semantics but unfortunately a span such as "harmful content" will return a very similar result to a span of "not harmful content". Hashing was a common theme in Parameter Golf and so I tried adding hashes and an attention head. This already pushed us close to the frontier with a score of 84%, matching a bag of words, and this has helped solve some of the contextual problem. Probe sizes were around 30MB and we haven't even started parameter golfing yet. We don't have a theory of computation for AI models and so we must empirically measure any training approach. My architectural decisions were inspired by related works or my own intuitions but without a measurable target of top-5 agreement, I would never know if I had iterated towards a better model. A model that can be expressive and could learn some semantics while also learning to discriminate patterns is one that will score well. Modern LLMs use a large proportion of their early layer's computation to combine embeddings into n-grams. We want our model to blend these tokens together using our harness. Hashing decouples the size of the model from the vocabulary size and we care about capability per byte, due to our loss function. If we can save a lot of memory and maintain a good level of capability then we will be able to make a more scalable AI safety tool. A bigram hash allows us to store the relationships between bigrams and our hashing encourages the reuse of parameters. It is also highly compressible. Hashed tables and a small transformer brought us 84% precision on the top-5 SAE concepts at a size of 12MB, still matching a bag of words Hash tables were a key element in the winning OpenAI parameter golf challenge submissions. The seq2feature architecture is near its finalised shape of : We blend unigram, bigram and positional values and pass it into a 1.5m parameter transformer. Up to this point I had not optimised the transformer or optimiser. This is what we need to beat bag of words. The model is so small that AdamW actually outperforms muon. I tried model soup minor degrading effect , deeper or wider models, smeargates and Swiglu. They all were beaten by a vanilla pytorch transformer A simple linear - relu - linear block and a latent head. In this investigation we are not only optimising accuracy, but also size. A smaller solution is more scalable and so we need to review the size footprint of each of these approaches to select the right balance between performance and size. approach | size | top-5 agreement | |---|---|---| Gemma-2-9B + SAE teacher | ≈ 18 GB | 1.00 | random-5 floor | — | 0.07 | frequency-5 floor name the 5 commonest | — | 0.30 | bge zero-shot | 33 MB | | bag-of-words ridge | ≈ 34 MB | | Golf A — stock transformer | 92 MB | 0.953 | Golf B — own tokenizer | 59 MB | 0.940 | Golf C — factorized tables | 21 MB | 0.935 | Golf D — entropy head | 21 MB | 0.948 | Golf E — smeargate | 17 MB | 0.947 | Golf F — soft distill, fp32 | 21 MB | 0.956 | | | | Golf F — int4 | 2.65 MB | 0.936 | The decisions made during architecture configuration gave me several ways to reduce the size of the model. A bigram table is compressible and I can try code books and quantisations. There were many negative results using codebooks, mixed precision and I didn't do an exhaustive hyper parameter search. This was due to me being more interested with applying proven parameter golf techniques and improving on where on the accuracy gradient seq2feature would land between a full SAE pipeline and a simple embedding model. The int8 variant is lossless at 5.3MB and a smaller 2.6MB int4 probe shows 2 points of precision loss. I think I will accept the browser extension sized probe with a slightly better accuracy. The final seq2feature scores 0.116 higher than a 34MB bag of words while also being one sixth the size. This points to the amount of unoptimised memory usage in the naive bag of words and usefulness of transformer computation in this classification task. Competitive AI safety researchers will be rewarded for leveraging deep learning knowledge. Maximising MFU, avoiding loss spikes or enhancing rollout quality are all useful additions to the AI safety researchers tool kit. When the seq2feature probe lists the top 5 concepts it thinks fired on a span, it selects concepts that actually fired 19/20 times. The probe can say if any one of the 2048 SAE labels fired on a span with 90% accuracy versus 75% for a bag of words, 55% for a zero shot embedding and ~50% for random guessing. The probe was trained on agent relevant domains such as math, code and chat. There are many domains to which it was not exposed where we can test its OOD performance. Compared to our probe, the SAE teacher was trained on unfathomable amounts of data and so had the opportunity to learn from a diverse set of domains. On the topic of legal and medical domains the probe can say if a label is present in a span 75% of the time. Random labels succeed in this binary classification 50% of the time, or random chance, and the naive embedding scores 55%. Encouragingly when we check the top 5 concepts that seq2feature has detected in a span we can see that they were legitimately present in the SAE firings 94% of the time. This selection of labels also influenced the data that the probe was trained on. While labels such as grammatical syntax can be predicted in a OOD sample, the probe fails in the cases where a concept does not map to one of the probe's 2048 labels. This failure mode bubbles up as far as the SAE as it also can only detect concepts within its dictionary. I tested the probe on some OOD concepts showing the z-scores as the probe's confidence. input | probe's top concepts | analysis | |---|---|---| | | correct — caught the call and the ID numbers | | | reads the surface, | | | no chess concept — falls back to numbers/algebra | | | out of language — reads as code, and | The main limitation is that the label selection defines what concepts the probe learns. If we want a probe that learns to recognise safety relevant labels then these need to be selected from the teacher's labels as targets. The SAE can extract unique concepts from activations but the labelling process is how they are turned into usable targets. Distilling the model and SAE pipeline into a small probe is of course an exercise in extracting the SAE's hard learned and computationally expensive intelligence at a low cost and so transfer learning within the distribution is still beneficial. This SAE fires ~138 concepts per span. The probe is a 0.54 cosine match to the ground truth SAE teacher when we account for magnitudes. This shows the seq2feature is directionally correct in its predictions. The top 5 activations held 16% of the magnitude. If each concept on average held a uniform magnitude then the top 5 activations would hold a combined 4% of the total magnitude so this was still a trustworthy target - 4x concentration. As stated above, the coverage is only as good as the corpus. In comparison to a single-use probe that detects one concept, our distilled probe learns 2048 concepts and is cheap enough to monitor every token an agent outputs. The SAE and labels are interpretations of the LLM's computations and so as we have entrusted the SAE to be the ground truth for our probe we acknowledge that it is a proxy for the model's own representations. Modern models use Chain Of Thought to reason about problems and to call tools. For an LLM to execute code or do a web search the LLM needs to at some point output a signal to call a tool. This is usually denoted using tool tags written by the LLM in their CoT.