{"slug": "datasmith-automating-data-research-to-beat-post-train-bench", "title": "DataSmith: Automating Data Research to Beat Post-Train Bench", "summary": "Datology released DataSmith, an autonomous data research harness that improves base models by iteratively changing only training data, and introduced CurateBench, a benchmark for data curation research. On CurateBench, LLMs using DataSmith consistently outperform the same models in a general coding harness. Datology created the benchmark because PostTrainBench was unsuitable for isolating data curation effects.", "body_md": "# DataSmith: Automating Data Research\n\nAutonomous AI research is starting to have a meaningful impact. We are already seeing signs of rapid progress across the full spectrum of model training: [architectures](https://www.weco.ai/blog/parameter-golf-aiden), [kernels](https://www.recursive.com/articles/first-steps-toward-automated-ai-research), [optimizers](https://github.com/KellerJordan/modded-nanogpt/tree/master/records/track_3_optimization), and, increasingly, [fully end-to-end training loops](https://posttrainbench.com/).\n\nAt Datology, we have long believed that training **data is the** **most important determinant of model quality**. Thus, data curation is a prime target for autonomous research.\n\nToday, we are excited to share a first glimpse of **DataSmith**, our autonomous data research harness. It sits on top of Datology’s curation stack and runs the data research loop end-to-end by proposing data interventions, executing them through scalable curation and training pipelines, diagnosing model failures, proposing new hypotheses, and deciding what to try next.\n\nBefore we get into how DataSmith works, we want to show you how well it does. After that we will walk you through the benchmark, and then the harness itself. On a benchmark that evaluates how well an autonomous agent can post-train a base model **by changing only its training data**, LLMs running on top of DataSmith consistently outperform the same models in a general coding harness.\n\n### Glossary\n\nThere are a lot of terms that we will be constantly using in this blog, and below is a shared vocabulary as we walk through this project.\n\nTerm | Definition |\n|---|---|\n| The model that gets trained: |\n| The model that does the reasoning in the outer loop, deciding what data to try next: |\n| The tools and software that the Harness LLM has access to. For instance, Datology’s curation stack natively provides for training and eval launchers, and built in curation algorithms like decontamination, filtering, clustering, synthetic generation, etc. |\n| The Harness orchestrates the LLM in the working environment. This includes the working loop, declaration of subagent roles, prompts, memory management, and the skills that let it interact with the environment ( |\n| LLM + Harness + Environment together. |\n\n## 1. CurateBench: an autoresearch benchmark for data curation\n\nWe are interested in whether autoresearch agents can train better models by iteratively improving only the training data. To study this, we set up an environment in which the base model, optimizer, training configuration, and evaluation are fixed, while the agent can only change and submit the training data.\n\n### 1.1 Why do we need a new benchmark?\n\nWe initially planned to use [PostTrainBench](https://posttrainbench.com/) (PTB), which benchmarks how well agents can autonomously improve base model capabilities through post-training. However, three parts of its design make it unsuitable for our goal.\n\n- PTB trains a different model for each benchmark, so each capability is optimized in isolation. In practice, the post-training stage trains a single model on a unified data mixture that simultaneously advances the performance on multiple benchmarks. This design choice misses important aspects of data mixing, and cross-domain performance transfer.\n- PTB starts from an empty sandbox, so agents waste a large proportion of the GPU budget building infrastructure rather than training models. In many cases, more than 25% of GPU time is spent on setup and overhead. Instead, we want to measure agents’ research ability given mature data infrastructure.\n- PTB gives agents an open intervention space allowing the model to change everything from the training algorithm (SFT, GRPO, on-policy distillation), the training configuration, the compute budget, to the data. This means that the training algorithms, FLOPs, and recipes across different sessions are often not comparable, making it difficult to attribute a strong run to data curation alone.\n\n### 1.2 CurateBench\n\nCurateBench takes a huge amount of inspiration from [PostTrainBench](https://posttrainbench.com/), but isolates the effect of data by fixing the optimizer, training recipe, then evaluating a single base model across multiple benchmarks. The goal is to reward data choices that make a model broadly better rather than choices that optimize for an individual benchmark.\n\nWe use the same three base models and four evaluations as [PostTrainBench Lite](https://deploymentsafety.openai.com/gpt-5-6/posttrainbench-lite): **Qwen3-1.7B-Base, Qwen3-4B-Base, and SmolLM3-3B-Base**, evaluated on **AIME2025, GSM8K, HumanEval, and BFCL.**\n\n### Setup Details\n\nEvery training submission uses the same Supervised Fine-Tuning (SFT) recipe:\n\n```\nTraining steps: 350\nGlobal batch size: 64\nSequence length: 16,384\nTokens seen: 367M\nLearning rate: 1e-5\nCompute: 8×H100\n```\n\nThe agent is limited to only changing the SFT training data. It is allowed to search the internet to download any relevant new datasets or generate synthetic data via API access to Qwen3-30B-A3B-Instruct-2507 or a previously trained model. In any case, training data assets must be decontaminated against the eval questions before use. In each autoresearch session, we allow up to 20 training job submissions and do not limit the total time for research.\n\n### 1.3 Datology’s Curation Stack Establishes a Strong Lower Bound for Autonomous Data Research\n\nGood infrastructure can be one of the biggest accelerants of good research. Before we introduce DataSmith, we ask what a general coding agent can achieve with Datology's curation stack. To this end, we run Claude Code with each of the four Harness LLMs, giving it access to our curation environment. This will later serve as the baseline against which we measure the value of DataSmith's research orchestration.\n\nFor a public reference point, we compare the baseline against PostTrainBench v1.0. Across the two matched configurations, the CurateBench baseline averages 4.9 percentage points above PTB, despite having a significantly restricted action space. These restrictions prevent using RL, changing training budgets, or hillclimbing on a single evaluation while ignoring regressions elsewhere. These results simultaneously validate CurateBench and show that Datology's curation stack already helps general coding agents achieve stronger results than they do in PTB's environment, before DataSmith adds any research orchestration.11.We use v1.0 because it supports external API-based synthetic generation, making its setup closer to the CurateBench environment.\n\n## 2. DataSmith is a strong research harness for data curation\n\nTo isolate the effect of the harness, we hold the curation environment fixed and change only the harness. For each Base Model, we compare matched runs of the same Harness LLM in Claude Code versus DataSmith.\n\nDataSmith outperforms Claude Code in all twelve matched comparisons, by 5.1 percentage points on average. The improvement is consistent across all three Base Models and all four Harness LLMs.22.[SmolLM3-3B-Base: 11.2T pretrain + 100B long-context mid-train. SmolLM3-3B: +140B reasoning mid-train (35B × 4) and +8B SFT (1.8B × 4). DataSmith’s 734M is ~200× less than that ~148B. From the SmolLM3 Tech Report.](https://huggingface.co/blog/smollm3)\n\nOn SmolLM3-3B, the strongest DataSmith run (with Opus 5) beats the released instruct model by 3 percentage points on the reported four-evaluation suite.It starts from the same base checkpoint and uses 734M post-training tokens, whereas the instruct model is trained with 148B post-training tokens, roughly 200× fewer.\n\nIt is also noteworthy that the choice of harness can also outweigh the choice of the Harness LLM. Opus 5 is the strongest and most expensive Harness LLM in our comparison, yet GLM 5.2 with DataSmith outperforms Opus 5 with Claude Code on all three base models. This is the second time we see the same pattern: what surrounds the Harness LLM matters more than which model it is. First, Datology’s curation environment provides the data-action space, and DataSmith helps the agent make better use of it.\n\n## 3. DataSmith\n\nDataSmith is an AutoResearch harness for data curation that can run over long horizons. It builds on top of Claude Code and Datology’s curation environment. Claude Code provides tool use, context management, and subagents, while Datology’s stack exposes training, evaluation, and curation jobs behind simple commands.\n\nDataSmith adds the ingredients that a long-running research process needs, like a persistent loop agent, specialist research agent roles, data curation skills, and a memory management system that tracks the state of every experiment.\n\n### 3.1 A Multi-Agent Research Team\n\nFirst, let us dive into the roles of various subagents in the multi-agent research team that DataSmith orchestrates. At the core, the **loop agent** coordinates the run and stays persistent till the goal is achieved. It observes events, dispatches tasks, executes decisions, and polls long-running jobs. It is supported by four specialist subagent roles, which are implemented as skills:\n\nThis separation lets the strategist maintain a research direction while individual investigations run in parallel. New evidence can arrive from a completed experiment while other agents are still preparing data for the next one.\n\n### 3.2 Skills and Tools as the Interface to the Curation Environment\n\nDataSmith's research roles interact with Datology's curation environment through skills. Each skill packages the context and tools required for a particular operation, giving the agent a consistent interface to the curation workflows it needs during research.\n\nThese capabilities are exposed as tools to the agent for `onboarding`\n\n, `profiling`\n\n, `filtering`\n\n, `deduplication`\n\n, and `decontamination`\n\n, and as skills for synthetic data generation, training, and evaluation. With these tools and skills, agents can directly operate our infrastructure to curate data rather than rebuild from scratch for every experiment.\n\nFor example, Autosynth agent is initiated with a data synthesis blueprint designed by Strategist agent. Autosynth agent can then assemble different tools to build an end-to-end synthetic loop: data generation, answer verification, failure repair, and eval decontamination. After enough data is synthesized, it will register as a trainable asset and send back to the strategist to determine how to use for the experiment.\n\n### 3.3 Long-Horizon AutoResearch Loop\n\nTo enable long-horizon research, most existing works ([AI Scientist](https://arxiv.org/abs/2408.06292), [Automated W2S Researcher](https://alignment.anthropic.com/2026/automated-w2s-researcher/), [ScientistOne](https://arxiv.org/abs/2605.26340)) use a pre-defined loop in which subagents are called in a fixed sequence. DataSmith instead uses a loop agent to orchestrate which subagents to call and what information to put into their context, following the principle of the Bitter Lesson by relying less on a hard-coded workflow.\n\nA typical run follows:\n\nBut the loop can deviate from this workflow depending on observation. After each training run, scores update the leaderboard, and the loop agent decides how to proceed. For example, if the current strategist begins to stagnate, the loop agent can spawn another strategist with a different context, while concurrently launching diagnosis agents to inspect the eval failures.\n\nWe use an experiment tree to track this history: each training job submission corresponds to a node containing its data recipe and evaluation results, while branches represent alternative research directions. This gives the agent a persistent record of what research ideas have been tried and how each experiment relates to previous runs.\n\nTraining and data preparation can proceed asynchronously. The agent can submit up to three concurrent training jobs while preparing new datasets for future runs. As soon as a training slot becomes available, it can launch another job if the required data is ready. The loop agent polls completed training jobs and feeds their evaluation results back to the subagents to decide the next move.\n\nTogether, the orchestration of specialized subagents, the experiment tree, and the research loop enable long-horizon research beyond the constraints of a single context window.33.DataSmith turns data curation into a long-horizon research loop, where specialized subagents produce data and research insights, and every experiment builds on what came before.\n\n## 4. Research Taste in DataSmith Agent\n\nA [common failure](https://edotenv.com/blog/alpha-autoresearch#what-are-the-common-failure-patterns-in-the-rollouts) of general coding agents tasked with data research is that they are too eager to launch experiments on the cluster: launch a training run, read the scores, make a micro adjustment to the mixture, and launch again. The AutoResearch loop is rather shallow: the agent keeps locally optimizing the recipe it started with instead of digging into the evaluation outputs, questioning its own approach, and pivoting when performance plateaus.\n\nDataSmith has no explicit research taste built into it. Even so, it spends 2.6× more reasoning tokens than a general coding agent, holding the harness and LLM fixed. DataSmith reasons about how to design experiments that test its research hypotheses and can self-critique its current research bets. In other words, it “runs fewer experiments and thinks about them more.”\n\n### 4.1 DataSmith explores wider\n\nTo understand how agents research between training jobs, we measured five metrics across all sessions: how many tokens they spend reasoning, how many new datasets they try, how often they delegate to subagents, how much research infrastructure they build, and how many distinct research bets they make.\n\nWhatever the base model, DataSmith leads on every axis: on average\n\n**6.6× more new datasets****48.4× more subagent calls per session****45× more infrastructure code lines written****1.8× more research ideas**\n\n### 4.2 DataSmith makes the exploration pay off\n\nWider exploration is only interesting if it is worth what it costs. Let’s use the SmolLM3-3B run with Opus 5 as the Harness LLM as a case study for this comparison.\n\nSpending more on the agent makes training more compute-efficient.DataSmith costs more per submission on the agent side, but it achieves a higher benchmark score per GPU-minute spent. At the same GPU time, it ends up 4.9 percentage points ahead of Claude Code. As training runs get longer and GPU costs grow, squeezing more value out of each run matters even more. We expect the cost of compute to far outpace the cost of thinking.\n\nDataSmith keeps searching after the baseline plateaus.Across all twelve pairs, every DataSmith run was still gaining in its final quarter. A sample of the trajectory can be found in the plot above. This suggests that the larger exploration tendency allows it to come out of plateaus.\n\n## 5. Towards a \"move 37\" for autonomous data research\n\nIn its 2016 championship match against Lee Sedol, AlphaGo made a baffling play on move 37 that shook human spectators and changed the game of Go forever. DataSmith is still in its infancy and has yet to make an equally groundbreaking move, but it has already started making research decisions that are surprising and effective.\n\nHere is one that we found especially striking.\n\n## 6. What’s next?\n\nToday, DataSmith begins automating the search for better post-training data in a research process that is still largely designed by human researchers. Our long-term goal is to make this process recursive by learning from experience: each experiment should teach DataSmith not only which data interventions work, but also how to design new specialist agents, build better tools, and improve coordination. DataSmith is an early step toward research systems that do not just accelerate and automate discovery, but learn from trials and failures how to become better researchers.\n\n## Citation\n\nPlease cite as:44.*Equal contribution.\n\n```\nAochong Oliver Li*, Haoli Yin*, Rishabh Adiga, Darren Teh, Siddharth Joshi, Bogdan Gaza, David Schwab, Ari Morcos, Pratyush Maini, \"DataSmith: Automating Data Research\", DatologyAI Blog, Aug 2026.\n```\n\nOr use the BibTeX citation:\n\n```\n@article{datology2026datasmith,\nauthor = {Aochong Oliver Li and Haoli Yin and Rishabh Adiga and Darren Teh and Siddharth Joshi and Bogdan Gaza and David Schwab and Ari Morcos and Pratyush Maini},\ntitle = {DataSmith: Automating Data Research},\njournal = {DatologyAI Blog},\nyear = {2026},\nmonth = {August},\nnote = {https://www.datologyai.com/blog/datasmith}\n}\n```\n\nWe thank Arya Gupta, Anshuman Suri, Dhruba Ghosh, Jasper Tan, Luke Merrick, and Matthew Leavitt for their thoughtful comments on the draft.\n\n## Ready for better data?\n\nLet’s make models better through better data, automatically.\n\n[Book a Call](/contact)", "url": "https://wpnews.pro/news/datasmith-automating-data-research-to-beat-post-train-bench", "canonical_source": "https://www.datologyai.com/blog/datasmith", "published_at": "2026-08-18 15:56:01+00:00", "updated_at": "2026-08-18 16:11:56.855000+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools"], "entities": ["Datology", "DataSmith", "CurateBench", "PostTrainBench"], "alternates": {"html": "https://wpnews.pro/news/datasmith-automating-data-research-to-beat-post-train-bench", "markdown": "https://wpnews.pro/news/datasmith-automating-data-research-to-beat-post-train-bench.md", "text": "https://wpnews.pro/news/datasmith-automating-data-research-to-beat-post-train-bench.txt", "jsonld": "https://wpnews.pro/news/datasmith-automating-data-research-to-beat-post-train-bench.jsonld"}}