This week's Paper Club is focused on data. As models have scaled across tasks and languages, old assumptions about training and evaluation are starting to break. So we gathered three domain experts to break down the challenges and frontiers of training data, benchmarks, and multilingual pre-training.
Timestamps #
00:00 - Francois Chaubard (YC): Why a whole night about data?9:51 - Vincent Sunn Chen (Snorkel AI): The Art & Science of Benchmarking Agents28:44 - Volodymyr Kuleshov (Inception AI): Inception: Diffusion Language Models for Production40:26 - Shayne Redford (Anthropic): ATLAS: Practical Scaling Laws for Multilingual Models
Transcript #
Francois: Welcome to YC Data Club this time. I hope you guys like the little picture that we have this time. Paul Graham didn’t like that I was slacking off in my little crown over there, but he does know my name now, so that’s good.
All right, so why data? Why does data warrant its own night to talk about? In 2016, I left my PhD program and I started Focal Systems. And back then the notion was that data is a commodity. I already have downloaded ImageNet. How much more data do I possibly need? And there was lots of chatter about Scale AI being worth a billion dollars. There’s no chance. And there was a lot of VCs — except who was the guy that did Scale AI’s Series A? I think it was Levy at Accel. Besides him, everyone said the terminal value of data businesses was zero. And since then, it’s like $100 billion in market cap creation. So rest assured, the entire VC community has changed their tune on that.
And why? Well, the interview question I would typically ask at Focal — I still ran the deep learning team even until I left nine years later — was, let’s say I had to interview these PhDs from Stanford or MIT, super smart people. And I would say, okay, you train your model, you’re at 85% F1 score on hotdog, not hotdog. What do you do next? And if your answer was, oh, I would go read some papers, I would try ReLU versus GELU, I would add more layers, I’d do that — nope, next. Literally, no, no, no, no, no.
And the reason why is because you have to look at the data. The correct answer is you look at the data, you classify, you look at all the false positives, you look at all the false negatives, you put them in Pareto buckets, you say, what’s the biggest issue? And for us, for in-stock, out-of-stock, it’s like, okay, it’s when the fridge has some fog on it. We can’t see through the fog, or when there’s a person in the way, or when something else is happening. The number of layers was not going to change if there’s a person in the way and you can’t possibly tell if there’s an in-stock or out-of-stock. And so it’s just looking at data.
Andrej, I would say, was the most on this. And he famously did the first measurement of what human level is on ImageNet, where he actually looked through and tried to see how good he could be. And we’re not that good. I can’t tell the difference between an Eskimo sheepdog and a Siberian husky. I’m not good at that. And 30, 40% of ImageNet is dogs, are types of dogs. And people kind of don’t know that stuff.
And so the data that you’re training on ends up being much more important than the architecture itself, especially when the architecture is very expressive. In your PhD, he says, he spent maybe 5% of his time thinking about data and 95% of his time focused on the architecture. And then when you go into production, that flips. He was saying 25, 75%. I think nowadays it’s like 3%, 97%. And it’s really about the data now that we have the transformer and it works really well.
And why is this? Why does the data matter so much? It’s because in your nice little ImageNet train set, everything is really nice. The train distribution is beautiful. It’s amazing. And then when you launch it into production — raise your hand if you’ve ever had this experience — that happens. And then you start getting a bunch of points in the test distribution that you didn’t have in the train. And the model pukes itself and it doesn’t work.
So to automate the economy, if this is GDPval, we need both expert data and expert RL environments. RL environments for the verifiable rewards. And then for the preference-style data, very subjective: do you like this design? Do you not like this design? Is this more usable or less usable? The code may work, but the coding style is awful and it’s unmaintainable code. For things like medical, two doctors don’t agree on the same answer for a given patient, and same thing with judges. And so these are preference-style answers, and that runs a lot of the economy.
And the right way to think about data sets and RL environments, for sure, is these are products. If you talk to Greg at ARC-AGI, the amount of craftsmanship that goes into each game — that Prime Intellect just crushed, to be fair — is incredible. It’s very, very difficult to make these data sets really, really good, fully comprehensive.
And even for Focal, such a simple thing as in-stock and out-of-stock, the amount of time that was spent on how do we handle product pushers? It looks like a product, but it’s just pushing product to the front. And so should we have that be a separate class in the ontology, or should we classify that as an out-of-stock? Because it technically is an out-of-stock, but it’s going to confuse the model, because an out-of-stock and a product pusher look very different. So how do you handle all this stuff? It’s very, very difficult. And a lot of people think that they’re just zip files, and it’s really not.
And then the other thing too that I deal with a lot in the batch is, let’s say you are automating Salesforce and you have a bunch of traces and you have screen captures of people using Salesforce. So you have a hundred billion hours, you have all the data that you could possibly capture. Then Salesforce changes the UI. Then what? You better get some more data, right? Because it’s not going to work. And so you can have all the data, you still need more data. And so this idea that the terminal value is going to be zero is just so, so wrong.
Brendan just put in this slide at Sequoia and I really agree with it. It is the bottleneck right now. It is not the architecture. It is not GPUs. It is not watts. It’s largely why ChatGPT still — I just did this analysis, I posted it on my Twitter — there’s no model that beats random on predicting next-seven-day returns on the S&P. Literally, if you just roll a coin, it’s better than every single AI model that you can use in production right now. And so if we had a bunch of traces from Goldman Sachs traders that are definitely better than random, I would think that they would do better, but they don’t have that. And same thing with accountants, doctors, lawyers. You need experts in each one of these domains for us to match or exceed human capabilities on it.
And like I said, in the last 10 years, despite most venture capitalists turning a blind eye to this, it’s produced over a hundred billion dollars of market cap. It’s probably been one of YC’s best categories that we’ve invested in.
And so, what is the right mental model to think about this? I don’t know, but I propose something here. If I were Apple and I manufactured these things and I wanted them to be able to deliver groceries to your door, or to bring a burrito to your house, or bring a car and pick you up, or turn on a flashlight, or give you directions — you need these things called apps. And if you want the LLM to be a good doctor, accountant, lawyer, carpenter, trader, or therapist — a lot of people are using it for that as well — you’re going to need data and RL environments. And if you don’t have up-to-date, very good curated data, then your LLM can’t do those things.
And so a lot of people say, how many possible data companies can we have in this space? And the answer is, how many apps do you have on your phone? That’s probably the answer. I really do think about it like that.
And then even better on this analogy is, does it make sense for there to be an Instacart only for Apple and an Instacart only for Android? Or should it probably just be one company? Or should Apple get into the Instacart business and start delivering groceries? It doesn’t make any sense. Some of these, it makes sense. Apple should and Android should have a flashlight functionality, because that’s kind of easy to do. But for a lot of things, it’s really, really difficult to stand up the entire Instacart network and then start competing with Instacart if you’re Apple. Why? That’s just not what you are great at.
And in the same way, going really deep into being a great doctor, dealing with HIPAA and all that stuff — should Anthropic, OpenAI, Google, Apple, all these companies all get really great at curating an AI doctor? Or should you just use one company that does a great job of creating an RL environment, has a network of doctors, curates all this preference data, parallel data for a given patient — here’s one trajectory, here’s another trajectory, and a doctor is preferring one or the other? Probably just do that. Have one company that just focuses on that. And that’s what we’re seeing.
All right. I’m very excited by the speakers that we have tonight. May couldn’t make it. She got sick, unfortunately. We’re going to try to squeeze her into the next one. But we have Vincent Chen, who is a Hazy Research alumni, one of the founding team members of Snorkel, grew it to a billion dollar market cap now and well beyond, I think. Volo, founder of Inception Labs, Cornell professor, and expert surfer. And then Shane, MIT, PhD student, just defended, focused on pre-training, founded the Data Provenance Initiative, and then recently joined Anthropic. Please help me give a round of applause to our speakers.
Vincent: Thanks, Francois. Wonderful. Thank you, Francois, for the awesome introduction, and thank you for giving data its flowers. I feel like we’ve been trying to shout from the roofs about this, and it’s awesome to see it come front and center these days.
My name is Vincent. I’m one of the founding team members at Snorkel. I started our frontier lab business at the company, and now I lead research on benchmarks and evaluation. I’m going to talk about scaling expert supervision. I’m cheating a little bit, but I’ll talk about a few research themes and how they’ve evolved over the years.
We’ve been working on data for over a decade now at Snorkel, dating back to our roots at the Stanford AI Lab. And our key thesis is that scaling expertise — actually giving leverage to experts in the field, doctors, clinicians, journalists, people who actually have the spec in their head for what good looks like — is the real bottleneck for building really effective data sets. And effectively, that’s what all of our research is about at the company.
As I mentioned, we’re a frontier lab focused on data in particular. We’re super excited to be partnered with effectively every global frontier lab and Fortune 10 enterprises on the data problem. And each one of these engagements is anchored on this key bottleneck of scaling expertise. How do we actually get the judgment and knowledge out of these people’s heads and into the data sets that they’re working on?
So I first want to start by framing the problem and how it’s evolved over time. Expert supervision — the process of taking what is in experts’ heads, what is in messy documents or knowledge corpuses, and putting it into really effective data — is a real bottleneck. There’s a real process that goes from, okay, this is raw data, to something that can be used for evaluation and training.
Data 1.0, as I’ll call it, was basic labeling for question answering, for example. These might have been thumbs up, thumbs down preference labels, basic prompt and response pairs. And let’s say it’s 30 seconds of human judgment to produce one of these labels.
As Francois alluded to, the shape of data today is evolving significantly. Now we have entire worlds that we’re building against that represent tasks and rubrics and verifiers and really nuanced grading mechanisms, all packaged in Docker environments that actually need to represent the types of spaces that these agents are going to operate in. And these could take single digit to triple digit hours for humans to develop on their own. And so the key bottleneck here is really, how do we scale expertise so that we can effectively build these types of data sets, when there are only so many of these experts in the world with increasing complexity, and they only have so much time in a day?
So let’s start with Data 1.0. This is a problem that I believe still exists today, but when we started Snorkel, I want to share a little bit about how we thought about this problem. The problem of manual labeling, with the ImageNet example that Francois gave — there are a bunch of problems. One, it’s not a scalable approach. Think of the cost of manual labeling as an O(N) problem. For every single data point, you need a manual label, you need new cognitive effort to actually develop that. Two, these types of approaches aren’t robust to noise. You can imagine that the only way to actually reduce noise in these settings is to introduce redundancy. So you need a K-by-N relabeling effort to actually reduce noise. With changes in spec, schema, task definition, you need to start from zero. And critically, there’s no provenance. There’s no notion of what is my rationale, how did I actually think about this problem?
And so this was the key challenge that we faced up to a decade ago and still face today in many ways, where the problem of manual labeling only compounds in expert domains. Think of an MD-PhD or a cardiologist in a very niche subdomain. How do you actually get them to produce these large corpuses of MRI data sets or EHR data? It’s actually fundamentally intractable in a lot of these settings.
And so the line of work that we worked on here, we call it data programming. Our CEO Alex was a grad student at the time, and we all kind of contributed to this work. The key intuition here was we want to encode expert supervision in software. And there’s a few benefits when you do that. You get the scale of software — you now get to label data programmatically. You get the adaptability of software. You can refactor it and adjust it over time as your spec changes. And critically, you can actually audit it and have conversations about it and collaborate on it.
And so in a nutshell, a bunch of our key grad school work was this. One, on the left-hand side, the main idea was to first model expertise as labeling functions. This could be a number of different forms — it’s a pretty general abstraction — but the main idea is, if you express the rationales or reasons why people are labeling spam versus not spam into specific heuristics, this helps get that knowledge out of people’s heads and into a form factor that’s actually way more reproducible and scalable. Number two, in the middle, the challenge with this is obviously that these signals, these sources of supervision, overlap. They’re inaccurate. And so we introduced a bunch of theory and work to actually work through this notion of weak supervision. How do you actually understand quality in these settings where signals are overlapping and your sources are weak or aren’t fully ground truth? And at the end of this, taking these denoised signals and using them in end models that are noise-aware helps generalize this beyond the coverage of the initial labeling functions.
So one click down, the intuition behind the label model, without getting into too much of the gory details. Number one, the goal here is to model each voter’s accuracy in a fully unsupervised way. So the analogy I like to share is, hey, you have a number of students in your class with different levels of skill or quality. You don’t have a grading key, you don’t have an answer key. How do you estimate the true ground truth answer when you only have the votes from each of these students? That’s the fundamental modeling problem you’re trying to solve. You want to learn this noise model and understand each source’s quality effectively, given that you don’t have any ground truth in the first place. So step one is to learn the voters’ accuracy. Step two, this is Bayes — you’re trying to compute the probability of the ground truth solution given each one of those voters’ actual votes. And then three, if you train a noise-aware model, you’re able to actually extend the coverage of this label model and minimize some of the noise in broader distributions and scale this a lot more effectively.
So I’ll zoom out for a sec. The main intuition here was that a lot of our work in the early days was about scaling expertise via introducing labels and expert judgment as software. And one of the key bottlenecks there was actually denoising the sources of signal that these came from, in a regime where you were missing ground truth. And some of these techniques we still use today in a lot of our production settings.
So I want to talk briefly about Data 2.0 now. As the surface area for data and frontier progress has expanded, it’s grown in complexity and importance as well. So let’s use coding as an example. I have plotted on the x-axis complexity. Input complexity — how complex are your prompts and the specs that you’re giving models? Output complexity, environment complexity — are you working in just Q&A settings now, or are they fundamentally new environments where you give it YOLO access to your desktop? And on the y-axis, we have sequence length. How long are these agents actually working in practice?
Now, you might think for a domain like software coding that the evals are actually saturated, the data is actually saturated. And I would contend that as the frontier advances, the data challenges actually continue to grow. We’ve seen basic evals like HumanEval saturate at this point, which is true, but we’re still seeing a big spectrum of terminal-based agents continue to grow. ProgramBench — I think the latest model just cracked 1%. We introduced a benchmark called Senior SWE-Bench, which I’ll talk about in a second. But as the complexity and responsibility of agents grows, the importance of the data that you’re producing to both train and evaluate these models also grows.
With this in mind, I want to anchor on some recent work, which was Senior SWE-Bench. We were very humbled and privileged to work with the original SWE-Bench team out of Princeton on this. The key idea here was, we’re already using coding agents like senior engineers. We’re vibe coding, we’re trusting them to make architectural decisions, to refactor entire swaths of code, but we don’t have good ways to evaluate them. We’re still evaluating them as junior engineers. And so the key challenge we wanted to solve was that we wanted to build a benchmark, we wanted to build data sets that were really representative of actual work that senior engineers would produce, scale the expertise needed to produce and build these types of data sets, and build a really high quality benchmark that represented all of these factors. So I’ll talk you through some of the methodology we used to get here. Obviously we won’t get through everything, but hopefully it builds some intuition for how some of the methods we’ve used have evolved over time.
A little bit about the tasks themselves. In Senior SWE-Bench, the task design specifically required many points of expert supervision or expert touchpoints. On the left-hand side, this is a Harbor task. We’re big fans of Harbor as an eval framework. These are natural language instructions. The key diff here was that, if you’re familiar with SWE-Bench or traditional coding benchmarks, if you actually look at the data, if you actually look at the tasks, a lot of the samples are very overspecified, very PRD-level guidance for what to do, what not to do. We wanted to represent more realistic and higher-level abstractions in terms of how we actually interact with these agents in practice. So more of a Slack message — “hey, here’s a dump of logs, figure out what to do with it,” or “here’s a few user stories about an idea that I have, can you help me implement this?” So actually injecting expertise and realism into the instructions was the super non-trivial part of building this.
On the right-hand side, you see a whole reward phase, which we spent a lot of time designing and specifically calibrating as well. The intuition here is we wanted to capture a number of different components of how you actually measure these agents in practice. So not just, is this correct or mergeable, but does this exhibit taste? Does this exhibit what you’d trust a staff or senior or principal level engineer to actually do in your code base completely unsupervised?
Let’s go a click deeper into the specific reward design. This was a bit nuanced, and again, an area where we wanted to get creative in scaling expertise. So we had Snorkel engineers and researchers — we have a really strong network of very senior principal, staff level engineers, but not unlimited time. And so we wanted to find ways to scale their intuition and effort rather than having them hand-grade every single one of these samples and agent trajectories.
The axis we tried to articulate here is that there’s a trade-off between reliability and flexibility when it comes to traditional verifiers and rewards, especially in the software domain. On the top, you have high reliability and low flexibility. These are pre-written verifiers — think of these as unit tests. These are tests that reliably execute against your code base. They’re actually running against the code, so you know that they’re doing the thing. But the challenge is they don’t actually adapt to any solution. If you have a POST /profile implementation versus another way to update a user’s information, you might actually have a false negative in the verifier. You might penalize behavior that is fully okay and fully valid if you’re too constrained in your verification approach.
And so on the other side of the spectrum, low reliability, high flexibility, you have LLM judges. This is a very common and effective way in many cases to actually adapt to different solution shapes. The code can be a little bit more flexible. You’re actually just asking an agent or LLM to look at the code and specifically grade it using its own calibrated or uncalibrated intuition. And obviously the challenge here is that this can be really unreliable, because you’re not really exercising any of the solutions. You’re not exercising the code path directly. You might be just asking the agent to take a glance at the code, and it’s hard to actually control for false positives, where you may over-reward plausible but incorrect solutions.
And so we found a middle ground for this, and introduced this notion of a validation agent, which specifically tried to capture and scale the expertise of a bunch of our senior researchers and engineers and experts in our community and network — one, to express the user stories or specification of what they thought good looked like, and two, use that specification to then write deterministic tests against the code surface that was actually relevant to the solution space. So a simple idea, but it did take a lot of calibration and effort to actually come up with this notion of, how do we actually capture the expertise of someone who’s defining user stories for a particular task, while actually exercising the real code path?
So in a little bit more detail, what does this actually look like in practice? The input to this validation agent — think, hey, I have Codex, I have Claude Code. It generates a patch against my instruction set. The expert produces what we call the validation spec. So this was just a few bullet points, a Slack message of, okay, here are my user stories, here are a few functional or non-functional requirements. Not so much that it was overspecified and overly prescriptive about a specific solution, but high level enough that it actually captured the space of what we wanted the model to do or not do. We then had a validation agent take the patch and the validation spec and specifically implement test scripts. These were deterministic scripts that would execute and run over the code itself. And at the end of this, we would score against both the execution and use a judge to ensure that the runs were sane, or resulted in collusion or reward hacking or some sort of behavior that we thought was not valid for the trial.
And so this idea of encoding expertise in a way that was still scalable was really the crux of how we tried to design this validation agent to map to dynamic implementations in a specific code base. We did a lot of work to calibrate this. We aligned these against specific test engineers internally in our network. We introduced LLM judges that measured things like fidelity, completeness, collusion. And ultimately each test was parameterized so that in the validation spec, you can think of it as a ramble of, hey, what is the shape of the type of use cases, edge cases that we want to capture, without having to write the full test at the end of the day, which could take days to weeks to months.
So this is live now. And one thing I want to point out before talking about the results, which are kind of fun — we introduced this notion of a tasteful pass. This is a new metric that we introduced in this benchmark to specifically measure not just correctness or mergeability, but a more nuanced definition of, is this actually code that aligns to your code base practices? Does this actually map to how a senior engineer would act in practice? Is bloat minimized in terms of patch size? We merged that with notions of correctness to actually rank a bunch of these models. And remarkably — and we double and triple checked this — Fable, Opus and Sol are all tied for first place as of last week. This is pretty exciting because it means that the Pareto frontier is definitely getting pushed in these cases. This is a live dashboard and a living benchmark that we’ve been keeping up to date, so definitely reach out and check it out if you’re curious about it.
This is some of our recent work on this topic, and so I want to give some flowers to the team here. Henry, our co-founder, was the lead. We were very humbled to work with Karthik’s group at Princeton, who led the original SWE-Bench, and our chief scientist Fred, who has a lab over at Wisconsin-Madison, and some of his students were also involved.
And so I’ll zoom out and wrap up here. As we enter Data 2.0 and see a lot more complexity, we need a lot more data research. It’s not just a problem of throwing humans at these tasks these days. We need a lot of thoughtful design to give these experts — software engineers, doctors, lawyers, people who actually understand the spec — leverage to exercise their own supervision and judgment in these data sets. And this slide at least shows a view of all the axes that complexity is going to continue to grow in. The environments are going to get more complex and dynamic. The outputs are going to get more and more unverifiable and nuanced and subjective. And the autonomy of these agents is also going to continue to extend. And you could imagine every one of these axes introduces compounding complexity. And so it’s really a research question, it’s really a research problem, to think through how do we scale and provide experts more judgment. And that’s really what we’re focused on and encourage the community to work on as well.
And so here’s a very concrete ask. We need more benchmarks, and we think this is actually a really high leverage way for folks to drive new data research. We’re super humbled to work with a number of the folks on the screen, from Agents Exam over at Berkeley, Continual Learning Bench, OSWorld, the Terminal-Bench folks. We’ve learned a lot working with these folks. And in general, we’re very excited to see a lot more benchmarks. So we like to put our money where our mouth is. We have this notion of open benchmark grants. Hopefully this is actually a resource to folks in the community. We’re very excited to accelerate and focus on more research that is out there and help you accelerate your data development and partner as a research team. But yeah, we’re really excited about the future of benchmarks and data for Data 2.0. Thanks so much.
Francois: All right, next up we have Volo.
Volo: Okay, great. Thank you for having me. Super excited to be here today to tell you about some of our work, some of the work that we’ve been doing at Inception on training real world large scale diffusion language models — with my co-founder Stefano and Aditya and a really, really talented team of engineers and researchers.
So in short, what we’re working on is a new generation of language models that is powered by diffusion. And what that means is that instead of generating tokens one at a time sequentially left to right, a diffusion model starts by generating tokens all at once, starting from an initial guess of the sequence. So starting from some kind of noisy initial version of the output, and then producing all the tokens in parallel over multiple steps of refinement. We’re very excited about this technology. We think it’s going to be the future of language models.
And in particular, a very important advantage these models have today is speed. Because these models can produce multiple tokens per step, they can produce many more tokens per second, and they can reach speeds of over a thousand tokens per second, which is way beyond what is achievable with traditional autoregressive modeling.
There are all kinds of applications for ultra-fast inference in real-time AI. One area that we’ve been working in a lot is real-time voice. So voice agents, for example, in customer support, but also in other domains like education. These are agents that synthesize voice and talk to you in real time. Usually the state-of-the-art voice pipelines are still built on a cascaded architecture, where you have a speech-to-text system, a text-to-speech system at the output, and in the middle you have an LLM. The overall latency of the system is highly bottlenecked by the LLM. This is a critical, critical piece of this workflow. And if you can make LLMs go much faster — if they can go at a thousand tokens per second, like our Mercury 2 models are able to achieve — then you can make real-time voice interactions feel more seamless, or you can deploy a bigger model, or you can get the model to reason for longer, which will then significantly improve your quality.
So this is a slide that shows the performance of Mercury 2 on voice benchmarks compared to other models. And what you see is Mercury 2 draws a new Pareto frontier of quality and latency — latency on the x-axis, quality on the y-axis. So we have a new Pareto frontier that’s achievable with diffusion. I also want to highlight some other models here that are baselines. So here, for example, we have a 120 billion parameter gpt-oss model running on Cerebras, and a diffusion Mercury model can achieve both higher quality, at least as measured here by Tau-bench — and I’m going to argue on real data as well — and it can also run faster.
So this is really exciting, because Cerebras are specialized chips designed and built to make autoregressive models run really fast. If you have diffusion, you can achieve similar speeds on GPUs by having more intelligent software. So you can build software to make models really fast instead of hardware. That’s really exciting. You can deploy these models on a lot more GPUs that are more easily available.
But to build really good voice models, algorithms and diffusion is one component. The other part that’s really, really, really important is the data. You need data both for training the model and for evals. Right now, perhaps the most widely used and representative benchmark for running evals is Tau-bench. Tau-bench has a few problems. First of all, Tau-bench doesn’t really capture all the range of complexity that you see in real world data. If you look at a Tau-bench environment, it probably has an order of magnitude fewer specifications than you would have in a real world setting where a business would be doing real world customer support. There are fewer tools in Tau-bench. It doesn’t have certain kinds of specifications that you would want to provide. And then another problem with Tau-bench is that it’s extremely benchmaxed. If you go to your favorite benchmarking website, you’re probably going to see most of the models achieving scores that are in the 90s. But then if you run these same models on real world production logs, you’re going to get much lower scores, and there’s also going to be much more variance across models of different sizes. So data is a key, key ingredient of building the kinds of performant models that we want to build, in addition to the algorithms.
Now, our approach for getting good data for our models involves a system that we call TauForge, which synthesizes a wide range of realistic RL environments similar to Tau-bench, based on real data, based on real user interactions — either based on data that we’re getting from our data providers, or we also have partnerships with companies that are using this model in production. And so we get a lot of real world data that then allows us to synthesize Tau-bench-like environments across different domains, which can then be used for evals as well as for training the model.
So this slide gives an overview of TauForge. TauForge is an agentic system. It’s a harness that orchestrates a large number of different agents, and it produces synthetic RL environments that are similar to Tau-bench. The input to TauForge is a description of the kind of environment that you would like to generate. So here, for example, you could ask it to generate a banking environment, as well as artifacts that represent real world usage of the model in that target domain.
The first step is to synthesize an initial environment that is based on that data. That involves, for example, a policy that specifies the kind of task or the kind of domain where you want the model to operate. So for example, if this is meant to simulate customer support in banking, then you could generate a policy that describes how the bank would operate, what the hours are, where the users are, different accounts, maybe synthesize a small database. And also it creates tools that the model can use to then interact with this environment. Again, crucially, this can be conditioned on real usage of the model, which ensures that this environment is representative of how the model is going to be used later in production. If this data is not available, we’ve also created what we call the business knowledge graph, which is data that has been crawled all across the internet and that describes various representative businesses. So we can also synthesize domains for which we don’t have real world data, but we have crawled the internet to gain real world knowledge about these businesses.
Now, given an environment, the most important next thing that we want to generate are tasks. So we’re going to have simulated users that will try to achieve certain goals in this environment, and the agent will assist them in these goals. And we need to synthesize what those goals are. In order to do this, we start with what is called a database of seeds. These are abstract scenarios — for example, I want to open an account, close an account, I want to change my phone number, I would like to get some information — as well as personas, which are certain types of humans that might be interacting with the model. Given that we have an environment, we can take our library of scenarios and other seeds and specialize it to this domain, again by invoking an agent. And then once we have a good database, we can start to generate tasks. These are real world interactions that a user would have with these models, and then we will simulate those users using other agents.
Now, getting a good set of tasks is also non-trivial. We need to strike a careful balance of tasks that are not too easy to solve. If a model consistently solves a task, it’s not useful for learning. And conversely, if the task is too hard, there’s also not enough learning signal. So there is an iterative procedure where we filter and reconstruct tasks based on the ability of the model to solve them. And to do this, we use what we call hardening traps, which means that we can modify the task to make it more difficult. So for example, you can take away certain information, or you can make the ask more complex. You can do all kinds of tricks to harden the task, or you can also filter out difficult tasks, until we get something that we feel strikes a good balance between being realistic and providing learning signal to the model.
So again, the output here are RL tasks. And then of course, this is continuously inspected by looking at real world data. These environments can still be verified. The harness can be updated. And also, for models that are coming out of this environment, we get real world feedback from users, and we can use that to further refine the harness and further refine this synthetic generation procedure.
So as an example of what this model can achieve — this is an experiment where we used our latest preview model, Mercury 2.5. Before it was trained on any sort of domain specific data, it achieved about 50% accuracy, and training it on an initial set of tasks synthesized from TauForge improved performance on another set of environments by over 23%, matching the performance of all kinds of state-of-the-art open source and closed source models. And here, just to clarify, the setup is that there’s a diverse set of environments and businesses. So for example, banking or booking airplane tickets and getting a doctor’s appointment. So you can have a whole range of businesses, and you can split that into a test set and a training set. You hill climb on one set of businesses and you test on a different set of businesses, which helps reduce overfitting. And again, these tasks — especially the test set — are closely derived from real world logs, which gives us confidence that this will correlate with real world performance once we deploy into production.
So this is just a bit of information on the kinds of techniques that we can use to make our models really high quality, not just fast, but also improve their intelligence in some very specific priority domains like voice.
And if you’d like to try these models, they’re available. They run at a thousand tokens per second. This is the pricing. And even though I mainly talked about voice in this talk, we also have customers across other domains, including search as well as code. And so these are all domains where Mercury 2 models are really good for latency sensitive tasks. And if you’re interested in feedback, we have YC companies and other companies that are using these, so happy to tell you more after the talk. And maybe I’ll just briefly add by saying that we have a program for YC startups. If you are in YC, we have $500,000 in credits that we’re very happy to give away. If you’re interested, please grab me and I would love to tell you more. Thank you.
Francois: All right, next up we have Shane.
Shane: Awesome. Thank you. Thank you so much for your attention. I think I’m the last talk. I’m Shane. I just graduated from my PhD at MIT. And I think this is my favorite project during my five-year PhD. It was work that I did while interning at Google — thank you for the TPUs to make this happen.
We basically looked at multilingual pre-training, and within that, transfer between the languages. So the synergies, interference, interactions between the different training sets, data set sources, that are different languages. While I walk you through this project for the next 10 minutes, you can imagine all of these experiments and all of these results and methods could be generalized to many different sources within your data. They don’t have to be languages. They could be different domains or different quality sources of data, different things, however you want to splice it up. And I think that’s important because increasingly, when we think about our model’s capabilities, its risks, its limitations, we think about how all of those different pieces interacted and came together.
So diving into this, I’m going to talk a little bit about why this is important, motivating it a little bit beyond English. Then I’m going to go through, really quickly, rapid fire, some of the research questions we asked and the answers we arrived at, before bringing it together. This was presented at ICLR earlier this year with these phenomenal folks at Google, UW, Stanford, and some other places.
If you look at scaling law and even mixing law papers in the literature right now, they are 99% about English training data and English evaluation. There’s very little about the rest of the world and the community, which is actually kind of shocking. And so this is thinking, what if the objective was a single language that wasn’t English, or multiple languages, a cluster of them, and exploring that domain? And why is that different than in English? The reason is because there are severe constraints when you go past the first few languages that are very abundant online. And we also provide tooling and methods both for scaling laws, mixing laws, fine-tuning versus pre-training, and other decisions that developers have to make. And this has implications for developing models for your own culture, society, language. And also trying to figure out, if you are Turkish, what are the constraints in pre-training for Turkish language models? How much will they lag behind language models for English, based off of just the availability of data that’s there, and what synergies and interference we can expect in pre-training? So let me start with the problem setup. Very simply, let’s pick a language. We’re going to pick Thai, because there’s something like 80 million speakers in the world. It’s a very unique language, but we could be talking about Swahili, Polish — it doesn’t matter. The point is that in MADLAD-400, a popular Common Crawl-based pre-training corpus, there are 0.6% the number of tokens in Thai as compared to English. So less than a percent of the amount of English data that you have.
Some frontier model, ChatGPT, whatever it is — we don’t know, but it has some training mixture among the natural languages. And it’s probably going to roughly follow, like every other language model, the distribution. They’ll do different sampling techniques and things like that, but you’re going to roughly have more English than anything else, followed by the next languages you care about that are more abundant, more heterogeneous, all the way down that list. And by the time you get to Thai, it’s a tiny sliver of the overall mixture, much less than 0.6%. And so the performance on Thai is going to be pretty poor. It might be pretty good overall because it’s a massive model, but it’s not going to be great for Thai speakers. And this is shown in the literature and in evaluations.
So you might say, okay, let’s train a model just for Thai. We’re going to monolingually only train on all of our Thai data. But the problem is there’s so little of it that we end up repeating it again and again and again. Those are those dashed lines, epochs. And you’ve overfit to your Thai data. You need a much smaller model. And so actually you don’t do very well there either.
Ideally, what you want to do, if we skip ahead to the solution, is something like this. It’s actually a mix where you use a lot of your Thai data, but by the time you’re hitting severe diminishing returns, you’re also incorporating — and this isn’t a curriculum, this is all mixed together — high quality synergistic data that happens to be English, Indonesian, Malay, Lao, Croatian, it turns out. But you’re not going to know that, because this isn’t just about what language families are similar. It’s about what data empirically is not noisy in that other source, is actually helpful, covers similar topics. And that is very distributed and uncertain on the web. It doesn’t just mean that they’re from the same language family. So it’s something you have to empirically measure. You can’t just bring a linguist and have them theorize about linguistics for this particular case.
So how do we get to the fantastic Thai model, or Swahili model, or Turkish model, and figure out the right mix and the right model size? First, we want to measure language synergies, and this is where a lot of the contributions of the paper come in. There are many ways you can think about doing this. And the way that we landed on was, imagine this: you have a training curve. So over your training steps, the red line is where you’re training only on Thai data. So it’s monolingual. It does pretty well, but it flattens very early. Now imagine that compared to models that are trained fifty-fifty on Thai and English, Thai and Telugu, Thai and other languages. And you can see that those learning curves are much higher, meaning that they’re not as good. And so actually the distance between these curves tells you something about how good that language is at being helpful for Thai. And so we can say, okay, well, Indonesian’s better than English, it’s better than Telugu, because that distance is longer.
And this efficiency gap can be formalized into something that looks like this. And let me abstract all this away — you don’t have to go into too much detail. If these two lines are really close, the number is big and positive. If the lines are very far away, it’s going to be a negative transfer, meaning there’s interference. These languages are competing for tokens or neurons in order to represent one another and perform well on them.
And if you do this over and over again, with many tricks that I’m not going to get into, you get this big cross-lingual transfer matrix, where red means high synergy and blue means high interference. And so if you look at — you can’t see it, but imagine you could zoom in and look at Spanish, the row — the bright red spots empirically happen to be Portuguese, Italian, and French. And the most negative, harmful language is Japanese, meaning if you train with Japanese, it’s going to hurt your Spanish. And so this is all empirically borne out, and there are some surprises in here, some interesting things. Some languages are more helpful broadly than others, and it’s complicated.
The other really cool thing about this matrix is it’s at one fixed size of language model. If we increase the size of the language model and make it much bigger, then actually a lot of these blues start to turn red or deeper red. There’s more synergy, because the model’s bigger and can accommodate understanding multiple languages at once, or together, and seeing their common patterns. But if the model gets really small, all of these turn to blue and they all interfere, competing for space within the model.
All right, quick question for the audience. Who thinks that if Portuguese is helpful for Spanish, that means that Spanish is helpful for Portuguese — that that symmetry exists, from what we measured? Raise your hand if you think that is true. We have a few people. If you think it’s not true, raise your hand. Okay, it’s kind of fifty-fifty. Well, actually it’s kind of hidden on the screen. I know it’s late. It’s not symmetric, as you can see. And actually there’s this weird sort of branching factor that happened. And this is important, because if you’re a practitioner and you do experiments, you say, okay, Telugu is really helpful for Swahili — you can’t assume the opposite. You have to go measure it. And so that’s where this breaks down, and we actually show that.
Okay, second question. You can redeem yourselves. What is more helpful, do you think — if two languages share the same script, or if they share the same language family? Who wants to vote for family? We have some. Who wants to vote for script? All right, congratulations team script. They both matter, but script matters a little bit more, probably because of tokenization artifacts and how those are represented.
If you’re taking pictures, don’t worry, it’s all online. I will point you to it. I’m going to go through, in the last three minutes, really quickly, a little bit of math. We have this great matrix. We have all these numbers and relationships between languages. But the last step is, how do we actually create a scaling law from this? How do we actually understand how to predict, to estimate loss, or figure out the best mixture of languages for a given target, which is our original goal? You have to fit scaling laws. Those of you that are familiar with Chinchilla probably remember this. You predict the loss based off of the model size and the amount of data you’ve trained on, and you fit it using these blue parameters here.
However, if you have a multilingual data source and you’re trying to figure out how good it was on French, the thing that matters most is how much French data you had. And if you have tons of other data, how’s that factored in? 1D is not helpful. You need to model all of it. And so that’s what we do. There are many ways to model this. We explored many options, but a simple solution that actually works really well is to break down Chinchilla’s D into a composite of the monolingual source, in this case Thai, the close transfer languages in the matrix, plus a bucket for everything else. And we account for diminishing returns when you do multiple epochs of certain data. There are a lot of details. Long story short, it looks something like this. And the tau is sort of a weighting factor that you can learn from the matrix. I’m going to rush along in the interest of time, but you get bold numbers in a table, state of the art. Yay. It works. It works really well, actually. Maybe because a lot of people can’t run 700 pre-training experiments, maybe because this is underserved and people aren’t looking at this enough. But either way, I think this is really useful machinery to use for mixing multiple sources and understanding how they interact to predict the final loss for your model and how to scale your model given what you have.
So it gets us back to this. I’m going to wrap up by saying we did some experiments to say — you have big pre-trained models available that are multilingual. Do you just fine-tune, or maybe should you pre-train from scratch for the language that you care about? We tell you which to do based on how much compute you have. We also look at the curse of multilinguality. So if you want to increase the size of a language model from four languages, and you want to retrain it for eight languages, how much more data do you need, and how much bigger does the model need to be in order to maintain the same performance as before? We do a bunch of math. It’s very cool. And we show you exactly what to do as a practitioner. And it actually holds really well over different target languages. So I’d recommend consulting this if you’re interested. Thank you so much. Appreciate your attention.