What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

Handshake's AI training business reached $1 billion in gross annualized revenue roughly a year after launch, Garrett Lord said on a Latent Space deep dive covering what frontier labs buy from expert-data providers. The episode walked through Handshake's reinforcement learning environments and benchmarks, including ATLAS-Finance, a simulated bank task where an agent misapplied a goodwill figure from a client email, and VIALS, a scientific image benchmark where a model picked slides H and I plus G when asked which two showed the most advanced vessel necrosis. Handshake researchers also demonstrated a simulated file-access coding task in which evaluators inspect which files an agent read and what it did to the shell history afterward.

This is our first Latent Space deep dive with an AI data company . In roughly a year, Handshake’s AI training business went from launch to $1 billion in gross annualized revenue https://www.theinformation.com/articles/handshake-mercor-revenue-surges-demand-human-contractors-train-ai?utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv . What are frontier labs buying at that scale? What evals are models still nowhere close to human level on? Garrett Lord brought the people who build these reinforcement learning environments to show us. For the first time, we are walking through them live To cover the scope of Handshake, we had to produce a very special joint podcast across our Science/Engineering/Research/Finance podcast hosts, because Handshake’s business runs across all these and more. Watch for three segments after the general intro: - Finance. In the finance demo, the agent has to find its assignment in a bank’s email, work out which files govern the deal, and deliver a workbook. Watch what happens to one line in the client’s email about existing goodwill. The agent carries the number into Python and subtracts it. Collin points out that the model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies. - Bio. Nick Magazine and Jonas Mueller show us VIALS, Handshake’s benchmark for interpreting the images scientists use at work. Nick puts nine slides of brain-tumor tissue on screen and asks which two show the most advanced vessel necrosis. Give yourself a moment before his answer. Nick picks H and I. The model picks those two and adds G. Watching Nick point to what’s missing from the tissue is more instructive than seeing the score. - Coding. In a simulated file-access task, the team can inspect which files an agent read, whether it checked the copies, and what it did to the shell history afterward. A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating. Who’s in the room - 🟣 AI data business — Garrett Lord, Sahil Bhaiwala and Nishant Ranka open and close the episode on the market for expert data. - 🌸 Life sciences — Nick Magazine and Jonas Mueller put the gel and brain-tissue tasks on screen. - 🟢 Finance — Collin Schweiker and Sahil Bhaiwala take us through Project Falcon and its Excel grader. - 🟠 Alignment — Steve Kelling and Nishant Ranka follow the agent’s tool calls through the safety simulations. Show notes - ATLAS-Finance https://joinhandshake.com/research/benchmarks/atlas-finance/?utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv — Explore the benchmark for agents working across a simulated bank’s inboxes, documents, and financial models. - Inside a bank https://joinhandshake.com/research/benchmarks/articles/atlas-finance-evaluating-ai-agents-inside-a-bank/?utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv — Handshake traces a separate published deal, Project Osprey, from the client’s email to a grading decision. The deal on screen in our interview is Project Falcon. - VIALS sample tasks https://joinhandshake.com/research/benchmarks/vials/?utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv — Try the scientific image questions, including the gel and brain-tissue tasks from the episode. - How Handshake trained a model to read scientific figures https://joinhandshake.com/research/ai/vials-nemotron-post-training/?utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv — The team describes using intermediate image descriptions to train and evaluate a vision-language model. Timestamps 0:00 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=0s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Inside Handshake AI 0:39 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=39s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Garrett, Sahil and Nishant on frontier training data 5:15 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=315s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How to build a banking environment for an agent 13:07 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=787s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How frontier labs buy data 15:51 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=951s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How a lab evaluates a new dataset 20:01 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=1201s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Testing training data on open models 27:38 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=1658s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Where rubrics fall short of expert judgment 31:30 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=1890s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Nick and Jonas introduce the VIALS benchmark 35:52 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=2152s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Reading molecular structures and plasmid maps 43:48 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=2628s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Training vision models to interpret scientific images 47:17 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=2837s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Task one: reading the bands on a gel 50:59 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=3059s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Task two: finding necrosis in brain tissue 55:49 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=3349s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Can Python tools help the model see? 1:02:52 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=3772s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Can biology models improve on their own? 1:06:19 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=3979s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv What a biology lab can buy from Handshake 1:12:33 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=4353s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Atlas Finance with Collin and Sahil 1:18:56 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=4736s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Inside a simulated investment bank 1:25:45 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=5145s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Where the agent’s financial model went wrong 1:31:18 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=5478s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How a banker grades the spreadsheet 1:41:12 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=6072s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Penalty traps, dependency gates and partial credit 1:46:41 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=6401s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How an agent gamed the Excel grader 1:48:52 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=6532s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv What would make an AI banker useful? 1:54:48 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=6888s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Safety and alignment with Steve and Nishant 1:58:25 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=7105s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv CAREBench and the limits of a refusal 2:04:42 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=7482s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv How safety feedback reaches model training 2:07:21 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=7641s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv When an agent sends the wrong files 2:11:14 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=7874s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv A loan application tests fair lending 2:16:09 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=8169s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Designing private benchmarks for enterprises 2:25:37 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=8737s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv The leadership team on Handshake’s mission 2:28:36 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=8916s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv When enterprises need their own models 2:33:03 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=9183s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv The market for compute and expert data 2:39:12 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=9552s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Why labs hold out evaluation tasks 2:43:01 https://www.youtube.com/watch?v=v5IJwVnTxzg&t=9781s&utm source=latent.space&utm medium=ainews&utm campaign=lsbeehiiv Hiring at Handshake AI