{"slug": "transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design", "title": "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-Design", "summary": "Researchers propose Transformer Transformer, a unified diffusion model that generates complete robot embodiments—every link, joint, motor, and inertial property—optimized for a given manipulation demonstration. In a cloth-flinging task on an ALOHA 2 bimanual platform, the generated robot reduced tracking error by 73% and max joint speed by 30% versus the original. The model uses RoboTokens, a unified tokenization of robot embodiments, states, and actions, and a technique called Dynamics Self-Guidance to optimize for unseen rewards at inference time.", "body_md": "Not all robots are created equal — but what if you could design one for a specific task?\nWe propose **Transformer Transformer**, a unified model that does exactly this: hand it\na manipulation demonstration, and it generates a complete robot — every link, joint, motor, and\ninertial property — optimized for that motion. We fabricated one such design for cloth\nflinging on an ALOHA 2 bimanual platform; it reduced\ntracking error by 73% and max joint speed by 30% versus the original.\n\nBehind that result is a diffusion transformer trained on **RoboTokens**, a unified\ntokenization of robot embodiments, states, and actions. The same architecture spans embodiment spaces\n(wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross-embodiment\ncontrol). Rather than overfitting to one reward function, it is a dynamics model whose reward-agnostic\npredictions are converted into reward-specific value predictions at inference time, then used to steer\nembodiment diffusion through a procedure we call **Dynamics Self-Guidance**. Experiments\nacross three design spaces show zero-shot optimization of unseen rewards and trajectories, improving\nperformance and runtime over an evolutionary baseline.\n\nA 17-minute walkthrough — or scroll for the figure-and-caption version.\n\nA robot's embodiment decides what tasks it can do well. We pour effort into making robots smarter — bigger datasets, better algorithms, more capable policy architectures — but a bad embodiment can hamper even a great policy. You can collect all the tossing data in the world, but if your robot's shape is far from optimal, it might toss itself instead of the ball.\n\nSo let's frame the problem concretely. Given as input a target end-effector motion and a reward function,\nwe want to generate a complete robot embodiment for the task — by *complete* we mean every link,\njoint, motor, and inertial property, plus a controller that drives the whole thing. We call this\n**motion-conditioned robot co-design**.\n\n*A randomly procedurally-generated robot attempting the target motion: it falls before it can\nfinish 🙃*\n\nOur framework reframes robot co-design as a three-step process: you **demonstrate** the\ndesired end-effector motion (e.g., from a UMI 3 human\ndemonstration), our model\n\nPART 1\n\nIf we want one model to co-design *any* robot, the network needs a shared vocabulary that spans\nevery embodiment. RoboTokens are that vocabulary: typed tokens for every link, joint, motor, state, and\naction — organized so a single sequence describes both a robot's *embodiment* (time-invariant) and\nits *dynamics* (time-varying).\n\nRoboTokens has the following key design goals:\n\nWe tokenized 11 robots from the MuJoCo Menagerie 10, spanning\ntwo orders of magnitude in mass (0.65 kg dexterous hand to 67.5 kg quadruped) and from 6 to 35\nactive joints. Each became a sequence of 28–101 RoboTokens — small enough to diffuse end-to-end.\n\nPART 2\n\nThe model is a diffusion transformer (DiT 1) trained on\nRoboTokens with a DDIM\n\n(And about the pun: the first “Transformer” refers to *robots* whose embodiments can\nchange. The second refers to the self-attention *architecture* 🥸)\n\nBecause the controller sees the full embodiment as input, it can adjust its control strategy to the robot\nit's driving. Change link inertias, joint ranges, or motor gains, and the same policy keeps tracking —\nwithout any retraining per robot. We behavior-cloned whole-body controllers from RL experts 7,\n\nRun the model with no conditioning at all and it diffuses out complete robots from Gaussian noise. Trained\non diverse robots from Menagerie 10, a single Transformer\nTransformer produces fixed-base arms, quadrupeds, humanoids, and dexterous hands. Because diffusion is\nstochastic, every run gives different robots.\n\nPART 3\n\nNow the actual co-design problem: how do we generate a high-reward robot for a reward function the model has never seen during training?\n\nHere is the trick. The same model predicts both the embodiment *and* its dynamics, so we can feed\nthose predictions into the user's reward function and get a *predicted reward*. With predicted\nrewards in hand, we can run the model many times in parallel, predict the reward for each candidate, and\nreturn the best one. That is best-of-N sampling, but at GPU speed.\n\nWe can do better. Because the predicted reward is a differentiable function of the embodiment tokens, we\ncan take its gradient and inject it back into the diffusion process — pushing samples toward higher reward\nat every denoising step, following classifier-guided DDIM 11.\nThe model is, in effect, asking itself: “how should this robot change\nto increase the reward?” We call this\n\nTake a randomly-generated quadruped that falls during a dynamic toss. Ask our model to generate a robot for the same trajectory with only a tracking reward and it lowers the center of mass and widens the support polygon — stable design choices it discovered as a side-effect of trying to maximize tracking reward it has never seen.\n\nNow add a torque-penalty term and the optimization landscape shifts. The model returns a smaller, lighter\nrobot — still tracking the motion, but using 30% less average torque. Add a size penalty and the\nlandscape shifts again. This is what people might call *zero-shotting rewards*: train on dynamics\nmodeling, optimize at inference for any reward you can write down.\n\nManipulation isn't one motion; it's a *distribution* of motions. We want to optimize a robot for\nan entire task — say, dish washing — not a single trajectory. Diffusion composition 14,\n\nWe tested this on the publicly-available UMI bimanual dish-washing dataset 3, holding out 26 validation trajectories. Transformer Transformer generated\nrobots optimized over the entire held-out distribution — navigating both the discrete space (spine\ndesigns:\nfixed, sliding, bending) and the continuous space (arm dimensions, mounting offsets) — and the same model\nthen served as the cross-embodiment controller to validate each generated design.\n\nWe compare against two baselines: **Random** (sample a robot from the procedural design\nspace) and **CMA-ES** 16, a black-box optimizer\nwidely used in co-design. CMA-ES rolls out candidate robots in MuJoCo\n\nWe give all methods the same task: generate a robot for a target trajectory and reward, control it, report the achieved reward.\n\nLike large language models that get better with longer thinking 19, Transformer Transformer gets better designs when given more inference\nseeds to sample over. Performance does eventually plateau after about a minute — unlike the seemingly\nendless gains LLMs get from longer reasoning. We discuss this in the limitations.\n\nTo stress-test the whole system on hardware, we picked dynamic cloth unfolding on ALOHA 2. Flinging\n\nWe asked our model to optimize the ALOHA design for the “Tracking Velocity” reward, then fabricated the resulting design. Two things changed compared to the original:\n\n**Geometry and scene.** RoboTokens currently cover primitive-based geometry, not arbitrary\nmeshes or deformables, and don't yet encode the surrounding scene or contact targets. Extending to\ncomplex geometry and scene context is a clear next step toward fully task-conditioned co-design.\n\n**Controller transfer.** Our cross-embodiment controller correlates with the RL oracle at\nPearson *r* = 0.53 on diffused embodiments — promising but with headroom. Some outlier\nembodiments expose where the controller fails to generalize.\n\n**Test-time scaling plateau.** Like other guided-diffusion methods, our test-time compute\ngains plateau after about a minute. Unlike LLMs 19, more\ninference time stops helping past a point. Improving reward-prediction accuracy and alternative\ninference schemes are open directions.\n\n**Data generation cost.** RL-based whole-body controller data generation is expensive (one\nRL expert per discrete design choice, 16 hours of A100 per policy). Scaling Transformer Transformer to\nnew design spaces hinges on reducing this cost.\n\nFull reference list available in the paper. The selection above covers the citations used on this page.", "url": "https://wpnews.pro/news/transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design", "canonical_source": "https://transformer-transformer.github.io/", "published_at": "2026-07-29 03:52:29+00:00", "updated_at": "2026-07-29 04:23:40.654695+00:00", "lang": "en", "topics": ["robotics", "machine-learning", "generative-ai", "ai-research"], "entities": ["Transformer Transformer", "RoboTokens", "ALOHA 2", "MuJoCo Menagerie", "DDIM", "DiT"], "alternates": {"html": "https://wpnews.pro/news/transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design", "markdown": "https://wpnews.pro/news/transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design.md", "text": "https://wpnews.pro/news/transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design.txt", "jsonld": "https://wpnews.pro/news/transformer-transformer-a-unified-model-for-motion-conditioned-robot-co-design.jsonld"}}