{"slug": "lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps", "title": "LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS", "summary": "LynnReal-AI published arXiv:2611.21479 on November 17, 2026 and released open inference code for LynnReal-Hand, a hand-controlled video world model that maps egocentric MANO-scale hand poses and camera trajectories into object interactions rather than camera-only fly-throughs. The 33B-parameter model ships in Base (bidirectional, 1344x768, 30 schedule points, CFG 3) and Fast (causal streaming, 960x540, three flow-map steps) variants; the Fast path streams at 18.03 FPS at 540p on one H200 with 741 ms mean warmed response per chunk and 34.89 GiB peak allocated GPU memory. On HandBench the authors report FVD 181.67 and MANO Dice 93.02 for the full-sequence model and MANO Dice 92.49 for the three-step causal model, with the Ultra-Fast-VAE decoder cutting decode latency from 831.30 ms (Native H3) to 46.77 ms at 1344x768.", "body_md": "**LynnReal-Hand is a hand-controlled video world model that maps egocentric hand motion and camera trajectories into coherent object interactions**, not just camera fly-throughs. The team published [arXiv:2611.21479](https://arxiv.org/abs/2611.21479?ref=explainx) on November 17, 2026 and shipped [open inference code](https://github.com/LynnReal-AI/LynnReal?ref=explainx) with Base and Fast variants.\n\nIf you build embodied agents or simulators, the control channel matters as much as pixels. Text can say \"pick up the cup.\" A wearable or teleop rig gives you **continuous MANO-scale hand poses** every frame. LynnReal-Hand is built for that second interface.\n\n## What problem does LynnReal-Hand solve?\n\nInteractive video world models already accept **text prompts** and **camera paths**. That is enough for exploration, but manipulation needs **contact-rich motion**: fingers closing on a handle, bimanual coordination, releases.\n\nThe [abstract](https://arxiv.org/abs/2611.21479?ref=explainx) states the core difficulty: hand-object dynamics are hard to model, and real-time computation is expensive. Prior work such as hand-conditional generators improved clips, yet full **streaming** at usable frame rates on one GPU remained scarce.\n\nLynnReal-Hand unifies **interaction modeling** with **efficient streaming generation** through four design pieces named in the paper:\n\nRobot hand reaching a small globe, illustrating hand-conditioned embodied video world models\n\n| Component | Role | \n|---|---|\n| 100K interaction clips | Real and synthetic environments with hand-aware supervision | \n| Causal flow map | Few-step generation with a three-step Fast variant | \n| Region-aware score distillation | Combines video and image priors on contact regions | \n| Ultra-Fast-VAE | Lightweight decoder to cut latency | \n\n## Base versus Fast: two operating points\n\nWireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motion\n\nThe [GitHub README](https://github.com/LynnReal-AI/LynnReal?ref=explainx) separates **LynnReal-Hand-Base** and **LynnReal-Hand-Fast**:\n\n| Model | Backbone | Mode | Default resolution | Sampling | \n|---|---|---|---|---|\n| LynnReal-Hand-Base | 33B | Bidirectional full sequence | 1344 x 768 | 30 schedule points, CFG 3 | \n| LynnReal-Hand-Fast | 33B | Causal streaming | 960 x 540 (768p config exists) | Three flow-map steps, W2 context | \n\n**Base** targets quality-first offline renders. **Fast** targets **online** use: fixed first-frame anchor, bounded sliding context, and local temporal-position resets so attention memory does not grow without bound.\n\nThe Fast path emits **17 RGB frames per chunk** at 540p. The CLI documents separate conditioning caches for Base and Fast; you cannot reuse a Base cache for Fast inference.\n\n## Reported benchmarks: HandBench and latency\n\nOn [HandBench](https://arxiv.org/abs/2611.21479?ref=explainx), the authors report:\n\n- **FVD 181.67** and**MANO Dice 93.02** for the full-sequence model\n- **MANO Dice 92.49** for the three-step causal model\n\nFor streaming at **540p on one H200**, the README lists **18.03 FPS**, **741 ms** mean warmed response per chunk, and **34.89 GiB** peak allocated GPU memory with sampling-specific AdaLN tables and heavy encoders offloaded.\n\nThe repository also publishes a **comparison table** against JointControlVideo, Generated Reality, and Hand2World variants on latency and memory. Treat those rows as **paper measurements** until you run `lynnreal-hand` on your own hardware.\n\n### Ultra-Fast-VAE decoding numbers\n\nDecoder-only benchmarks at 1344 x 768 (17 frames, one H200) from the README:\n\n| Decoder | Params (M) | Latency (ms) | FPS | \n|---|---|---|---|\n| Native H3 | 2423.5 | 831.30 | 20.44 | \n| Ultra-Fast-VAE | 361.2 | 46.77 | 363.21 | \n\nThat gap explains why the team treats VAE replacement as a first-class contribution, not an afterthought.\n\nGreen leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generation\n\n## How to try LynnReal-Hand locally\n\nThe maintainers document a **Linux + CUDA 12.8** stack with FlashAttention-3. High-level steps from the README:\n\n1. Clone `https://github.com/LynnReal-AI/LynnReal` and install the package in a venv.\n2. Run `bash scripts/install_fa3.sh` and`lynnreal-hand doctor` .\n3. Download weights: `python scripts/download_weights.py --component fast` (or`base` ).\n4. Validate packaged example 05: `lynnreal-hand validate --config configs/fast.json --example examples/05/example.json` .\n5. Prepare cache, then infer with `lynnreal-hand prepare` and`lynnreal-hand infer` .\n\nSeven examples ship with aligned **images, hand videos, camera npz files, captions, and reference MP4s**. Example 05 is the featured **pet interaction** scene in the README.\n\nFor custom data, you supply `example.json` with `hand_video`, `camera.npz`, `caption`, and frame counts in **124, 158, 243, or 260** frames. Paths must stay consistent with the same field of view.\n\nAdd `--accelerated` on Fast inference after the first compile pass to reuse CUDA graphs in-process.\n\n### What you should expect on smaller GPUs\n\nThe project openly targets **H200**. README memory for Base full sequence is **82.59 GiB** at 768p in their table. Plan cloud H200 time for serious replication; consumer cards are not the reference platform.\n\n## How LynnReal-Hand fits next to other world models\n\nexplainx.ai covered **[MBZUAI PAN 2](https://explainx.ai/blog/mbzuai-pan-2-open-world-model-simulative-reasoning-2026)** as a **29B open world model** for **simulative agent reasoning** up to 20 seconds from text actions. PAN 2 is about **plan comparison before acting** in latent video space.\n\nLynnReal-Hand is complementary:\n\n- **Control signal:** continuous**hand articulation** plus camera pose, not only language actions.\n- **Use case:** teleoperation previews, egocentric AR, and manipulation-centric simulators.\n- **Architecture emphasis:****causal streaming** and decoder efficiency for**interactive FPS** .\n\nIf your agent stack already uses **[ForceHarness](https://explainx.ai/blog/forceharness-frozen-frontier-models-force-robot-harness-2026)** or **[ScaleVLA](https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026)** for robot policies, LynnReal-Hand is closer to the **video imagination layer** that sits above low-level torque commands.\n\nFor policy learned in latent video without hands, see **[Gemini Robotics 2 whole-body intelligence](https://explainx.ai/blog/gemini-robotics-2-whole-body-intelligence-july-2026)**. Hand-controlled worlds are the interface when the human body remains in the loop.\n\n## Training story in plain language\n\nThe paper describes **Interaction-Preserving Training (IPT)** on **100K clips** with **hand-aware multilevel supervision** so small contact regions keep gradient signal. Causal distillation preserves control fidelity when compressing to three steps.\n\n**Region-aware score distillation** merges complementary **video and image priors** so fingers and object surfaces stay sharp while global motion stays stable. That matters because hand errors are visually loud: a floating grip breaks immersion faster than a blurry background.\n\nThe team also cites related **[BiWM](https://arxiv.org/abs/2606.10135?ref=explainx)** bidirectional autoregressive world modeling and **[LynnReal-Omni](https://arxiv.org/abs/2609.15863?ref=explainx)** for multimodal agent workflows in the README citation block.\n\n## Risks, limits, and open questions\n\n**License:** MiniMax H3 Community terms govern the repo. Read NOTICE before product use.\n\n**Data release timing:** The paper commits to datasets and training code; verify Hugging Face and GitHub release tags when you plan reproducibility studies.\n\n**Evaluation generalization:** HandBench scores are author-reported. Cross-benchmark comparison to closed labs is not provided in the public README.\n\n**Input rigidity:** Frame counts are discrete allowed lengths; arbitrary-length phone videos need resampling and camera alignment work described in the data preparation docs.\n\n## LynnReal-Hand CLI workflow in practice\n\nThe maintainers ship a **`lynnreal-hand`** CLI instead of a single opaque script. That matters when you integrate with robot teleop loops where prepare and infer run on different machines.\n\n| Command | GPU required | Purpose | \n|---|---|---|\n| `lynnreal-hand doctor` | Optional | Check FFmpeg, FlashAttention-3, CUDA | \n| `lynnreal-hand validate` | No | Verify example.json paths and frame counts | \n| `lynnreal-hand prepare` | Yes | Build conditioning cache from image + hand video + camera | \n| `lynnreal-hand infer` | Yes | Stream MP4 from prepared cache | \n| `lynnreal-hand infer --accelerated` | Yes | Reuse compiled CUDA graphs in-process | \n\n**Validation without weights** is deliberate: teams can fix camera alignment and caption mismatches before downloading **multi-gigabyte Hugging Face packages**.\n\n### Example 05 pet interaction path\n\nThe README spotlights **example 05** because it shows **bimanual affordances** near a pet, a scene where hand motion dominates semantics.\n\n1. Download Fast weights with `python scripts/download_weights.py --component fast` .\n2. Run validate on `examples/05/example.json` with`configs/fast.json` .\n3. Prepare cache to `outputs/cache/fast/example_05` .\n4. Infer to `outputs/fast/example_05.mp4` with seed 42.\n5. Compare against packaged **Fast reference MP4** in the same folder.\n\nIf MANO alignment is wrong, output fingers may float. The README stresses that **image, hand motion, and camera poses must share field of view and frame order**. That is the most common integration bug in hand-conditioned video.\n\n### Frame budget constraints\n\nPackaged inputs accept **124, 158, 243, or 260 frames** at the source frame rate. Arbitrary phone footage needs resampling documented in **data preparation** guides on GitHub.\n\nFor **768p Fast**, switch to `configs/fast_768.json` and rebuild caches; you cannot hot-swap resolution on an existing 540p cache.\n\n## Paper metrics versus README metrics\n\nThe [arXiv PDF](https://arxiv.org/abs/2611.21479?ref=explainx) and README occasionally use different resolutions in tables:\n\n- Paper HandBench highlights **MANO Dice** and**FVD** at reported training resolution.\n- README latency table mixes **768p full-sequence Base** with**540p Fast chunks** .\n\nWhen you cite numbers externally, specify **which row** you mean. Reviewers rejected vague \"18 FPS world model\" claims when the measurement is **warmed Fast chunk throughput**, not first-frame cold start.\n\n### Comparison to Hand2World and JointControlVideo\n\nThe README reproduction table positions LynnReal-Hand against **Hand2World-Bi**, **Hand2World-AR**, **Generated Reality**, and **JointControlVideo**. LynnReal-Hand-Base is slower than Hand2World-Bi on median full-sequence latency in that table, but reports higher **MANO Dice** on HandBench.\n\n| Direction | Implication | \n|---|---|\n| Accuracy-first offline | Consider Base at 1344 x 768 | \n| Teleop streaming | Fast at 540p with Ultra-Fast-VAE | \n| Smaller backbones | Hand2World 5B may fit tighter VRAM | \n\nNone of those tradeoffs remove the **H200-class** assumption for official Fast benchmarks.\n\n## Integration ideas for agent builders\n\n**Simulators:** Pipe MANO or MediaPipe hand tracks into `hand_video`, feed SLAM poses into `camera.npz`, and let LynnReal-Hand predict **visual consequences** before executing torques on hardware.\n\n**Data flywheel:** Use generated failures to label grasp policies; pair with **[ScaleVLA](https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026)** only after you verify labels on real video.\n\n**Safety review:** Hand-conditioned worlds can depict dangerous contact. Treat outputs like any synthetic media pipeline with policy filters before user-facing playback.\n\n## Troubleshooting checklist\n\n| Symptom | Likely cause | Fix | \n|---|---|---|\n| OOM on infer | Base at 768p on smaller GPU | Switch to Fast 540p or rent H200 | \n| Drifting fingers | Misaligned camera npz | Re-export poses with shared intrinsics | \n| Black frames after prepare | Missing hand_video path | Run validate with `--check-weights` | \n| Slow first infer | CUDA graph compile | Keep process alive; use `--accelerated` | \n\n## Who should care?\n\n| Reader | Takeaway | \n|---|---|\n| Robotics engineer | Test whether hand-video conditioning reduces sim-to-real gap for grasp planning demos | \n| AR/VR builder | Streaming 540p interaction may pair with passthrough headsets when latency holds on your GPU | \n| Agent researcher | Combine with text planners: language picks the task, hands supply the low-level trajectory | \n| Educator | Open examples make a concrete lab for world models beyond static image diffusion | \n\n## Related reading\n\n*Primary sources: [arXiv:2611.21479](https://arxiv.org/abs/2611.21479?ref=explainx) and [LynnReal-AI/LynnReal](https://github.com/LynnReal-AI/LynnReal?ref=explainx) as of November 18, 2026.*", "url": "https://wpnews.pro/news/lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps", "canonical_source": "https://explainx.ai/blog/lynnreal-hand-hand-controlled-video-world-model-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 10:15:16.824260+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "ai-research", "ai-agents", "ai-infrastructure"], "entities": ["LynnReal-AI", "LynnReal-Hand", "LynnReal-Hand-Base", "LynnReal-Hand-Fast", "Ultra-Fast-VAE", "HandBench", "arXiv:2611.21479", "NVIDIA H200"], "also_reported_by": [], "takeaway": "LynnReal-Hand-Fast streams 17 RGB frames per chunk at 540p and 18.03 FPS on one H200 using a three-step causal flow map with bounded sliding context, but its conditioning caches are separate from Base's and cannot be reused.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps", "markdown": "https://wpnews.pro/news/lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps.md", "text": "https://wpnews.pro/news/lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps.txt", "jsonld": "https://wpnews.pro/news/lynnreal-hand-hand-controlled-video-world-models-with-streaming-at-18-fps.jsonld"}}