LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS LynnReal-AI published arXiv:2611.21479 on November 17, 2026 and released open inference code for LynnReal-Hand, a hand-controlled video world model that maps egocentric MANO-scale hand poses and camera trajectories into object interactions rather than camera-only fly-throughs. The 33B-parameter model ships in Base (bidirectional, 1344x768, 30 schedule points, CFG 3) and Fast (causal streaming, 960x540, three flow-map steps) variants; the Fast path streams at 18.03 FPS at 540p on one H200 with 741 ms mean warmed response per chunk and 34.89 GiB peak allocated GPU memory. On HandBench the authors report FVD 181.67 and MANO Dice 93.02 for the full-sequence model and MANO Dice 92.49 for the three-step causal model, with the Ultra-Fast-VAE decoder cutting decode latency from 831.30 ms (Native H3) to 46.77 ms at 1344x768. LynnReal-Hand is a hand-controlled video world model that maps egocentric hand motion and camera trajectories into coherent object interactions , not just camera fly-throughs. The team published arXiv:2611.21479 https://arxiv.org/abs/2611.21479?ref=explainx on November 17, 2026 and shipped open inference code https://github.com/LynnReal-AI/LynnReal?ref=explainx with Base and Fast variants. If you build embodied agents or simulators, the control channel matters as much as pixels. Text can say "pick up the cup." A wearable or teleop rig gives you continuous MANO-scale hand poses every frame. LynnReal-Hand is built for that second interface. What problem does LynnReal-Hand solve? Interactive video world models already accept text prompts and camera paths . That is enough for exploration, but manipulation needs contact-rich motion : fingers closing on a handle, bimanual coordination, releases. The abstract https://arxiv.org/abs/2611.21479?ref=explainx states the core difficulty: hand-object dynamics are hard to model, and real-time computation is expensive. Prior work such as hand-conditional generators improved clips, yet full streaming at usable frame rates on one GPU remained scarce. LynnReal-Hand unifies interaction modeling with efficient streaming generation through four design pieces named in the paper: Robot hand reaching a small globe, illustrating hand-conditioned embodied video world models | Component | Role | |---|---| | 100K interaction clips | Real and synthetic environments with hand-aware supervision | | Causal flow map | Few-step generation with a three-step Fast variant | | Region-aware score distillation | Combines video and image priors on contact regions | | Ultra-Fast-VAE | Lightweight decoder to cut latency | Base versus Fast: two operating points Wireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motion The GitHub README https://github.com/LynnReal-AI/LynnReal?ref=explainx separates LynnReal-Hand-Base and LynnReal-Hand-Fast : | Model | Backbone | Mode | Default resolution | Sampling | |---|---|---|---|---| | LynnReal-Hand-Base | 33B | Bidirectional full sequence | 1344 x 768 | 30 schedule points, CFG 3 | | LynnReal-Hand-Fast | 33B | Causal streaming | 960 x 540 768p config exists | Three flow-map steps, W2 context | Base targets quality-first offline renders. Fast targets online use: fixed first-frame anchor, bounded sliding context, and local temporal-position resets so attention memory does not grow without bound. The Fast path emits 17 RGB frames per chunk at 540p. The CLI documents separate conditioning caches for Base and Fast; you cannot reuse a Base cache for Fast inference. Reported benchmarks: HandBench and latency On HandBench https://arxiv.org/abs/2611.21479?ref=explainx , the authors report: - FVD 181.67 and MANO Dice 93.02 for the full-sequence model - MANO Dice 92.49 for the three-step causal model For streaming at 540p on one H200 , the README lists 18.03 FPS , 741 ms mean warmed response per chunk, and 34.89 GiB peak allocated GPU memory with sampling-specific AdaLN tables and heavy encoders offloaded. The repository also publishes a comparison table against JointControlVideo, Generated Reality, and Hand2World variants on latency and memory. Treat those rows as paper measurements until you run lynnreal-hand on your own hardware. Ultra-Fast-VAE decoding numbers Decoder-only benchmarks at 1344 x 768 17 frames, one H200 from the README: | Decoder | Params M | Latency ms | FPS | |---|---|---|---| | Native H3 | 2423.5 | 831.30 | 20.44 | | Ultra-Fast-VAE | 361.2 | 46.77 | 363.21 | That gap explains why the team treats VAE replacement as a first-class contribution, not an afterthought. Green leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generation How to try LynnReal-Hand locally The maintainers document a Linux + CUDA 12.8 stack with FlashAttention-3. High-level steps from the README: 1. Clone https://github.com/LynnReal-AI/LynnReal and install the package in a venv. 2. Run bash scripts/install fa3.sh and lynnreal-hand doctor . 3. Download weights: python scripts/download weights.py --component fast or base . 4. Validate packaged example 05: lynnreal-hand validate --config configs/fast.json --example examples/05/example.json . 5. Prepare cache, then infer with lynnreal-hand prepare and lynnreal-hand infer . Seven examples ship with aligned images, hand videos, camera npz files, captions, and reference MP4s . Example 05 is the featured pet interaction scene in the README. For custom data, you supply example.json with hand video , camera.npz , caption , and frame counts in 124, 158, 243, or 260 frames. Paths must stay consistent with the same field of view. Add --accelerated on Fast inference after the first compile pass to reuse CUDA graphs in-process. What you should expect on smaller GPUs The project openly targets H200 . README memory for Base full sequence is 82.59 GiB at 768p in their table. Plan cloud H200 time for serious replication; consumer cards are not the reference platform. How LynnReal-Hand fits next to other world models explainx.ai covered MBZUAI PAN 2 https://explainx.ai/blog/mbzuai-pan-2-open-world-model-simulative-reasoning-2026 as a 29B open world model for simulative agent reasoning up to 20 seconds from text actions. PAN 2 is about plan comparison before acting in latent video space. LynnReal-Hand is complementary: - Control signal: continuous hand articulation plus camera pose, not only language actions. - Use case: teleoperation previews, egocentric AR, and manipulation-centric simulators. - Architecture emphasis: causal streaming and decoder efficiency for interactive FPS . If your agent stack already uses ForceHarness https://explainx.ai/blog/forceharness-frozen-frontier-models-force-robot-harness-2026 or ScaleVLA https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026 for robot policies, LynnReal-Hand is closer to the video imagination layer that sits above low-level torque commands. For policy learned in latent video without hands, see Gemini Robotics 2 whole-body intelligence https://explainx.ai/blog/gemini-robotics-2-whole-body-intelligence-july-2026 . Hand-controlled worlds are the interface when the human body remains in the loop. Training story in plain language The paper describes Interaction-Preserving Training IPT on 100K clips with hand-aware multilevel supervision so small contact regions keep gradient signal. Causal distillation preserves control fidelity when compressing to three steps. Region-aware score distillation merges complementary video and image priors so fingers and object surfaces stay sharp while global motion stays stable. That matters because hand errors are visually loud: a floating grip breaks immersion faster than a blurry background. The team also cites related BiWM https://arxiv.org/abs/2606.10135?ref=explainx bidirectional autoregressive world modeling and LynnReal-Omni https://arxiv.org/abs/2609.15863?ref=explainx for multimodal agent workflows in the README citation block. Risks, limits, and open questions License: MiniMax H3 Community terms govern the repo. Read NOTICE before product use. Data release timing: The paper commits to datasets and training code; verify Hugging Face and GitHub release tags when you plan reproducibility studies. Evaluation generalization: HandBench scores are author-reported. Cross-benchmark comparison to closed labs is not provided in the public README. Input rigidity: Frame counts are discrete allowed lengths; arbitrary-length phone videos need resampling and camera alignment work described in the data preparation docs. LynnReal-Hand CLI workflow in practice The maintainers ship a lynnreal-hand CLI instead of a single opaque script. That matters when you integrate with robot teleop loops where prepare and infer run on different machines. | Command | GPU required | Purpose | |---|---|---| | lynnreal-hand doctor | Optional | Check FFmpeg, FlashAttention-3, CUDA | | lynnreal-hand validate | No | Verify example.json paths and frame counts | | lynnreal-hand prepare | Yes | Build conditioning cache from image + hand video + camera | | lynnreal-hand infer | Yes | Stream MP4 from prepared cache | | lynnreal-hand infer --accelerated | Yes | Reuse compiled CUDA graphs in-process | Validation without weights is deliberate: teams can fix camera alignment and caption mismatches before downloading multi-gigabyte Hugging Face packages . Example 05 pet interaction path The README spotlights example 05 because it shows bimanual affordances near a pet, a scene where hand motion dominates semantics. 1. Download Fast weights with python scripts/download weights.py --component fast . 2. Run validate on examples/05/example.json with configs/fast.json . 3. Prepare cache to outputs/cache/fast/example 05 . 4. Infer to outputs/fast/example 05.mp4 with seed 42. 5. Compare against packaged Fast reference MP4 in the same folder. If MANO alignment is wrong, output fingers may float. The README stresses that image, hand motion, and camera poses must share field of view and frame order . That is the most common integration bug in hand-conditioned video. Frame budget constraints Packaged inputs accept 124, 158, 243, or 260 frames at the source frame rate. Arbitrary phone footage needs resampling documented in data preparation guides on GitHub. For 768p Fast , switch to configs/fast 768.json and rebuild caches; you cannot hot-swap resolution on an existing 540p cache. Paper metrics versus README metrics The arXiv PDF https://arxiv.org/abs/2611.21479?ref=explainx and README occasionally use different resolutions in tables: - Paper HandBench highlights MANO Dice and FVD at reported training resolution. - README latency table mixes 768p full-sequence Base with 540p Fast chunks . When you cite numbers externally, specify which row you mean. Reviewers rejected vague "18 FPS world model" claims when the measurement is warmed Fast chunk throughput , not first-frame cold start. Comparison to Hand2World and JointControlVideo The README reproduction table positions LynnReal-Hand against Hand2World-Bi , Hand2World-AR , Generated Reality , and JointControlVideo . LynnReal-Hand-Base is slower than Hand2World-Bi on median full-sequence latency in that table, but reports higher MANO Dice on HandBench. | Direction | Implication | |---|---| | Accuracy-first offline | Consider Base at 1344 x 768 | | Teleop streaming | Fast at 540p with Ultra-Fast-VAE | | Smaller backbones | Hand2World 5B may fit tighter VRAM | None of those tradeoffs remove the H200-class assumption for official Fast benchmarks. Integration ideas for agent builders Simulators: Pipe MANO or MediaPipe hand tracks into hand video , feed SLAM poses into camera.npz , and let LynnReal-Hand predict visual consequences before executing torques on hardware. Data flywheel: Use generated failures to label grasp policies; pair with ScaleVLA https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026 only after you verify labels on real video. Safety review: Hand-conditioned worlds can depict dangerous contact. Treat outputs like any synthetic media pipeline with policy filters before user-facing playback. Troubleshooting checklist | Symptom | Likely cause | Fix | |---|---|---| | OOM on infer | Base at 768p on smaller GPU | Switch to Fast 540p or rent H200 | | Drifting fingers | Misaligned camera npz | Re-export poses with shared intrinsics | | Black frames after prepare | Missing hand video path | Run validate with --check-weights | | Slow first infer | CUDA graph compile | Keep process alive; use --accelerated | Who should care? | Reader | Takeaway | |---|---| | Robotics engineer | Test whether hand-video conditioning reduces sim-to-real gap for grasp planning demos | | AR/VR builder | Streaming 540p interaction may pair with passthrough headsets when latency holds on your GPU | | Agent researcher | Combine with text planners: language picks the task, hands supply the low-level trajectory | | Educator | Open examples make a concrete lab for world models beyond static image diffusion | Related reading Primary sources: arXiv:2611.21479 https://arxiv.org/abs/2611.21479?ref=explainx and LynnReal-AI/LynnReal https://github.com/LynnReal-AI/LynnReal?ref=explainx as of November 18, 2026.