cd /news/artificial-intelligence/lynnreal-hand-hand-controlled-video-… · home › topics › artificial-intelligence › article
[ARTICLE · art-171709] src=explainx.ai ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS

LynnReal-AI published arXiv:2611.21479 on November 17, 2026 and released open inference code for LynnReal-Hand, a hand-controlled video world model that maps egocentric MANO-scale hand poses and camera trajectories into object interactions rather than camera-only fly-throughs. The 33B-parameter model ships in Base (bidirectional, 1344x768, 30 schedule points, CFG 3) and Fast (causal streaming, 960x540, three flow-map steps) variants; the Fast path streams at 18.03 FPS at 540p on one H200 with 741 ms mean warmed response per chunk and 34.89 GiB peak allocated GPU memory. On HandBench the authors report FVD 181.67 and MANO Dice 93.02 for the full-sequence model and MANO Dice 92.49 for the three-step causal model, with the Ultra-Fast-VAE decoder cutting decode latency from 831.30 ms (Native H3) to 46.77 ms at 1344x768.

key takeaway

LynnReal-Hand-Fast streams 17 RGB frames per chunk at 540p and 18.03 FPS on one H200 using a three-step causal flow map with bounded sliding context, but its conditioning caches are separate from Base's and cannot be reused.

by read9 min views2 publishedNov 18, 2026
LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS
Image: Explainx (auto-discovered)

LynnReal-Hand is a hand-controlled video world model that maps egocentric hand motion and camera trajectories into coherent object interactions, not just camera fly-throughs. The team published arXiv:2611.21479 on November 17, 2026 and shipped open inference code with Base and Fast variants.

If you build embodied agents or simulators, the control channel matters as much as pixels. Text can say "pick up the cup." A wearable or teleop rig gives you continuous MANO-scale hand poses every frame. LynnReal-Hand is built for that second interface.

What problem does LynnReal-Hand solve? #

Interactive video world models already accept text prompts and camera paths. That is enough for exploration, but manipulation needs contact-rich motion: fingers closing on a handle, bimanual coordination, releases.

The abstract states the core difficulty: hand-object dynamics are hard to model, and real-time computation is expensive. Prior work such as hand-conditional generators improved clips, yet full streaming at usable frame rates on one GPU remained scarce.

LynnReal-Hand unifies interaction modeling with efficient streaming generation through four design pieces named in the paper:

Robot hand reaching a small globe, illustrating hand-conditioned embodied video world models

Component Role
100K interaction clips Real and synthetic environments with hand-aware supervision
Causal flow map Few-step generation with a three-step Fast variant
Region-aware score distillation Combines video and image priors on contact regions
Ultra-Fast-VAE Lightweight decoder to cut latency

Base versus Fast: two operating points #

Wireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motion

The GitHub README separates LynnReal-Hand-Base and LynnReal-Hand-Fast:

Model Backbone Mode Default resolution Sampling
LynnReal-Hand-Base 33B Bidirectional full sequence 1344 x 768 30 schedule points, CFG 3
LynnReal-Hand-Fast 33B Causal streaming 960 x 540 (768p config exists) Three flow-map steps, W2 context

Base targets quality-first offline renders. Fast targets online use: fixed first-frame anchor, bounded sliding context, and local temporal-position resets so attention memory does not grow without bound.

The Fast path emits 17 RGB frames per chunk at 540p. The CLI documents separate conditioning caches for Base and Fast; you cannot reuse a Base cache for Fast inference.

Reported benchmarks: HandBench and latency #

On HandBench, the authors report:

  • FVD 181.67 andMANO Dice 93.02 for the full-sequence model
  • MANO Dice 92.49 for the three-step causal model

For streaming at 540p on one H200, the README lists 18.03 FPS, 741 ms mean warmed response per chunk, and 34.89 GiB peak allocated GPU memory with sampling-specific AdaLN tables and heavy encoders offloaded. The repository also publishes a comparison table against JointControlVideo, Generated Reality, and Hand2World variants on latency and memory. Treat those rows as paper measurements until you run lynnreal-hand on your own hardware.

### Ultra-Fast-VAE decoding numbers

Decoder-only benchmarks at 1344 x 768 (17 frames, one H200) from the README:

| Decoder | Params (M) | Latency (ms) | FPS | 
|---|---|---|---|

| Native H3 | 2423.5 | 831.30 | 20.44 | | Ultra-Fast-VAE | 361.2 | 46.77 | 363.21 |

That gap explains why the team treats VAE replacement as a first-class contribution, not an afterthought.

Green leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generation

How to try LynnReal-Hand locally #

The maintainers document a Linux + CUDA 12.8 stack with FlashAttention-3. High-level steps from the README:

  1. Clone https://github.com/LynnReal-AI/LynnReal and install the package in a venv.
  2. Run bash scripts/install_fa3.sh andlynnreal-hand doctor .
  3. Download weights: python scripts/download_weights.py --component fast (orbase ).
  4. Validate packaged example 05: lynnreal-hand validate --config configs/fast.json --example examples/05/example.json .
  5. Prepare cache, then infer with lynnreal-hand prepare andlynnreal-hand infer .

Seven examples ship with aligned images, hand videos, camera npz files, captions, and reference MP4s. Example 05 is the featured pet interaction scene in the README.

For custom data, you supply example.json with hand_video, camera.npz, caption, and frame counts in 124, 158, 243, or 260 frames. Paths must stay consistent with the same field of view. Add --accelerated on Fast inference after the first compile pass to reuse CUDA graphs in-process.

What you should expect on smaller GPUs

The project openly targets H200. README memory for Base full sequence is 82.59 GiB at 768p in their table. Plan cloud H200 time for serious replication; consumer cards are not the reference platform.

How LynnReal-Hand fits next to other world models #

explainx.ai covered MBZUAI PAN 2 as a 29B open world model for simulative agent reasoning up to 20 seconds from text actions. PAN 2 is about plan comparison before acting in latent video space.

LynnReal-Hand is complementary:

  • Control signal: continuoushand articulation plus camera pose, not only language actions.
  • Use case: teleoperation previews, egocentric AR, and manipulation-centric simulators.
  • Architecture emphasis:**causal streaming and decoder efficiency forinteractive FPS** .
If your agent stack already uses **[ForceHarness](https://explainx.ai/blog/forceharness-frozen-frontier-models-force-robot-harness-2026)** or **[ScaleVLA](https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026)** for robot policies, LynnReal-Hand is closer to the **video imagination layer** that sits above low-level torque commands.

For policy learned in latent video without hands, see **[Gemini Robotics 2 whole-body intelligence](https://explainx.ai/blog/gemini-robotics-2-whole-body-intelligence-july-2026)**. Hand-controlled worlds are the interface when the human body remains in the loop.

Training story in plain language #

The paper describes Interaction-Preserving Training (IPT) on 100K clips with hand-aware multilevel supervision so small contact regions keep gradient signal. Causal distillation preserves control fidelity when compressing to three steps.

Region-aware score distillation merges complementary video and image priors so fingers and object surfaces stay sharp while global motion stays stable. That matters because hand errors are visually loud: a floating grip breaks immersion faster than a blurry background.

The team also cites related BiWM bidirectional autoregressive world modeling and LynnReal-Omni for multimodal agent workflows in the README citation block.

Risks, limits, and open questions #

License: MiniMax H3 Community terms govern the repo. Read NOTICE before product use.

Data release timing: The paper commits to datasets and training code; verify Hugging Face and GitHub release tags when you plan reproducibility studies.

Evaluation generalization: HandBench scores are author-reported. Cross-benchmark comparison to closed labs is not provided in the public README.

Input rigidity: Frame counts are discrete allowed lengths; arbitrary-length phone videos need resampling and camera alignment work described in the data preparation docs.

LynnReal-Hand CLI workflow in practice #

The maintainers ship a lynnreal-hand CLI instead of a single opaque script. That matters when you integrate with robot teleop loops where prepare and infer run on different machines.

Command GPU required Purpose
lynnreal-hand doctor Optional Check FFmpeg, FlashAttention-3, CUDA
lynnreal-hand validate No Verify example.json paths and frame counts
lynnreal-hand prepare Yes Build conditioning cache from image + hand video + camera
lynnreal-hand infer Yes Stream MP4 from prepared cache
lynnreal-hand infer --accelerated Yes Reuse compiled CUDA graphs in-process

Validation without weights is deliberate: teams can fix camera alignment and caption mismatches before down multi-gigabyte Hugging Face packages.

Example 05 pet interaction path

The README spotlights example 05 because it shows bimanual affordances near a pet, a scene where hand motion dominates semantics.

  1. Download Fast weights with python scripts/download_weights.py --component fast .
  2. Run validate on examples/05/example.json withconfigs/fast.json .
  3. Prepare cache to outputs/cache/fast/example_05 .
  4. Infer to outputs/fast/example_05.mp4 with seed 42.
  5. Compare against packaged Fast reference MP4 in the same folder.

If MANO alignment is wrong, output fingers may float. The README stresses that image, hand motion, and camera poses must share field of view and frame order. That is the most common integration bug in hand-conditioned video.

Frame budget constraints

Packaged inputs accept 124, 158, 243, or 260 frames at the source frame rate. Arbitrary phone footage needs resampling documented in data preparation guides on GitHub.

For 768p Fast, switch to configs/fast_768.json and rebuild caches; you cannot hot-swap resolution on an existing 540p cache.

Paper metrics versus README metrics #

The arXiv PDF and README occasionally use different resolutions in tables:

  • Paper HandBench highlights MANO Dice andFVD at reported training resolution.
  • README latency table mixes 768p full-sequence Base with540p Fast chunks .

When you cite numbers externally, specify which row you mean. Reviewers rejected vague "18 FPS world model" claims when the measurement is warmed Fast chunk throughput, not first-frame cold start.

Comparison to Hand2World and JointControlVideo

The README reproduction table positions LynnReal-Hand against Hand2World-Bi, Hand2World-AR, Generated Reality, and JointControlVideo. LynnReal-Hand-Base is slower than Hand2World-Bi on median full-sequence latency in that table, but reports higher MANO Dice on HandBench.

Direction Implication
Accuracy-first offline Consider Base at 1344 x 768
Teleop streaming Fast at 540p with Ultra-Fast-VAE
Smaller backbones Hand2World 5B may fit tighter VRAM

None of those tradeoffs remove the H200-class assumption for official Fast benchmarks.

Integration ideas for agent builders #

Simulators: Pipe MANO or MediaPipe hand tracks into hand_video, feed SLAM poses into camera.npz, and let LynnReal-Hand predict visual consequences before executing torques on hardware.

Data flywheel: Use generated failures to label grasp policies; pair with ScaleVLA only after you verify labels on real video.

Safety review: Hand-conditioned worlds can depict dangerous contact. Treat outputs like any synthetic media pipeline with policy filters before user-facing playback.

Troubleshooting checklist #

Symptom Likely cause Fix
OOM on infer Base at 768p on smaller GPU Switch to Fast 540p or rent H200
Drifting fingers Misaligned camera npz Re-export poses with shared intrinsics
Black frames after prepare Missing hand_video path Run validate with --check-weights
Slow first infer CUDA graph compile Keep process alive; use --accelerated

Who should care? #

Reader Takeaway
Robotics engineer Test whether hand-video conditioning reduces sim-to-real gap for grasp planning demos
AR/VR builder Streaming 540p interaction may pair with passthrough headsets when latency holds on your GPU
Agent researcher Combine with text planners: language picks the task, hands supply the low-level trajectory
Educator Open examples make a concrete lab for world models beyond static image diffusion

Primary sources: arXiv:2611.21479 and LynnReal-AI/LynnReal as of November 18, 2026.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @lynnreal-ai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/lynnreal-hand-hand-c…] indexed:0 read:9min 2026-11-18 · —