# LynnReal-Hand: Hand-Controlled Video World Models with Streaming at 18 FPS

> Source: <https://explainx.ai/blog/lynnreal-hand-hand-controlled-video-world-model-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-18 00:00:00+00:00

**Key takeaway:** LynnReal-Hand-Fast streams 17 RGB frames per chunk at 540p and 18.03 FPS on one H200 using a three-step causal flow map with bounded sliding context, but its conditioning caches are separate from Base's and cannot be reused.

**LynnReal-Hand is a hand-controlled video world model that maps egocentric hand motion and camera trajectories into coherent object interactions**, not just camera fly-throughs. The team published [arXiv:2611.21479](https://arxiv.org/abs/2611.21479?ref=explainx) on November 17, 2026 and shipped [open inference code](https://github.com/LynnReal-AI/LynnReal?ref=explainx) with Base and Fast variants.

If you build embodied agents or simulators, the control channel matters as much as pixels. Text can say "pick up the cup." A wearable or teleop rig gives you **continuous MANO-scale hand poses** every frame. LynnReal-Hand is built for that second interface.

## What problem does LynnReal-Hand solve?

Interactive video world models already accept **text prompts** and **camera paths**. That is enough for exploration, but manipulation needs **contact-rich motion**: fingers closing on a handle, bimanual coordination, releases.

The [abstract](https://arxiv.org/abs/2611.21479?ref=explainx) states the core difficulty: hand-object dynamics are hard to model, and real-time computation is expensive. Prior work such as hand-conditional generators improved clips, yet full **streaming** at usable frame rates on one GPU remained scarce.

LynnReal-Hand unifies **interaction modeling** with **efficient streaming generation** through four design pieces named in the paper:

Robot hand reaching a small globe, illustrating hand-conditioned embodied video world models

| Component | Role | 
|---|---|
| 100K interaction clips | Real and synthetic environments with hand-aware supervision | 
| Causal flow map | Few-step generation with a three-step Fast variant | 
| Region-aware score distillation | Combines video and image priors on contact regions | 
| Ultra-Fast-VAE | Lightweight decoder to cut latency | 

## Base versus Fast: two operating points

Wireframe cube illustrating a 3D video world model control space for LynnReal-Hand hand motion

The [GitHub README](https://github.com/LynnReal-AI/LynnReal?ref=explainx) separates **LynnReal-Hand-Base** and **LynnReal-Hand-Fast**:

| Model | Backbone | Mode | Default resolution | Sampling | 
|---|---|---|---|---|
| LynnReal-Hand-Base | 33B | Bidirectional full sequence | 1344 x 768 | 30 schedule points, CFG 3 | 
| LynnReal-Hand-Fast | 33B | Causal streaming | 960 x 540 (768p config exists) | Three flow-map steps, W2 context | 

**Base** targets quality-first offline renders. **Fast** targets **online** use: fixed first-frame anchor, bounded sliding context, and local temporal-position resets so attention memory does not grow without bound.

The Fast path emits **17 RGB frames per chunk** at 540p. The CLI documents separate conditioning caches for Base and Fast; you cannot reuse a Base cache for Fast inference.

## Reported benchmarks: HandBench and latency

On [HandBench](https://arxiv.org/abs/2611.21479?ref=explainx), the authors report:

- **FVD 181.67** and**MANO Dice 93.02** for the full-sequence model
- **MANO Dice 92.49** for the three-step causal model

For streaming at **540p on one H200**, the README lists **18.03 FPS**, **741 ms** mean warmed response per chunk, and **34.89 GiB** peak allocated GPU memory with sampling-specific AdaLN tables and heavy encoders offloaded.

The repository also publishes a **comparison table** against JointControlVideo, Generated Reality, and Hand2World variants on latency and memory. Treat those rows as **paper measurements** until you run `lynnreal-hand` on your own hardware.

### Ultra-Fast-VAE decoding numbers

Decoder-only benchmarks at 1344 x 768 (17 frames, one H200) from the README:

| Decoder | Params (M) | Latency (ms) | FPS | 
|---|---|---|---|
| Native H3 | 2423.5 | 831.30 | 20.44 | 
| Ultra-Fast-VAE | 361.2 | 46.77 | 363.21 | 

That gap explains why the team treats VAE replacement as a first-class contribution, not an afterthought.

Green leaf lifted from a shaped recess, metaphor for isolating hand contact regions in video generation

## How to try LynnReal-Hand locally

The maintainers document a **Linux + CUDA 12.8** stack with FlashAttention-3. High-level steps from the README:

1. Clone `https://github.com/LynnReal-AI/LynnReal` and install the package in a venv.
2. Run `bash scripts/install_fa3.sh` and`lynnreal-hand doctor` .
3. Download weights: `python scripts/download_weights.py --component fast` (or`base` ).
4. Validate packaged example 05: `lynnreal-hand validate --config configs/fast.json --example examples/05/example.json` .
5. Prepare cache, then infer with `lynnreal-hand prepare` and`lynnreal-hand infer` .

Seven examples ship with aligned **images, hand videos, camera npz files, captions, and reference MP4s**. Example 05 is the featured **pet interaction** scene in the README.

For custom data, you supply `example.json` with `hand_video`, `camera.npz`, `caption`, and frame counts in **124, 158, 243, or 260** frames. Paths must stay consistent with the same field of view.

Add `--accelerated` on Fast inference after the first compile pass to reuse CUDA graphs in-process.

### What you should expect on smaller GPUs

The project openly targets **H200**. README memory for Base full sequence is **82.59 GiB** at 768p in their table. Plan cloud H200 time for serious replication; consumer cards are not the reference platform.

## How LynnReal-Hand fits next to other world models

explainx.ai covered **[MBZUAI PAN 2](https://explainx.ai/blog/mbzuai-pan-2-open-world-model-simulative-reasoning-2026)** as a **29B open world model** for **simulative agent reasoning** up to 20 seconds from text actions. PAN 2 is about **plan comparison before acting** in latent video space.

LynnReal-Hand is complementary:

- **Control signal:** continuous**hand articulation** plus camera pose, not only language actions.
- **Use case:** teleoperation previews, egocentric AR, and manipulation-centric simulators.
- **Architecture emphasis:****causal streaming** and decoder efficiency for**interactive FPS** .

If your agent stack already uses **[ForceHarness](https://explainx.ai/blog/forceharness-frozen-frontier-models-force-robot-harness-2026)** or **[ScaleVLA](https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026)** for robot policies, LynnReal-Hand is closer to the **video imagination layer** that sits above low-level torque commands.

For policy learned in latent video without hands, see **[Gemini Robotics 2 whole-body intelligence](https://explainx.ai/blog/gemini-robotics-2-whole-body-intelligence-july-2026)**. Hand-controlled worlds are the interface when the human body remains in the loop.

## Training story in plain language

The paper describes **Interaction-Preserving Training (IPT)** on **100K clips** with **hand-aware multilevel supervision** so small contact regions keep gradient signal. Causal distillation preserves control fidelity when compressing to three steps.

**Region-aware score distillation** merges complementary **video and image priors** so fingers and object surfaces stay sharp while global motion stays stable. That matters because hand errors are visually loud: a floating grip breaks immersion faster than a blurry background.

The team also cites related **[BiWM](https://arxiv.org/abs/2606.10135?ref=explainx)** bidirectional autoregressive world modeling and **[LynnReal-Omni](https://arxiv.org/abs/2609.15863?ref=explainx)** for multimodal agent workflows in the README citation block.

## Risks, limits, and open questions

**License:** MiniMax H3 Community terms govern the repo. Read NOTICE before product use.

**Data release timing:** The paper commits to datasets and training code; verify Hugging Face and GitHub release tags when you plan reproducibility studies.

**Evaluation generalization:** HandBench scores are author-reported. Cross-benchmark comparison to closed labs is not provided in the public README.

**Input rigidity:** Frame counts are discrete allowed lengths; arbitrary-length phone videos need resampling and camera alignment work described in the data preparation docs.

## LynnReal-Hand CLI workflow in practice

The maintainers ship a **`lynnreal-hand`** CLI instead of a single opaque script. That matters when you integrate with robot teleop loops where prepare and infer run on different machines.

| Command | GPU required | Purpose | 
|---|---|---|
| `lynnreal-hand doctor` | Optional | Check FFmpeg, FlashAttention-3, CUDA | 
| `lynnreal-hand validate` | No | Verify example.json paths and frame counts | 
| `lynnreal-hand prepare` | Yes | Build conditioning cache from image + hand video + camera | 
| `lynnreal-hand infer` | Yes | Stream MP4 from prepared cache | 
| `lynnreal-hand infer --accelerated` | Yes | Reuse compiled CUDA graphs in-process | 

**Validation without weights** is deliberate: teams can fix camera alignment and caption mismatches before downloading **multi-gigabyte Hugging Face packages**.

### Example 05 pet interaction path

The README spotlights **example 05** because it shows **bimanual affordances** near a pet, a scene where hand motion dominates semantics.

1. Download Fast weights with `python scripts/download_weights.py --component fast` .
2. Run validate on `examples/05/example.json` with`configs/fast.json` .
3. Prepare cache to `outputs/cache/fast/example_05` .
4. Infer to `outputs/fast/example_05.mp4` with seed 42.
5. Compare against packaged **Fast reference MP4** in the same folder.

If MANO alignment is wrong, output fingers may float. The README stresses that **image, hand motion, and camera poses must share field of view and frame order**. That is the most common integration bug in hand-conditioned video.

### Frame budget constraints

Packaged inputs accept **124, 158, 243, or 260 frames** at the source frame rate. Arbitrary phone footage needs resampling documented in **data preparation** guides on GitHub.

For **768p Fast**, switch to `configs/fast_768.json` and rebuild caches; you cannot hot-swap resolution on an existing 540p cache.

## Paper metrics versus README metrics

The [arXiv PDF](https://arxiv.org/abs/2611.21479?ref=explainx) and README occasionally use different resolutions in tables:

- Paper HandBench highlights **MANO Dice** and**FVD** at reported training resolution.
- README latency table mixes **768p full-sequence Base** with**540p Fast chunks** .

When you cite numbers externally, specify **which row** you mean. Reviewers rejected vague "18 FPS world model" claims when the measurement is **warmed Fast chunk throughput**, not first-frame cold start.

### Comparison to Hand2World and JointControlVideo

The README reproduction table positions LynnReal-Hand against **Hand2World-Bi**, **Hand2World-AR**, **Generated Reality**, and **JointControlVideo**. LynnReal-Hand-Base is slower than Hand2World-Bi on median full-sequence latency in that table, but reports higher **MANO Dice** on HandBench.

| Direction | Implication | 
|---|---|
| Accuracy-first offline | Consider Base at 1344 x 768 | 
| Teleop streaming | Fast at 540p with Ultra-Fast-VAE | 
| Smaller backbones | Hand2World 5B may fit tighter VRAM | 

None of those tradeoffs remove the **H200-class** assumption for official Fast benchmarks.

## Integration ideas for agent builders

**Simulators:** Pipe MANO or MediaPipe hand tracks into `hand_video`, feed SLAM poses into `camera.npz`, and let LynnReal-Hand predict **visual consequences** before executing torques on hardware.

**Data flywheel:** Use generated failures to label grasp policies; pair with **[ScaleVLA](https://explainx.ai/blog/scalevla-552b-vision-language-action-robot-backbone-2026)** only after you verify labels on real video.

**Safety review:** Hand-conditioned worlds can depict dangerous contact. Treat outputs like any synthetic media pipeline with policy filters before user-facing playback.

## Troubleshooting checklist

| Symptom | Likely cause | Fix | 
|---|---|---|
| OOM on infer | Base at 768p on smaller GPU | Switch to Fast 540p or rent H200 | 
| Drifting fingers | Misaligned camera npz | Re-export poses with shared intrinsics | 
| Black frames after prepare | Missing hand_video path | Run validate with `--check-weights` | 
| Slow first infer | CUDA graph compile | Keep process alive; use `--accelerated` | 

## Who should care?

| Reader | Takeaway | 
|---|---|
| Robotics engineer | Test whether hand-video conditioning reduces sim-to-real gap for grasp planning demos | 
| AR/VR builder | Streaming 540p interaction may pair with passthrough headsets when latency holds on your GPU | 
| Agent researcher | Combine with text planners: language picks the task, hands supply the low-level trajectory | 
| Educator | Open examples make a concrete lab for world models beyond static image diffusion | 

## Related reading

*Primary sources: [arXiv:2611.21479](https://arxiv.org/abs/2611.21479?ref=explainx) and [LynnReal-AI/LynnReal](https://github.com/LynnReal-AI/LynnReal?ref=explainx) as of November 18, 2026.*
