Meta InteractiveWorld: A Simulator and 120-Task Benchmark for Wearable AI Assistants

Meta's research group quietly published a new repository, facebookresearch/InteractiveWorld, on October 24, 2026. It is the official code for a paper titled "InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants." In plain terms, it is a simulator and benchmark for AI assistants that live on your face: smart glasses and similar wearables that see what you see, talk to you, and are expected to help while you move around a real home.

The repo had one star when we read it, so this is not a viral story. It is a release from a top lab's research org that points at where wearable AI is heading, and it gives builders a concrete way to test an assistant on tasks that text-only benchmarks cannot touch. It sits next to the hardware push we covered in Meta Connect 2026 and the Muse gadgets SDK.

TL;DR: what InteractiveWorld is #

Why a wearable assistant needs its own benchmark #

Most multimodal benchmarks ask a model a question about a fixed image or clip and grade the written answer. A wearable assistant works differently. It has a body to talk to, and that body changes the scene. If the assistant says "the keys are on the shelf behind you," the person turns, moves a jacket, and the next frame is different. Whether the advice worked depends on what happened in the world, not on how fluent the sentence was.

That is the gap the README describes: "the assistant sees through a person's eyes and talks, the person decides and acts, and a stateful 3D world responds with the next first-person view." The loop is observe, communicate, act, update the world. Three design choices follow from it:

• Closed-loop assistance. The assistant's words have consequences that show up in the next frame.
• Human-centered simulation. An animated human walks, grasps, opens and uses objects, rather than a floating camera or a robot gripper.
• Verifiable evaluation. A task passes only when every goal predicate holds in the final world state.

The third point matters most. LLM-as-judge scoring of conversations is cheap but noisy, and it is easy for an assistant to sound helpful while the person ends up with the wrong snack. A state check removes that ambiguity. This is the same instinct behind the verifiable robot evaluations we covered in LIBERO-PERM, where success is something you can measure in a scene.

The two layers: simulator and benchmark #

The repo has two layers, according to its own description.

InteractiveWorld is the reusable simulator. Its simulator guide describes three internal layers: a physical layer (BEHAVIOR-1K scenes in Isaac Sim with object physics and state), a human-action layer (an SMPL+H human that executes commands with planned stances, routes, reaches and grips) and a semantic layer for free-form human and assistant interaction grounded in the live scene.

You drive the human with short commands. The README's first example opens a scene and chains [Walk], [Grab], [Put] and [Sit] commands on a can of soda, a coffee table and a sofa. The human works out where to stand, how to walk and how to grip. There are 36 action families, from movement and looking to opening doors, cutting, stirring, pouring, wiping, using a computer and answering a phone.

InteractiveBench sits on top: 120 frozen multi-turn tasks, with one model as the assistant and a fixed second model as the human.

Illustrative simulation scene of a block moved between trays, standing in for a stateful simulated world (not an InteractiveWorld screenshot)

The six task categories #

The benchmark guide lists 20 tasks per category, each with an example ID:

Two details stand out. First, the Accessibility tasks withhold the images from the simulated human, so the assistant has to do the seeing. That is a test of scene description under a real constraint, closer to what a blind user wants than a captioning score. Second, Prevention is scored on restraint: warn before a mistake, but do not interrupt needlessly. Over-talking is a failure mode every glasses assistant will face.

The guide also notes that 22 continuous-watch tasks (all of Prevention, plus one Decision and one Finding task) keep the human acting while the assistant watches, and they run only through the benchmark runner.

How an episode runs #

According to the guide, an episode goes like this:

1. The simulator opens the scene and replays a prologue , what the person did before asking (reading a report, taking a call, moving things). The assistant sees those first-person frames, which is what makes the "Finding" tasks a memory test.
2. The human speaks the opening request.
3. Turns alternate. The assistant sees only frames and dialogue and replies with speech, silence or a done declaration. The human reads the reply and the scene and answers with speech and simulator commands.
4. The simulator executes the commands and renders the next frames.
5. The episode ends when the assistant declares done, the human finishes, or the tick budget runs out.

"Silence" is a legal move. That is a quiet but important design decision, because a proactive assistant that never learns when to stay quiet is exactly the product people complain about, a theme in our look at proactive assistants like Hark Pro.

Two scores: native and release rubric #

Every task gets two independent scores:

• Native TaskPack score , available immediately.strict_success is true when every goal predicate holds in the final world state and process requirements are met;completion_score is the fraction of goals met.
• Release rubric score , which the docs call the official benchmark number. It uses 464 criteria across six protocols in a rubric file. The run writes a template per task; a person checks the transcript and video, fills in the criteria, and reruns with the adjudication folder.

So the headline number is human-adjudicated, not fully automatic. That raises the cost of a leaderboard-quality run, and it is worth knowing before you plan an evaluation. The docs also state a run is VALID only when the simulator, the human and the assistant transport all behaved; invalid runs are reported rather than counted as failures, which avoids penalizing a model for infrastructure errors.

How to try it #

The setup is heavy. You need four downloads, then one prepare script:

1. Isaac Sim 5.1, OmniGibson 3.9.0 and the 31 GB BEHAVIOR-1K dataset through the BEHAVIOR-1K installer (accepting the NVIDIA Isaac Sim EULA and dataset terms).
2. pip install -r requirements.txt in the InteractiveWorld clone, and setISAAC_PYTHON .
3. The SMPL+H (MANO) body model and ACCAD walking motion from the MANO and AMASS sites, which require registration and cannot be redistributed.
4. ./launch/prepare_assets.sh , which bakes 6,831 BEHAVIOR objects, converts the body model and verifies frozen checksums; the README says baking takes about two minutes with eight workers.

Then ./launch/simulator.sh doctor should print "ok": true. Discovery commands ( worlds, capabilities, tasks list) run without launching Isaac Sim, so you can browse the 44 scenes, 36 actions and 120 tasks on a laptop first. Opening a raw scene takes roughly 2 to 4 minutes and a task 5 to 10 minutes, per the docs, and a single task takes about 5 to 20 minutes depending on length and model latency.

To evaluate a model, one command picks the models, tasks and GPUs. Here is the README's example with a locally served open model as the assistant and a hosted model as the human:

vllm serve Qwen/Qwen3-VL-8B-Instruct --port 8000

env -u CUDA_VISIBLE_DEVICES ./launch/interactiveworld.sh \
  --assistant-provider openai-compatible \
  --assistant-model Qwen/Qwen3-VL-8B-Instruct \
  --assistant-base-url http://127.0.0.1:8000/v1 \
  --human-provider gemini \
  --human-model gemini-3.1-pro-preview \
  --human-api-key-env GEMINI_API_KEY \
  --category Guidance \
  --num-gpus 2 \
  --video failures \
  --output-dir runs/qwen3-vl-8b-guidance

Use --dry-run first to print the execution plan without starting Isaac Sim. Keys are read from a named environment variable, never from the command line, and remote endpoints must use HTTPS. The paper uses Gemini 3.1 Pro as the human for every assistant so results stay comparable; the docs say to keep the human fixed when you compare assistants.

You can also add your own tasks. A TaskPack is one JSON file covering the scene and object placement, the success state, the prologue, how the human reacts to guidance and an oracle solution. The repo's example script derives a new task from a bundled one, and custom tasks should live outside the repository because the release tree is integrity-checked before every benchmark run.

What is claimed, what is verified #

We read the repository and its docs. We did not run the simulator, which needs an A100-class GPU and about 100 GB of assets, and we did not locate the paper's text, so we are not reporting any benchmark results.

The last row is the honest limit. A simulated human in a BEHAVIOR-1K home is a controlled stand-in for a person in a messy kitchen. The simulator can verify that the soda ended up on the coffee table; it cannot tell you whether a real wearer would have understood the assistant's phrasing, or whether the glasses' camera would have seen the shelf at all.

Limitations and open questions #

• Heavy and non-commercial. Isaac Sim, a 31 GB dataset, registration-gated body and motion data, and a CC BY-NC license make this a research tool, not something to drop into a product pipeline.
• The human is a model. Results depend on the fixed human model's behavior (Gemini 3.1 Pro in the paper's setup), so a score is a score of an assistant paired with that human, not of an assistant in general.
• Adjudication cost. The official number needs a person to check criteria against transcripts and video.
• Speed. Several minutes per task and CPU-bound planning mean a full 120-task sweep is a multi-GPU job.
• Scope. Indoor home scenes only; no outdoors, crowds, or noisy audio.

Why it matters for builders #

If you build assistants for glasses, earbuds with cameras, or any always-on multimodal helper, the useful idea is not the Isaac Sim stack but the evaluation design: separate the assistant from a fixed simulated user, give the assistant only what the device would see, let silence be a valid action, and score the end state. You can copy that pattern with a much lighter environment for your own product tests.

For the wider field, it is another sign that labs are moving from "does the model describe the picture" to "does the model change what happens next." That is the same shift behind egocentric training pipelines we explained in egocentric data collection for robots and world-model releases like NVIDIA Cosmos 3 and Long-WAM. It also raises the same privacy questions as any always-watching camera, which our smart glasses misuse coverage and BGBlur's guide to blurring faces in video walk through.

Watch for two follow-ups: a published results table from the authors, and third-party runs of open models such as Qwen3-VL as the assistant. When either appears we will add an update here. For product context on the hardware side, see our Meta VR glasses reaction roundup.

Related reading #

Details reflect the repository as of October 25, 2026; the project is new and may change.