Inside NVIDIA’s IsaacTeleop: From Hand and Controller Tracking to Robot Actions with the Graph-Based Retargeting Engine

NVIDIA's IsaacTeleop framework converts XR hand tracking and motion-controller input into commands for simulated and real robots through a graph-based retargeting engine, according to a Marktechpost tutorial built on the isaacteleop 1.4.145 PyPI wheel with the retargeters-lite extra. The tutorial constructs all hand and controller inputs in NumPy on a plain Colab CPU, with no headset, OpenXR runtime, or simulator, and drives the built-in gripper and SE(3) retargeters to emit one action vector per step. It also covers a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts.

In this tutorial https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Robotics/nvidia isaacteleop retargeting tutorial Marktechpost.ipynb , we work through the retargeting engine at the core of NVIDIA IsaacTeleop https://github.com/NVIDIA/IsaacTeleop , the framework that turns XR hand tracking and motion-controller input into commands for simulated and real robots. Rather than plugging in a headset, we build every input ourselves in NumPy, so each step runs on a plain Colab CPU and prints what it computes. We start with the type system every node speaks, generate synthetic hand and controller data, write our own retargeter with live-tunable parameters, and then drive the built-in gripper and SE 3 retargeters with it. From there, we compose a full graph that emits one action vector per step, apply a world-frame transform, step through the run, pause and kill the state machine, and finish with a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts. python import os import sys import json import math import tempfile import traceback import subprocess import numpy as np RESULTS = {} def banner title : print "\n" + "=" 78 print title print "=" 78 def section name : def wrap fn : def run a, kw : banner name try: out = fn a, kw RESULTS name = out if isinstance out, str else "ok" return out except Exception as e: RESULTS name = f"SKIPPED / FAILED - {type e . name }: {e}" print f"\n {name} did not complete: {type e . name }: {e}" traceback.print exc limit=3 return None return run return wrap banner "0. Install isaacteleop and check the environment" subprocess.run sys.executable, "-m", "pip", "install", "-q", "isaacteleop retargeters-lite ==1.4.145" , check=True, import pkgutil import isaacteleop from isaacteleop import schema print f" isaacteleop {isaacteleop. version } | Python {sys.version.split 0 } | numpy {np. version }" print " top-level modules :", ", ".join sorted m.name for m in pkgutil.iter modules isaacteleop. path message types = n for n in dir schema if n 0 .isupper print f" schema message types: {len message types }, e.g. {', '.join message types :6 }" print " no headset, no OpenXR runtime, no simulator is used anywhere below." We install the stable isaacteleop wheel from PyPI with the retargeters-lite extra, which adds only SciPy, and report the version, interpreter, and NumPy build. The package splits into device I/O modules that wrap OpenXR and CloudXR, a schema module holding the FlatBuffer message types every tracker emits, and the pure-Python retargeting engine we work with here. Listing the schema types shows the vocabulary of the data layer, but nothing below opens a headset session; from here on, every input is a tensor we build by hand. from isaacteleop.retargeting engine.interface import TensorGroup, OptionalTensorGroup, TensorGroupType, OptionalType, from isaacteleop.retargeting engine.tensor types import HandInput, ControllerInput, HandInputIndex, ControllerInputIndex, HandJointIndex, @section "1. The type contract: TensorGroupType, TensorGroup, Optional" def type contract : hand t = HandInput ctrl t = ControllerInput print " HandInput :", hand t print " slot index :", ", ".join f"{m.name}={m.value}" for m in HandInputIndex print f" ControllerInput: {len ctrl t } slots - ", ", ".join t.name.replace "controller ", "" for t in ctrl t.types print f" OpenXR hand joints: {len HandJointIndex } " f" WRIST={int HandJointIndex.WRIST }, THUMB TIP={int HandJointIndex.THUMB TIP }, " f"INDEX TIP={int HandJointIndex.INDEX TIP } " hand = TensorGroup hand t hand HandInputIndex.JOINT POSITIONS = np.zeros 26, 3 , dtype=np.float32 print "\n wrote a 26, 3 float32 array into JOINT POSITIONS - ", hand try: hand HandInputIndex.JOINT POSITIONS = np.zeros 26, 3 except TypeError as e: print " float64 rejected at write time :", str e :110 try: = hand HandInputIndex.JOINT VALID except ValueError as e: print " reading a slot nobody wrote :", e maybe = OptionalTensorGroup OptionalType ctrl t print f"\n Optional group starts absent : is none={maybe.is none} {maybe}" maybe ControllerInputIndex.TRIGGER VALUE = 0.0 print f" one write flips it to present : is none={maybe.is none} {maybe}" return f"{len hand t } hand slots, {len ctrl t } controller slots" type contract The engine’s contract is a TensorGroupType, an ordered list of typed slots, and a TensorGroup, the runtime container that holds one value per slot and validates each write. HandInput carries four NumPy arrays for the 26 OpenXR hand joints, and ControllerInput carries fourteen slots for poses, buttons, and axes, addressed through generated IntEnum indices rather than magic numbers. Writing a float64 array where float32 is declared fails at the write, and reading a slot nobody wrote raises instead of returning stale data. OptionalType marks inputs a tracker may not deliver; the matching OptionalTensorGroup starts absent and becomes present on its first write, which is how every downstream node learns that a hand has left the tracking volume. J = HandJointIndex C = ControllerInputIndex RIGHT WRIST = np.array 0.30, 1.05, -0.45 , dtype=np.float32 OpenXR: x right, y up, z back def make hand pinch m, wrist=RIGHT WRIST, quat= 0.0, 0.0, 0.0, 1.0 : """A synthetic right hand in HandInput layout: 26 joints in OpenXR order.""" pos = np.zeros 26, 3 , dtype=np.float32 pos J.WRIST = wrist pos J.PALM = wrist + 0.0, 0.0, -0.06 fingers = J.INDEX METACARPAL, 0.03 , J.MIDDLE METACARPAL, 0.01 , J.RING METACARPAL, -0.01 , J.LITTLE METACARPAL, -0.03 for base, x off in fingers: metacarpal .. tip, five joints each for k in range 5 : pos base + k = wrist + x off, 0.0, -0.05 - 0.02 k for k in range 4 : thumb: metacarpal .. tip, four joints pos J.THUMB METACARPAL + k = wrist + 0.05, -0.01, -0.02 - 0.015 k pos J.THUMB TIP = pos J.INDEX TIP + pinch m, 0.0, 0.0 g = TensorGroup HandInput g HandInputIndex.JOINT POSITIONS = pos g HandInputIndex.JOINT ORIENTATIONS = np.tile np.asarray quat, dtype=np.float32 , 26, 1 g HandInputIndex.JOINT RADII = np.full 26, 0.01, dtype=np.float32 g HandInputIndex.JOINT VALID = np.ones 26, dtype=np.uint8 return g def make controller pos, quat= 0.0, 0.0, 0.0, 1.0 , trigger=0.0, squeeze=0.0, thumbstick= 0.0, 0.0 , valid=True : """A synthetic controller snapshot in ControllerInput layout.""" p = np.asarray pos, dtype=np.float32 q = np.asarray quat, dtype=np.float32 g = TensorGroup ControllerInput g C.GRIP POSITION , g C.GRIP ORIENTATION , g C.GRIP IS VALID = p, q, bool valid g C.AIM POSITION = p + np.array 0.0, 0.0, -0.05 , dtype=np.float32 g C.AIM ORIENTATION , g C.AIM IS VALID = q.copy , bool valid for idx in C.PRIMARY CLICK, C.SECONDARY CLICK, C.THUMBSTICK CLICK, C.MENU CLICK : g idx = 0.0 g C.THUMBSTICK X , g C.THUMBSTICK Y = float thumbstick 0 , float thumbstick 1 g C.SQUEEZE VALUE , g C.TRIGGER VALUE = float squeeze , float trigger return g @section "2. Synthetic tracking data: a hand and a controller in numpy" def synthetic inputs : hand = make hand pinch m=0.06 pos = hand HandInputIndex.JOINT POSITIONS print " joint x y z" for j in J.WRIST, J.PALM, J.THUMB TIP, J.INDEX TIP, J.LITTLE TIP : print f" {j.name:12s} {pos j 0 :6.3f} {pos j 1 :6.3f} {pos j 2 :6.3f}" pinch = np.linalg.norm pos J.THUMB TIP - pos J.INDEX TIP print f" thumb-to-index distance: {pinch 100:.1f} cm " f"valid joints: {int hand HandInputIndex.JOINT VALID .sum }/26" ctrl = make controller 0.40, 1.20, -0.30 , trigger=0.8, thumbstick= 0.0, 0.5 print f"\n controller grip {np.round ctrl C.GRIP POSITION , 2 } aim {np.round ctrl C.AIM POSITION , 2 } " f"trigger {ctrl C.TRIGGER VALUE } squeeze {ctrl C.SQUEEZE VALUE } " f"thumbstick {ctrl C.THUMBSTICK X }, {ctrl C.THUMBSTICK Y } " return "synthetic HandInput + ControllerInput builders" synthetic inputs We build the tracking data that a headset would normally supply. make hand lays out 26 joints in OpenXR order around a wrist position, runs four finger chains and a thumb chain away from the palm, and places the thumb tip a chosen pinch distance from the index tip, so the one number the later steps depend on is under our control. make controller fills every ControllerInput slot: grip and aim poses with validity flags, four buttons, the thumbstick axes, and the analog squeeze and trigger values. Both return ordinary TensorGroups, which is all a retargeter ever sees, whether the numbers came from OpenXR or from NumPy. from isaacteleop.retargeting engine.interface import BaseRetargeter, ParameterState, FloatParameter, BoolParameter, from isaacteleop.retargeting engine.tensor types import FloatType, BoolType class PinchRetargeter BaseRetargeter : """Thumb-to-index distance - distance cm, is pinching , with live-tunable parameters.""" def init self, name, config file=None : params = FloatParameter "threshold cm", "Pinching when closer than this", default value=3.0, min value=0.5, max value=10.0, sync fn=lambda v: setattr self, "threshold cm", v , BoolParameter "use distal joints", "Measure between distal joints, not tips", default value=False, sync fn=lambda v: setattr self, "use distal joints", v , super . init name, parameter state=ParameterState name, params, config file=config file def input spec self : return {"hand right": OptionalType HandInput } def output spec self : return {"pinch": TensorGroupType "pinch", FloatType "distance cm" , BoolType "is pinching" } def compute fn self, inputs, outputs, context : hand = inputs "hand right" if hand.is none: tracking lost: say so, do not guess outputs "pinch" 0 = -1.0 outputs "pinch" 1 = False return pos = np.from dlpack hand HandInputIndex.JOINT POSITIONS a, b = J.THUMB DISTAL, J.INDEX DISTAL if self.use distal joints else J.THUMB TIP, J.INDEX TIP d cm = float np.linalg.norm pos a - pos b 100.0 outputs "pinch" 0 = d cm outputs "pinch" 1 = bool d cm < self.threshold cm @section "3. Write a retargeter: pinch detection with live-tunable parameters" def custom retargeter : pinch = PinchRetargeter "pinch" for d in 0.06, 0.02 : out = pinch {"hand right": make hand d } print f" hand at {d 100:.0f} cm - distance cm={out 'pinch' 0 :.2f} is pinching={out 'pinch' 1 }" out = pinch {} optional input omitted entirely print f" no hand tracked - distance cm={out 'pinch' 0 :.1f} is pinching={out 'pinch' 1 }" state = pinch.get parameter state print "\n tunable parameters:", state.get all values for threshold in 2.5, 3.5 : state.set {"threshold cm": threshold} what the tuning UI does from its own thread out = pinch {"hand right": make hand 0.028 } print f" threshold cm={threshold}: a 2.8 cm pinch - is pinching={out 'pinch' 1 }" state.set {"use distal joints": True} out = pinch {"hand right": make hand 0.028 } print f" use distal joints=True: distance cm={out 'pinch' 0 :.2f} is pinching={out 'pinch' 1 }" return "custom retargeter + 2 tunable parameters" custom retargeter A retargeter is a BaseRetargeter subclass that declares input spec and output spec and implements compute fn; the framework fills missing optional inputs with absent groups, validates types, syncs parameters, and only then calls our code. PinchRetargeter measures the thumb-to-index distance and emits a float and a bool, reporting -1 when no hand is tracked instead of guessing. Its two parameters live in a ParameterState whose sync functions write onto the instance before every compute, so a value set from another thread by the tuning UI takes effect on the next frame. We reproduce that by calling set directly: the same 2.8 cm pinch flips between not pinching and pinching as the threshold moves, and switching the measurement to the distal joints changes the distance itself. from isaacteleop.retargeters import GripperRetargeter, GripperRetargeterConfig, Se3AbsRetargeter, Se3RelRetargeter, Se3RetargeterConfig, @section "4. Built-in retargeters: gripper hysteresis and SE 3 end-effector pose" def builtin retargeters : gripper = GripperRetargeter GripperRetargeterConfig hand side="right", gripper close meters=0.03, gripper open meters=0.05 , name="gripper", print " pinch sweep close below 3 cm, open above 5 cm, hold in between :" for d in 0.07, 0.045, 0.028, 0.040, 0.052, 0.020 : cmd = gripper {"hand right": make hand d } "gripper command" 0 print f" {d 100:4.1f} cm - {cmd:+.0f} {'CLOSED' if cmd < 0 else 'open'}" cmd = gripper {"hand right": make hand 0.07 , "controller right": make controller 0.3, 1.0, -0.4 , trigger=0.9 } "gripper command" 0 print f" open hand + trigger 0.9 - {cmd:+.0f} a present controller outranks hand tracking " se3 = Se3AbsRetargeter Se3RetargeterConfig input device="controller right", target offset roll=90.0, zero out xy rotation=True , name="ee pose", ctrl = make controller 0.40, 1.20, -0.30 pose = np.from dlpack se3 {"controller right": ctrl} "ee pose" 0 print f"\n Se3Abs: grip {np.round ctrl C.GRIP POSITION , 2 } - ee pose pos {np.round pose :3 , 3 } " f"quat xyzw {np.round pose 3: , 3 }" lost = make controller 9.0, 9.0, 9.0 , valid=False pose2 = np.from dlpack se3 {"controller right": lost} "ee pose" 0 print f" grip is valid=False - holds the last pose: {np.round pose2 :3 , 3 }" rel = Se3RelRetargeter Se3RetargeterConfig input device="controller right", delta pos scale factor=10.0, alpha pos=0.5 , name="ee delta", print "\n Se3Rel: the controller moves +2 cm in x every frame delta x10, EMA alpha 0.5 :" for i in range 4 : ctrl = make controller 0.40 + 0.02 i, 1.20, -0.30 delta = np.from dlpack rel {"controller right": ctrl} "ee delta" 0 print f" frame {i}: ee delta dx, dy, dz, rx, ry, rz = {np.round delta, 3 }" return "gripper, Se3Abs and Se3Rel driven from synthetic input" builtin retargeters We drive two of the built-in retargeters with the same synthetic groups. GripperRetargeter turns pinch distance into the -1/+1 gripper command Isaac Lab expects, with hysteresis: the gripper closes below 3 cm, opens above 5 c,m and holds its state in between, so 4.0 cm stays closed after a close and the sweep back through 5.2 cm reopens it. A present controller takes priority, and a trigger above the threshold closes an open hand. Se3AbsRetargeter maps the controller grip pose to a 7-D end-effector target, applying the configured roll offset and keeping only yaw when zero out xy rotation is set. It holds the last pose when the grip becomes invalid rather than passing a zero quaternion downstream. Se3RelRetargeter emits deltas instead: a constant 2 cm step per frame appears scaled by ten and smoothed by the EMA, converging toward 0.2. python from isaacteleop.retargeting engine.interface import ValueInput, OutputCombiner from isaacteleop.retargeters import TensorReorderer class CountingInput ValueInput : """A ValueInput leaf that counts how many times the graph asked it to compute.""" def init self, name, tensor type : super . init name, tensor type self.calls = 0 def compute fn self, inputs, outputs, context : self.calls += 1 super . compute fn inputs, outputs, context def build pipeline : controller = CountingInput "controller right", OptionalType ControllerInput hand = CountingInput "hand right", OptionalType HandInput ee pose = Se3AbsRetargeter Se3RetargeterConfig input device="controller right" , name="ee pose" .connect {"controller right": controller.output "value" } gripper = GripperRetargeter GripperRetargeterConfig hand side="right" , name="gripper" .connect {"controller right": controller.output "value" , "hand right": hand.output "value" } ee = "pos x", "pos y", "pos z", "quat x", "quat y", "quat z", "quat w" action = TensorReorderer input config={"ee pose": ee, "gripper command": "gripper" }, output order=ee + "gripper" , name="action", input types={"ee pose": "array", "gripper command": "scalar"}, .connect {"ee pose": ee pose.output "ee pose" , "gripper command": gripper.output "gripper command" } return OutputCombiner {"action": action.output "output" } , controller, hand @section "5. Compose the graph: leaves - retargeters - one action vector per step" def compose graph : pipeline, controller, hand = build pipeline print " leaf nodes :", n.name for n in pipeline.get leaf nodes print " outputs :", {k: str v for k, v in pipeline.output types .items } print "\n frame trigger action = x, y, z, qx, qy, qz, qw, gripper " for f in range 6 : t = f / 6 trigger = 1.0 if f = 4 else 0.0 ctrl = make controller 0.40 + 0.10 math.cos 2 math.pi t , 1.20, -0.30 + 0.10 math.sin 2 math.pi t , trigger=trigger leaf inputs = {"controller right": {"value": ctrl}, "hand right": {"value": make hand 0.06 }} action = np.from dlpack pipeline.execute pipeline leaf inputs "action" 0 print f" {f:3d} {trigger:.1f} {np.round action, 3 }" print f"\n the controller leaf feeds two retargeters, yet computed {controller.calls} times " f"in 6 frames: one ExecutionCache per step" return f"8-D action vector; {controller.calls} leaf computes for 6 frames" compose graph This is the shape of a real Isaac Teleop pipeline. ValueInput nodes stand in for the DeviceIO source nodes as graph leaves, connect wires each retargeter’s inputs to upstream outputs with type checking at connect time, TensorReorderer flattens the 7-D pose and the gripper scalar into the action layout an environment expects, and OutputCombiner exposes the result under a single action key. execute pipeline takes inputs keyed by leaf name and runs the DAG once per step against a shared ExecutionCache, so the controller leaf, although wired into both the SE 3 and the gripper retargeter, computes exactly once per frame, which our counting subclass confirms. Swapping the leaves for HandsSource and ControllersSource is the only change needed to run this graph against a headset. python from scipy.spatial.transform import Rotation from isaacteleop.retargeting engine.utilities import ControllerTransform from isaacteleop.retargeting engine.tensor types import TransformMatrix @section "6. Coordinate frames: ControllerTransform with a world T anchor matrix" def coordinate frames : world T anchor = np.eye 4, dtype=np.float32 world T anchor :3, :3 = Rotation.from euler "z", 90, degrees=True .as matrix .astype np.float32 world T anchor :3, 3 = 1.0, 0.0, 0.5 xf = TensorGroup TransformMatrix xf 0 = world T anchor ctrl = make controller 0.40, 1.20, -0.30 , trigger=0.7 out = ControllerTransform "controller xform" {"controller right": ctrl, "transform": xf} p in, p out = ctrl C.GRIP POSITION , out "controller right" C.GRIP POSITION print f" grip position anchor frame {np.round p in, 3 } - world frame {np.round p out, 3 }" print f" check R @ p + t = {np.round world T anchor :3, :3 @ p in + world T anchor :3, 3 , 3 }" print f" grip orientation xyzw {np.round ctrl C.GRIP ORIENTATION , 3 } - " f"{np.round out 'controller right' C.GRIP ORIENTATION , 3 }" print f" trigger passes through {ctrl C.TRIGGER VALUE } - {out 'controller right' C.TRIGGER VALUE }" print f" left controller not given - {out 'controller left' }" return "yaw 90 deg + translation applied to grip and aim, inputs untouched" coordinate frames Headsets report poses in the runtime’s anchor frame and the robot lives in the world frame, so every pipeline that talks to a simulator applies a world T anchor transform first. ControllerTransform takes the two optional controller groups plus a TransformMatrix group holding a 4×4 homogeneous matrix, rewrites the grip and aim positions as R p + t and the orientations by the rotation part, and passes buttons, axes and validity flags through untouched. We verify the position against the same matrix product in NumPy and see the absent left controller propagate as absent, which is what lets a single-controller session flow through a two-controller graph. python from isaacteleop.teleop session manager import DefaultTeleopStateManager, bool signal from isaacteleop.retargeting engine.interface import ComputeContext, ExecutionEvents, ExecutionState from isaacteleop.retargeters import LocomotionRootCmdRetargeter, LocomotionRootCmdRetargeterConfig def button name, pressed : g = TensorGroup bool signal name g 0 = bool pressed return g def state name out : st = out "teleop state" return next t.name for i, t in enumerate st.group type.types if st i @section "7. The control state machine: run, pause, kill, and the reset pulse" def state machine : sm = DefaultTeleopStateManager "state" script = "idle", 0, 0, 0 , "press run", 1, 0, 0 , "release", 0, 0, 0 , "press run", 1, 0, 0 , "hold run", 1, 0, 0 , "release", 0, 0, 0 , "press reset", 0, 0, 1 , "release", 0, 0, 0 , "KILL", 0, 1, 0 , "release", 0, 0, 0 print " input run kill reset | state reset event" for label, run, kill, reset in script: out = sm {"run toggle button": button "run toggle button", run , "kill button": button "kill button", kill , "reset button": button "reset button", reset } print f" {label:12s} {run} {kill} {reset} | {state name out :8s} {out 'reset event' 0 }" out = sm {"run toggle button": button "run toggle button", 0 } print f" kill signal lost | {state name out :8s} fail-safe: required input absent " loco = LocomotionRootCmdRetargeter LocomotionRootCmdRetargeterConfig initial hip height=0.72 , name="loco" print "\n right thumbstick Y=+1 raises the hip height each frame; a reset pulse snaps it back:" for i, reset in enumerate False, False, False, True, False : ctx = ComputeContext execution events=ExecutionEvents reset=reset, execution state=ExecutionState.RUNNING cmd = loco {"controller left": make controller 0, 1, 0 , thumbstick= 0.0, 0.6 , "controller right": make controller 0, 1, 0 , thumbstick= 0.2, 1.0 }, context=ctx "root command" 0 print f" frame {i} reset={str reset :5s} root command vx, vy, wz, hip = {np.round cmd, 4 }" return "STOPPED - PAUSED - RUNNING - STOPPED, reset pulses delivered through ComputeContext" state machine Teleoperation needs a way to arm, pause, and kill the robot that doesn’t depend on the retargeters behaving. DefaultTeleopStateManager is itself a retargeter: three optional bool inputs go in, a one-hot teleop state and a reset event pulse come out. A rising edge on the run toggle walks STOPPED to PAUSED to RUNNING and back to PAUSED, the reset button emits a one-frame pulse without changing state, kill forces STOPPED and pulses reset, and losing the kill or run signal fails safe to STOPPED. Those events reach every other node through ComputeContext.execution events; LocomotionRootCmdRetargeter integrates the right thumbstick into a hip height each frame and, on the frame carrying reset=True, snaps back to its initial height before integrating again. python from isaacteleop.retargeters import TriHandMotionControllerRetargeter, TriHandMotionControllerConfig @section "8. Controller - dexterous hand joints, and tuned parameters that survive a restart" def trihand and persistence : joints = "thumb rotation", "thumb proximal", "thumb distal", "index proximal", "index distal", "middle proximal", "middle distal" tri = TriHandMotionControllerRetargeter TriHandMotionControllerConfig hand joint names=joints, controller side="right" , name="trihand right" print " trigger squeeze |" + "".join f"{j :10 : 11s}" for j in joints for trig, sq in 0.0, 0.0 , 1.0, 0.0 , 0.0, 1.0 , 1.0, 1.0 : out = tri {"controller right": make controller 0.3, 1.0, -0.4 , trigger=trig, squeeze=sq } "hand joints" print f" {trig:.1f} {sq:.1f} |" + "".join f"{out i :11.2f}" for i in range 7 cfg path = os.path.join tempfile.mkdtemp , "ee pose tuning.json" se3 = Se3AbsRetargeter Se3RetargeterConfig input device="controller right", parameter config path=cfg path , name="ee pose" se3.get parameter state .set {"rotation offset rpy": np.array 90.0, 0.0, 45.0 , "position offset xyz": np.array 0.0, 0.0, 0.10 } print se3.get parameter state .save to file print " saved :", json.load open cfg path ctrl = make controller 0.4, 1.2, -0.3 pose a = np.from dlpack se3 {"controller right": ctrl} "ee pose" 0 se3 b = Se3AbsRetargeter Se3RetargeterConfig input device="controller right", parameter config path=cfg path , name="ee pose restarted" pose b = np.from dlpack se3 b {"controller right": ctrl} "ee pose" 0 print f" tuned instance ee pose = {np.round pose a, 3 }" print f" restarted instance ee pose = {np.round pose b, 3 } <- same tuning, read back from JSON" return "7-DOF trihand mapping; parameter JSON round trip" trihand and persistence Two production concerns close the tutorial. TriHandMotionControllerRetargeter is the controller-only route to a dexterous hand: trigger drives the index finger, squeeze drives the middle finger, the larger of the two curls the thumb and their difference rotates it, giving seven named joint angles with no hand tracking and no optimization library. Then we make tuning stick: setting the rotation and position offsets on Se3AbsRetargeter’s ParameterState and calling save to file writes a JSON file, and a fresh instance constructed with the same parameter config path loads it on startup and produces an identical end-effector pose, which is how a calibration tuned in the UI survives a restart of the session. banner "SUMMARY" for name, res in RESULTS.items : print f" {name:<78s} {res}" print """ Where to go next - Put a headset on the graph: swap the ValueInput leaves for HandsSource / ControllersSource and run the same pipeline inside TeleopSession with CloudXRLauncher examples/teleop/python/gripper retargeting example simple.py . - Record and replay: McapRecordingConfig / McapReplayConfig on TeleopSessionConfig replay a .mcap through this exact graph with no headset attached. - Drive a simulator: the action vector from Step 5 is what Isaac Lab's IsaacTeleopDevice consumes; the TensorReorderer order must match the env. - Tune live: MultiRetargeterTuningUI isaacteleop ui edits the same ParameterState objects you set by hand in Steps 3 and 8. """ The summary collects the headline of every section from the RESULTS dictionary that the section decorator filled in, so a skipped or failed step shows up here with its error instead of silently disappearing from the run. In conclusion, we built a working Isaac Teleop pipeline without any of the hardware the framework is designed around, because everything below the device layer is plain Python operating on typed tensor groups. The type system caught wrong dtypes and unwritten slots at the point of the mistake; absent optional groups gave every node one explicit way to learn that tracking was lost; and ParameterState let us tune behavior between frames and keep it on disk. The built-in gripper, SE 3 , locomotion and TriHand retargeters behaved as documented under synthetic input, and composing them with ValueInput leaves, TensorReorderer and OutputCombiner produced the same action vector a headset session would feed to Isaac Lab, with each leaf computed once per frame. Where hardware enters, the graph does not change: the leaves become HandsSource and ControllersSource inside a TeleopSession, a recording becomes an MCAP replay, and the anchor transform we applied by hand arrives from the simulator. Check out the FULL CODES here https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Robotics/nvidia isaacteleop retargeting tutorial Marktechpost.ipynb . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter https://x.com/intent/follow?screen name=marktechpost and don’t forget to join our 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ and Subscribe to our Newsletter https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}} . Wait are you on telegram? now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/MJjjVDPS7whH8Ngs6