{"slug": "lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export", "title": "LingBot-Map Tutorial: GPU-Aware Inference and Point Cloud Export", "summary": "LingBot-Map, an open-source streaming 3D reconstruction pipeline, enables GPU-aware inference and point cloud export through a tutorial that auto-tunes frame limits, camera iterations, scale frames, and KV-cache parameters based on detected VRAM. The pipeline, available on GitHub, processes image or video frames to reconstruct scenes and export results as PLY, NPZ, or GLB files, with configurations for different GPU tiers (small <18 GB, medium 18-30 GB, large 30+ GB).", "body_md": "In this tutorial, we implement an end-to-end streaming 3D reconstruction pipeline with[ LingBot-Map](https://github.com/robbyant/lingbot-map). We begin by configuring the input source, reconstruction settings, checkpoint selection, and output controls, then probe the available GPU and automatically tune frame limits, camera iterations, scale frames, and KV-cache parameters according to the detected VRAM. We install the repository and its dependencies, download the pretrained checkpoint, preprocess image or video frames, and construct the GCTStream model with streaming attention and long-range trajectory memory. We then perform mixed-precision inference, decode the predicted camera poses and intrinsic parameters, convert depth maps into world-coordinate point clouds, validate the recovered geometry, visualize the reconstructed scene and camera trajectory, and export the results as PLY, NPZ, or GLB artifacts.\n\n```\nCFG = {\n   \"scene\":        \"courthouse\",\n   \"image_folder\": None,\n   \"video_path\":   None,\n   \"fps\":          10,\n   \"max_frames\":   None,\n   \"stride\":       None,\n   \"checkpoint\":   \"lingbot-map.pt\",\n   \"image_size\":   518,\n   \"patch_size\":   14,\n   \"use_sdpa\":     True,\n   \"mode\":                  \"streaming\",\n   \"num_scale_frames\":      None,\n   \"keyframe_interval\":     None,\n   \"kv_cache_sliding_window\": 64,\n   \"camera_num_iterations\": None,\n   \"offload_to_cpu\":        True,\n   \"window_size\":           128,\n   \"overlap_keyframes\":     8,\n   \"conf_percentile\":  55.0,\n   \"pixel_stride\":     2,\n   \"max_plot_points\":  60000,\n   \"export_ply\":       True,\n   \"export_glb\":       False,\n   \"launch_viser\":     False,\n   \"run_ablation\":     False,\n   \"seed\":             0,\n}\nWORK = \"/content\"\nREPO = f\"{WORK}/lingbot-map\"\nOUT  = f\"{WORK}/lingbot_out\"\nimport os, sys, subprocess, glob, json, time, math, shutil, textwrap\nos.environ.setdefault(\"PYTORCH_CUDA_ALLOC_CONF\", \"expandable_segments:True\")\nos.makedirs(OUT, exist_ok=True)\ndef sh(cmd, check=True):\n   p = subprocess.run(cmd, shell=True, capture_output=True, text=True)\n   if p.returncode != 0 and check:\n       print(p.stdout[-3000:]); print(p.stderr[-3000:])\n       raise RuntimeError(f\"command failed: {cmd}\")\n   return p\ndef probe_gpu():\n   try:\n       q = sh(\"nvidia-smi --query-gpu=name,memory.total --format=csv,noheader\", check=False)\n       if q.returncode != 0 or not q.stdout.strip():\n           return None, 0.0\n       name, mem = [x.strip() for x in q.stdout.strip().split(\"\\n\")[0].split(\",\")]\n       return name, float(mem.split()[0]) / 1024.0\n   except Exception:\n       return None, 0.0\nGPU_NAME, VRAM_GB = probe_gpu()\nprint(\"=\" * 78)\nprint(f\"GPU        : {GPU_NAME or 'NONE — enable Runtime > Change runtime type > GPU'}\")\nprint(f\"VRAM       : {VRAM_GB:.1f} GB\")\ntry:\n   import psutil\n   print(f\"System RAM : {psutil.virtual_memory().total/2**30:.1f} GB\")\nexcept Exception:\n   pass\nprint(f\"Free disk  : {shutil.disk_usage(WORK).free/2**30:.1f} GB   (checkpoint is 4.6 GB)\")\nprint(\"=\" * 78)\nif GPU_NAME is None:\n   raise SystemExit(\"This model needs a CUDA GPU. Switch the Colab runtime to GPU.\")\nif VRAM_GB < 18:\n   AUTO = dict(max_frames=48, num_scale_frames=4, camera_num_iterations=2, kvsw=48)\n   tier = \"small (<18 GB)\"\nelif VRAM_GB < 30:\n   AUTO = dict(max_frames=96, num_scale_frames=8, camera_num_iterations=4, kvsw=64)\n   tier = \"medium (18-30 GB)\"\nelse:\n   AUTO = dict(max_frames=240, num_scale_frames=8, camera_num_iterations=4, kvsw=64)\n   tier = \"large (30+ GB)\"\nif CFG[\"max_frames\"] is None: CFG[\"max_frames\"] = AUTO[\"max_frames\"]\nif CFG[\"num_scale_frames\"] is None: CFG[\"num_scale_frames\"] = AUTO[\"num_scale_frames\"]\nif CFG[\"camera_num_iterations\"] is None: CFG[\"camera_num_iterations\"] = AUTO[\"camera_num_iterations\"]\nif VRAM_GB < 18: CFG[\"kv_cache_sliding_window\"] = AUTO[\"kvsw\"]\nprint(f\"Auto-tuned for {tier}: max_frames={CFG['max_frames']}, \"\n     f\"scale_frames={CFG['num_scale_frames']}, \"\n     f\"cam_iters={CFG['camera_num_iterations']}, \"\n     f\"kv_window={CFG['kv_cache_sliding_window']}\")\nprint(\"Raise CFG['max_frames'] if you have headroom — reconstruction quality \"\n     \"improves a lot with more views.\\n\")\n```\n\nWe define the configuration parameters that control the input source, model behavior, inference mode, point-cloud generation, and export settings. We inspect the available GPU, system memory, and disk space before automatically adjusting frame limits, scale frames, camera iterations, and KV-cache size according to the detected VRAM. We also create reusable shell and GPU-probing functions that prepare the Colab environment for the remaining reconstruction workflow.\n\n```\nif not os.path.isdir(REPO):\n   print(\"Cloning repo (shallow) ...\")\n   sh(f\"git clone --depth 1 https://github.com/Robbyant/lingbot-map.git {REPO}\")\nprint(\"Installing deps ...\")\nsh(\"pip install -q einops safetensors huggingface_hub\")\nsh(f\"pip install -q -e {REPO} --no-deps\", check=False)\nif REPO not in sys.path:\n   sys.path.insert(0, REPO)\nimport importlib\nimportlib.invalidate_caches()\nimport cv2, numpy as np\nprint(f\"numpy {np.__version__} | opencv {cv2.__version__}\")\nfrom huggingface_hub import hf_hub_download\nprint(f\"\\nFetching {CFG['checkpoint']} from robbyant/lingbot-map ...\")\nt0 = time.time()\nCKPT = hf_hub_download(repo_id=\"robbyant/lingbot-map\",\n                      filename=CFG[\"checkpoint\"],\n                      cache_dir=f\"{WORK}/hf_cache\")\nprint(f\"  -> {CKPT}  ({os.path.getsize(CKPT)/2**30:.2f} GB, {time.time()-t0:.0f}s)\")\n```\n\nWe clone the LingBot-Map repository and install the required dependencies without replacing Colab’s existing NumPy and OpenCV packages. We register the repository in the Python environment and verify the installed library versions to ensure that the runtime is ready for execution. We then download the selected pretrained LingBot-Map checkpoint from Hugging Face and store its local path for model initialization.\n\n``` python\nimport torch\nimport matplotlib.pyplot as plt\nfrom lingbot_map.models.gct_stream import GCTStream\nfrom lingbot_map.utils.load_fn import load_and_preprocess_images\nfrom lingbot_map.utils.pose_enc import pose_encoding_to_extri_intri\nfrom lingbot_map.utils.geometry import (\n   closed_form_inverse_se3,\n   closed_form_inverse_se3_general,\n   depth_to_world_coords_points,\n)\ntorch.manual_seed(CFG[\"seed\"]); np.random.seed(CFG[\"seed\"])\nDEVICE = torch.device(\"cuda\")\nCC = torch.cuda.get_device_capability()\nDTYPE = torch.bfloat16 if CC[0] >= 8 else torch.float16\nprint(f\"torch {torch.__version__} | sm_{CC[0]}{CC[1]} | inference dtype {DTYPE}\")\ndef extract_video_frames(video_path, out_dir, fps=10):\n   os.makedirs(out_dir, exist_ok=True)\n   cap = cv2.VideoCapture(video_path)\n   src_fps = cap.get(cv2.CAP_PROP_FPS) or 30\n   interval = max(1, round(src_fps / fps))\n   paths, idx = [], 0\n   while True:\n       ok, frame = cap.read()\n       if not ok:\n           break\n       if idx % interval == 0:\n           p = os.path.join(out_dir, f\"{len(paths):06d}.jpg\")\n           cv2.imwrite(p, frame)\n           paths.append(p)\n       idx += 1\n   cap.release()\n   print(f\"  decoded {idx} frames @ {src_fps:.1f} fps -> kept {len(paths)} (every {interval})\")\n   return paths\ndef gather_paths():\n   if CFG[\"video_path\"]:\n       src = f\"{OUT}/video_frames\"\n       return extract_video_frames(CFG[\"video_path\"], src, CFG[\"fps\"]), src\n   folder = CFG[\"image_folder\"] or f\"{REPO}/example/{CFG['scene']}\"\n   if not os.path.isdir(folder):\n       raise FileNotFoundError(folder)\n   paths = sorted(sum([glob.glob(os.path.join(folder, f\"*{e}\"))\n                       for e in (\".jpg\", \".jpeg\", \".png\", \".JPG\", \".PNG\")], []))\n   if not paths:\n       raise FileNotFoundError(f\"no images in {folder}\")\n   return paths, folder\nprint(\"\\n[5] Loading frames\")\nall_paths, SRC_FOLDER = gather_paths()\nprint(f\"  source: {SRC_FOLDER}  ({len(all_paths)} frames available)\")\nstride = CFG[\"stride\"]\nif stride is None:\n   stride = max(1, math.ceil(len(all_paths) / CFG[\"max_frames\"]))\npaths = all_paths[::stride][:CFG[\"max_frames\"]]\nprint(f\"  stride={stride} -> using {len(paths)} frames \"\n     f\"(covering {(len(paths)-1)*stride+1}/{len(all_paths)} of the sequence)\")\nt0 = time.time()\nimages = load_and_preprocess_images(\n   paths, mode=\"crop\",\n   image_size=CFG[\"image_size\"], patch_size=CFG[\"patch_size\"],\n)\nS, _, H, W = images.shape\nn_tok = (H // CFG[\"patch_size\"]) * (W // CFG[\"patch_size\"])\nprint(f\"  tensor {tuple(images.shape)} in [0,1]  |  {W}x{H}  |  ~{n_tok} patch tokens/frame\"\n     f\"  |  {time.time()-t0:.1f}s\")\nk = min(6, S)\nfig, ax = plt.subplots(1, k, figsize=(3 * k, 2.4))\nfor i, a in enumerate(np.atleast_1d(ax)):\n   a.imshow(images[int(i * (S - 1) / max(k - 1, 1))].permute(1, 2, 0).numpy())\n   a.set_title(f\"frame {int(i*(S-1)/max(k-1,1))}\", fontsize=9); a.axis(\"off\")\nplt.suptitle(\"Input frames (after crop/resize)\"); plt.tight_layout(); plt.show()\n```\n\nWe import PyTorch, LingBot-Map components, geometry utilities, and visualization libraries before selecting the appropriate inference precision for the available GPU. We load frames from a bundled scene, custom image directory, or video file and uniformly sample them according to the configured frame limit and stride. We preprocess the selected images into model-ready tensors and preview representative frames after cropping and resizing.\n\n```\nprint(\"\\n[6] Building GCTStream\")\nmodel = GCTStream(\n   img_size=CFG[\"image_size\"],\n   patch_size=CFG[\"patch_size\"],\n   enable_3d_rope=True,\n   max_frame_num=1024,\n   kv_cache_sliding_window=CFG[\"kv_cache_sliding_window\"],\n   kv_cache_scale_frames=CFG[\"num_scale_frames\"],\n   kv_cache_cross_frame_special=True,\n   kv_cache_include_scale_frames=True,\n   use_sdpa=CFG[\"use_sdpa\"],\n   camera_num_iterations=CFG[\"camera_num_iterations\"],\n)\nn_par = sum(p.numel() for p in model.parameters())\nprint(f\"  {n_par/1e9:.2f} B params  |  heads: \"\n     f\"camera={model.camera_head is not None}, depth={model.depth_head is not None}, \"\n     f\"point={model.point_head is not None}\")\nprint(\"  loading state dict ...\")\nt0 = time.time()\ntry:\n   ckpt = torch.load(CKPT, map_location=\"cpu\", weights_only=False, mmap=True)\nexcept Exception:\n   ckpt = torch.load(CKPT, map_location=\"cpu\", weights_only=False)\nsd = ckpt.get(\"model\", ckpt)\nmissing, unexpected = model.load_state_dict(sd, strict=False)\ndel ckpt, sd\nprint(f\"  loaded in {time.time()-t0:.0f}s | missing={len(missing)} unexpected={len(unexpected)}\")\nif len(missing) > 50:\n   print(\"  !! lots of missing keys — wrong checkpoint for this model class?\")\nmodel = model.to(DEVICE).eval()\nif model.aggregator is not None:\n   model.aggregator = model.aggregator.to(dtype=DTYPE)\ntorch.cuda.empty_cache()\nprint(f\"  GPU after load: {torch.cuda.memory_allocated()/2**30:.2f} GB allocated, \"\n     f\"{torch.cuda.memory_reserved()/2**30:.2f} GB reserved\")\nkfi = CFG[\"keyframe_interval\"]\nif kfi is None:\n   kfi = math.ceil(S / 320) if (CFG[\"mode\"] == \"streaming\" and S > 320) else 1\nprint(f\"\\n[7] {CFG['mode']} inference | frames={S} | keyframe_interval={kfi} | dtype={DTYPE}\")\ntorch.cuda.reset_peak_memory_stats()\nout_dev = torch.device(\"cpu\") if CFG[\"offload_to_cpu\"] else None\nt0 = time.time()\nwith torch.no_grad(), torch.amp.autocast(\"cuda\", dtype=DTYPE):\n   if CFG[\"mode\"] == \"streaming\":\n       preds = model.inference_streaming(\n           images,\n           num_scale_frames=CFG[\"num_scale_frames\"],\n           keyframe_interval=kfi,\n           output_device=out_dev,\n       )\n   else:\n       preds = model.inference_windowed(\n           images,\n           window_size=CFG[\"window_size\"],\n           overlap_size=16,\n           overlap_keyframes=CFG[\"overlap_keyframes\"],\n           num_scale_frames=CFG[\"num_scale_frames\"],\n           keyframe_interval=kfi,\n           output_device=out_dev,\n       )\nelapsed = time.time() - t0\nprint(f\"  done in {elapsed:.1f}s  ->  {S/elapsed:.2f} FPS\")\nprint(f\"  GPU peak: {torch.cuda.max_memory_allocated()/2**30:.2f} GB allocated \"\n     f\"/ {torch.cuda.max_memory_reserved()/2**30:.2f} GB reserved\")\nprint(f\"  outputs: \" + \", \".join(f\"{k}{tuple(v.shape)}\" for k, v in preds.items()\n                                if torch.is_tensor(v)))\ntry:\n   print(f\"  kv cache: {model.get_kv_cache_info()}\")\nexcept Exception:\n   pass\nmodel.clean_kv_cache()\ntorch.cuda.empty_cache()\n```\n\nWe construct the GCTStream model with streaming KV-cache controls, camera refinement, 3D rotary embeddings, and scaled dot-product attention. We load the pretrained checkpoint, move the model to the GPU, cast the aggregation trunk to the selected precision, and report its parameter count and memory usage. We then run streaming or windowed inference, record processing speed and peak GPU consumption, and collect the predicted images, depths, confidence maps, and camera pose encodings.\n\n``` python\nprint(\"\\n[8] Decoding camera poses\")\ndef decode_poses(pose_enc, hw):\n   extr, intr = pose_encoding_to_extri_intri(pose_enc, hw)\n   E = torch.zeros((*extr.shape[:-2], 4, 4), device=extr.device, dtype=extr.dtype)\n   E[..., :3, :4] = extr\n   E[..., 3, 3] = 1.0\n   E = closed_form_inverse_se3_general(E)\n   return E[..., :3, :4], intr\ndef unbatch(t):\n   return t[0] if (torch.is_tensor(t) and t.shape[0] == 1) else t\nextrinsic, intrinsic = decode_poses(preds[\"pose_enc\"].float(), (H, W))\nextrinsic = unbatch(extrinsic).cpu().numpy()\nintrinsic = unbatch(intrinsic).cpu().numpy()\ndepth = unbatch(preds[\"depth\"]).float().cpu().numpy()\ndepth_conf = unbatch(preds[\"depth_conf\"]).float().cpu().numpy()\nrgb = unbatch(preds[\"images\"]).float().cpu().numpy()\nc2w = closed_form_inverse_se3(extrinsic)\ncam_centers = c2w[:, :3, 3]\npath_len = float(np.linalg.norm(np.diff(cam_centers, axis=0), axis=1).sum())\nprint(f\"  focal (px)     : {intrinsic[0,0,0]:.1f}   principal pt \"\n     f\"({intrinsic[0,0,2]:.1f}, {intrinsic[0,1,2]:.1f})\")\nprint(f\"  depth range    : {np.percentile(depth,1):.3f} .. {np.percentile(depth,99):.3f} \"\n     f\"(arbitrary but metric-consistent units)\")\nprint(f\"  trajectory len : {path_len:.3f}  |  extent \"\n     f\"{np.ptp(cam_centers,axis=0).round(3).tolist()}\")\nprint(\"\\n[9] Unprojecting depth to a world point cloud\")\ndef conf_threshold(conf, pct):\n   sub = conf[::max(1, len(conf) // 20)].ravel()\n   return float(np.percentile(sub, pct))\nTHR = conf_threshold(depth_conf, CFG[\"conf_percentile\"])\nprint(f\"  conf threshold @ {CFG['conf_percentile']}th pct = {THR:.3f}\")\nps = CFG[\"pixel_stride\"]\npts_all, col_all, frame_id = [], [], []\nfor i in range(S):\n   wp, _, valid = depth_to_world_coords_points(\n       depth[i].squeeze(-1), extrinsic[i], intrinsic[i]\n   )\n   m = valid & (depth_conf[i] > THR)\n   m_sub = np.zeros_like(m); m_sub[::ps, ::ps] = True\n   m = m & m_sub\n   if not m.any():\n       continue\n   pts_all.append(wp[m].astype(np.float32))\n   col_all.append(rgb[i].transpose(1, 2, 0)[m].astype(np.float32))\n   frame_id.append(np.full(int(m.sum()), i, dtype=np.int32))\npoints = np.concatenate(pts_all, 0)\ncolors = np.clip(np.concatenate(col_all, 0), 0, 1)\nframes_of = np.concatenate(frame_id, 0)\ndel pts_all, col_all, frame_id\nprint(f\"  {len(points):,} points kept \"\n     f\"({100*len(points)/(S*H*W):.1f}% of all pixels, stride {ps}, conf>{THR:.2f})\")\ni = S // 2\nwp, _, valid = depth_to_world_coords_points(depth[i].squeeze(-1), extrinsic[i], intrinsic[i])\nm = valid & (depth_conf[i] > THR)\nratio = np.linalg.norm(wp[m] - cam_centers[i], axis=1) / np.maximum(depth[i].squeeze(-1)[m], 1e-6)\nprint(f\"  sanity: median ||p - cam|| / depth = {np.median(ratio):.3f} \"\n     f\"(expected ~1.00-1.15) -> {'OK' if 0.98 < np.median(ratio) < 1.35 else 'SUSPECT'}\")\nlo, hi = np.percentile(points, [1, 99], axis=0)\nscene_scale = float(np.linalg.norm(hi - lo))\nprint(f\"  scene diagonal : {scene_scale:.3f}\")\n```\n\nWe decode the predicted pose encodings into camera extrinsic and intrinsic matrices and calculate the reconstructed camera trajectory. We convert every confidence-filtered depth map into world-coordinate points while preserving the corresponding RGB values and source-frame identifiers. We also perform a geometric quality check and calculate robust scene boundaries to verify that the reconstructed point cloud follows the expected camera and depth conventions.\n\n```\nprint(\"\\n[10] Plots\")\nk = min(4, S)\nidxs = np.linspace(0, S - 1, k).astype(int)\nfig, axes = plt.subplots(3, k, figsize=(3.1 * k, 7.2))\naxes = np.atleast_2d(axes)\nfor c, i in enumerate(idxs):\n   axes[0, c].imshow(rgb[i].transpose(1, 2, 0)); axes[0, c].set_title(f\"frame {i}\", fontsize=9)\n   d = depth[i].squeeze(-1)\n   axes[1, c].imshow(d, cmap=\"turbo\", vmin=np.percentile(d, 2), vmax=np.percentile(d, 98))\n   axes[2, c].imshow(depth_conf[i] > THR, cmap=\"gray\")\nfor r, lbl in enumerate([\"RGB\", \"depth\", f\"conf > {THR:.2f}\"]):\n   axes[r, 0].set_ylabel(lbl, fontsize=10)\nfor a in axes.ravel():\n   a.set_xticks([]); a.set_yticks([])\nplt.tight_layout(); plt.savefig(f\"{OUT}/depth_strip.png\", dpi=110); plt.show()\nplt.figure(figsize=(11, 3))\nplt.subplot(1, 2, 1)\nplt.hist(depth_conf[::max(1, S // 15)].ravel(), bins=120, color=\"#4477aa\")\nplt.axvline(THR, color=\"crimson\", ls=\"--\", label=f\"threshold {THR:.2f}\")\nplt.yscale(\"log\"); plt.xlabel(\"depth confidence\"); plt.ylabel(\"pixels (log)\")\nplt.legend(); plt.title(\"Confidence distribution\")\nplt.subplot(1, 2, 2)\nplt.plot([(depth_conf[i] > THR).mean() for i in range(S)], lw=1.4, color=\"#228833\")\nplt.xlabel(\"frame\"); plt.ylabel(\"fraction kept\"); plt.ylim(0, 1)\nplt.title(\"Per-frame confident-pixel ratio\")\nplt.tight_layout(); plt.show()\nfig = plt.figure(figsize=(11, 4.2))\naxA = fig.add_subplot(1, 2, 1, projection=\"3d\")\naxA.plot(*cam_centers.T, color=\"#cc3311\", lw=1.8)\naxA.scatter(*cam_centers[0], s=45, c=\"green\", label=\"start\")\naxA.scatter(*cam_centers[-1], s=45, c=\"black\", label=\"end\")\nsub = points[np.random.choice(len(points), min(4000, len(points)), replace=False)]\naxA.scatter(*sub.T, s=0.4, c=\"#bbbbbb\", alpha=0.35)\naxA.set_title(\"Camera trajectory (3D)\"); axA.legend(fontsize=8)\naxB = fig.add_subplot(1, 2, 2)\naxB.plot(cam_centers[:, 0], cam_centers[:, 2], color=\"#cc3311\", lw=1.8)\naxB.scatter(sub[:, 0], sub[:, 2], s=0.4, c=\"#bbbbbb\", alpha=0.35)\naxB.set_xlabel(\"x\"); axB.set_ylabel(\"z\"); axB.set_aspect(\"equal\")\naxB.set_title(\"Top-down (x-z)\")\nplt.tight_layout(); plt.savefig(f\"{OUT}/trajectory.png\", dpi=110); plt.show()\nimport plotly.graph_objects as go\nn_show = min(CFG[\"max_plot_points\"], len(points))\nsel = np.random.choice(len(points), n_show, replace=False)\nP, C = points[sel], (colors[sel] * 255).astype(np.uint8)\ninb = np.all((P >= lo) & (P <= hi), axis=1)\nP, C = P[inb], C[inb]\nfig = go.Figure([\n   go.Scatter3d(x=P[:, 0], y=P[:, 1], z=P[:, 2], mode=\"markers\",\n                marker=dict(size=1.2, color=[f\"rgb({r},{g},{b})\" for r, g, b in C]),\n                name=\"points\", hoverinfo=\"skip\"),\n   go.Scatter3d(x=cam_centers[:, 0], y=cam_centers[:, 1], z=cam_centers[:, 2],\n                mode=\"lines+markers\", line=dict(color=\"red\", width=4),\n                marker=dict(size=2, color=\"red\"), name=\"camera path\"),\n])\nfig.update_layout(height=680, margin=dict(l=0, r=0, t=28, b=0),\n                 title=f\"LingBot-Map reconstruction — {len(P):,} of {len(points):,} points\",\n                 scene=dict(aspectmode=\"data\",\n                            xaxis=dict(visible=False), yaxis=dict(visible=False),\n                            zaxis=dict(visible=False), bgcolor=\"rgb(15,15,20)\"))\nfig.show()\ndef write_ply(path, xyz, rgb01):\n   rgb8 = (np.clip(rgb01, 0, 1) * 255).astype(np.uint8)\n   hdr = (f\"ply\\nformat binary_little_endian 1.0\\nelement vertex {len(xyz)}\\n\"\n          \"property float x\\nproperty float y\\nproperty float z\\n\"\n          \"property uchar red\\nproperty uchar green\\nproperty uchar blue\\n\"\n          \"end_header\\n\")\n   dt = np.dtype([(\"x\", \"<f4\"), (\"y\", \"<f4\"), (\"z\", \"<f4\"),\n                  (\"red\", \"u1\"), (\"green\", \"u1\"), (\"blue\", \"u1\")])\n   arr = np.empty(len(xyz), dtype=dt)\n   arr[\"x\"], arr[\"y\"], arr[\"z\"] = xyz[:, 0], xyz[:, 1], xyz[:, 2]\n   arr[\"red\"], arr[\"green\"], arr[\"blue\"] = rgb8[:, 0], rgb8[:, 1], rgb8[:, 2]\n   with open(path, \"wb\") as f:\n       f.write(hdr.encode()); f.write(arr.tobytes())\nif CFG[\"export_ply\"]:\n   ply = f\"{OUT}/{CFG['scene']}_lingbot.ply\"\n   write_ply(ply, points, colors)\n   print(f\"  PLY -> {ply}  ({os.path.getsize(ply)/2**20:.1f} MB) \"\n         \"— open in MeshLab / CloudCompare / Blender\")\nnp.savez_compressed(f\"{OUT}/predictions.npz\",\n                   extrinsic=extrinsic, intrinsic=intrinsic,\n                   cam_centers=cam_centers, depth=depth.astype(np.float16),\n                   depth_conf=depth_conf.astype(np.float16))\nprint(f\"  NPZ -> {OUT}/predictions.npz (poses + depth, fp16)\")\nif CFG[\"export_glb\"]:\n   sh(\"pip install -q trimesh\", check=False)\n   from lingbot_map.vis import predictions_to_glb\n   wp_full = np.stack([depth_to_world_coords_points(\n       depth[i].squeeze(-1), extrinsic[i], intrinsic[i])[0] for i in range(S)])\n   scene = predictions_to_glb(\n       {\"world_points_from_depth\": wp_full, \"depth_conf\": depth_conf,\n        \"images\": rgb, \"extrinsic\": extrinsic, \"intrinsic\": intrinsic},\n       conf_thres=CFG[\"conf_percentile\"], prediction_mode=\"Predicted Depthmap\")\n   scene.export(f\"{OUT}/{CFG['scene']}.glb\")\n   print(f\"  GLB -> {OUT}/{CFG['scene']}.glb\")\ntry:\n   from google.colab import files\n   print(\"  (run `files.download(path)` in a new cell to pull a file down)\")\nexcept Exception:\n   pass\nif CFG[\"launch_viser\"]:\n   sh(\"pip install -q 'viser>=0.2.23' trimesh\", check=False)\n   import threading\n   from lingbot_map.vis import PointCloudViewer\n   vis_pred = {\"images\": rgb, \"depth\": depth, \"depth_conf\": depth_conf,\n               \"extrinsic\": extrinsic, \"intrinsic\": intrinsic}\n   viewer = PointCloudViewer(pred_dict=vis_pred, port=8080,\n                             vis_threshold=1.5, downsample_factor=10,\n                             point_size=0.00001, use_point_map=False)\n   threading.Thread(target=lambda: viewer.run(background_mode=True), daemon=True).start()\n   time.sleep(3)\n   from google.colab.output import serve_kernel_port_as_window\n   serve_kernel_port_as_window(8080)\n   print(\"  viser opened in a new tab (allow pop-ups)\")\nif CFG[\"run_ablation\"]:\n   print(\"\\n[11b] Ablation on the first 24 frames\")\n   sub_imgs = images[:24]\n   rows = []\n   for label, kw in [(\"cam_iters=4, kf=1\", dict(keyframe_interval=1)),\n                     (\"cam_iters=4, kf=2\", dict(keyframe_interval=2)),\n                     (\"cam_iters=4, kf=4\", dict(keyframe_interval=4))]:\n       model.clean_kv_cache(); torch.cuda.empty_cache()\n       torch.cuda.reset_peak_memory_stats()\n       t = time.time()\n       with torch.no_grad(), torch.amp.autocast(\"cuda\", dtype=DTYPE):\n           p = model.inference_streaming(sub_imgs,\n                                         num_scale_frames=CFG[\"num_scale_frames\"],\n                                         output_device=torch.device(\"cpu\"), **kw)\n       dt = time.time() - t\n       e, _ = decode_poses(p[\"pose_enc\"].float(), (H, W))\n       cc = closed_form_inverse_se3(unbatch(e).cpu().numpy())[:, :3, 3]\n       rows.append((label, 24 / dt, torch.cuda.max_memory_allocated() / 2**30,\n                    float(np.linalg.norm(np.diff(cc, axis=0), axis=1).sum())))\n       del p\n   print(f\"  {'setting':<20}{'FPS':>8}{'peak GB':>10}{'traj len':>11}\")\n   for r in rows:\n       print(f\"  {r[0]:<20}{r[1]:>8.2f}{r[2]:>10.2f}{r[3]:>11.3f}\")\n   print(\"  Higher keyframe_interval = less KV memory and more speed; the \"\n         \"trajectory length drifting away from the kf=1 row is your quality cost.\")\nprint(\"\\n\" + \"=\" * 78)\nprint(f\"DONE. {S} frames -> {len(points):,} points at {S/elapsed:.2f} FPS. \"\n     f\"Artifacts in {OUT}/\")\nprint(\"=\" * 78)\nprint(textwrap.dedent(\"\"\"\n   Where to go next\n   ----------------\n   * More frames is the single biggest quality lever. Raise CFG['max_frames']\n     until you hit OOM, then back off.\n   * >320 frames: the KV cache exceeds the 320-view RoPE training range. Set\n     keyframe_interval (auto-computed here) rather than growing the cache.\n   * >3000 frames: switch CFG['mode'] to 'windowed'. window_size counts KV\n     slots, not frames — with scale_frames=8 and keyframe_interval=k, one\n     window covers 8 + (window_size - 8) * k actual frames.\n   * Outdoor scenes: pip install onnxruntime and use the repo's sky masking\n     (lingbot_map.vis.apply_sky_segmentation) to drop sky points, which\n     otherwise smear into the far field.\n   * FlashInfer (use_sdpa=False) gives paged-KV attention and roughly 20 FPS at\n     518x378 on a proper GPU, but JIT-compiles kernels on first call.\n   * Pose collapse on long runs = state drift. Shorten the run, raise\n     keyframe_interval, or move to windowed mode.\n\"\"\"))\n```\n\nWe visualize the RGB frames, depth maps, confidence masks, confidence distribution, camera trajectory, and reconstructed point cloud. We export the reconstruction as PLY and NPZ files, while optionally generating a GLB scene or launching an interactive Viser viewer inside Colab. We also support an optional ablation experiment that compares keyframe configurations in terms of speed, GPU memory usage, and reconstructed trajectory behavior.\n\nIn conclusion, we established a complete and configurable workflow for converting an ordered image or video sequence into a spatially consistent 3D reconstruction. We used LingBot-Map’s streaming architecture to process frames efficiently while controlling KV-cache growth, GPU memory consumption, keyframe density, and camera-refinement cost. We transformed the model’s pose and depth predictions into camera extrinsics, intrinsics, trajectory coordinates, and confidence-filtered world points, and we verified the geometry before generating visualizations and reusable reconstruction files. We also retained support for windowed inference, interactive Viser inspection, alternative export formats, and ablation experiments, which allows us to extend the notebook from a Colab demonstration into a practical environment for evaluating reconstruction quality, runtime performance, and memory trade-offs across different GPUs and scene lengths.\n\nCheck out the ** Full Code here. **Also, feel free to follow us on\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export", "canonical_source": "https://www.marktechpost.com/2026/07/31/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export/", "published_at": "2026-07-31 20:27:13+00:00", "updated_at": "2026-07-31 20:42:17.622561+00:00", "lang": "en", "topics": ["computer-vision", "ai-infrastructure", "ai-tools"], "entities": ["LingBot-Map", "GitHub", "GCTStream"], "alternates": {"html": "https://wpnews.pro/news/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export", "markdown": "https://wpnews.pro/news/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export.md", "text": "https://wpnews.pro/news/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export.txt", "jsonld": "https://wpnews.pro/news/lingbot-map-tutorial-gpu-aware-inference-and-point-cloud-export.jsonld"}}