{"slug": "ant-group-open-sources-lingbot-vision-for-dense-spatial-perception", "title": "Ant Group open-sources LingBot-Vision for dense spatial perception", "summary": "Ant Group open-sourced LingBot-Vision, a family of self-supervised Vision Transformer backbones for dense spatial perception, with models ranging from ViT-S/16 to a 1.1B-parameter ViT-g/16, released on Hugging Face and ModelScope. The flagship model uses masked boundary modeling to learn boundaries, shapes, and semantic regions, and serves as the visual encoder for LingBot-Depth 2.0, which improves depth estimation on mirror and glass scenes by scaling training data from 3M to 150M samples.", "body_md": "**LingBot-Vision** is a family of self-supervised Vision Transformer backbones for dense spatial perception, from ViT-S/16 up to a 1.1B-parameter ViT-g/16. The flagship model is pretrained with **masked boundary modeling** — a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations.\n\n**Boundary-centric masked modeling.** Each row shows the input image, the PCA projection of frozen patch tokens, teacher-discovered boundary tokens, and cosine-similarity maps from selected boundary-token queries. The features capture semantic grouping and geometric structure at the same time.\n\nLingBot-Vision learns boundaries, shapes, and semantic regions all together, making it a drop-in visual encoder for dense downstream tasks:\n\n- 🎨\n**Dense feature visualization**— PCA maps of frozen patch tokens reveal coherent object regions and crisp boundaries - 📏\n**Depth estimation**— frozen patch tokens expose spatial structure to lightweight dense readouts - 🧩\n**Semantic segmentation**— boundary-faithful features align region transitions with object contours - 🎬\n**Video object segmentation**— training-free token matching and label propagation with frozen features - 🤖\n**Depth completion**— LingBot-Vision is the visual encoder initialization for LingBot-Depth 2.0 (see below)\n\nBy simply replacing the encoder with LingBot-Vision at the ViT-L/16 and ViT-g/16 scales, and scaling the curated RGB-D training corpus from 3M to 150M samples, LingBot-Depth 2.0 achieves substantial performance gains over the previous and other system, as detailed in the technical report.\n\n**LingBot-Depth 2.0 on mirror and glass scenes.** Each group shows input RGB, raw sensor depth, refined depth, and refined point clouds. Raw depth is missing on difficult surfaces such as window panes, glass balustrades, and reflective floors. LingBot-Depth 2.0 completes these regions as stable, contiguous surfaces across frames.\n\nWe train a ViT-g/16 teacher with roughly 1.1B parameters and distill ViT-L, ViT-B, and ViT-S backbones from it for inference and downstream use. Full training and evaluation details are covered in the technical report.\n\nAll released weights are **backbone-only** `.pt`\n\ncheckpoints, stored as `model.pt`\n\nin each model repository (see the full [Hugging Face collection](https://huggingface.co/collections/robbyant/lingbot-vision)):\n\n| Model | Backbone | Embed dim | Hugging Face Weights | ModelScope Weights |\n|---|---|---|---|---|\nLingBot-Vision-Gianthighest-quality dense features |\nViT-g/16 · SwiGLU · fp32 RoPE · 4 register tokens | 1536 |\n|\n\n[vit-giant](https://www.modelscope.cn/models/Robbyant/lingbot-vision-vit-giant)**LingBot-Vision-Large**⭐recommended: strong features, practical inference[vit-large](https://huggingface.co/robbyant/lingbot-vision-vit-large)[vit-large](https://www.modelscope.cn/models/Robbyant/lingbot-vision-vit-large)**LingBot-Vision-Base** balanced inference cost[vit-base](https://huggingface.co/robbyant/lingbot-vision-vit-base)[vit-base](https://www.modelscope.cn/models/Robbyant/lingbot-vision-vit-base)**LingBot-Vision-Small** lightweight demos and downstream use[vit-small](https://huggingface.co/robbyant/lingbot-vision-vit-small)[vit-small](https://www.modelscope.cn/models/Robbyant/lingbot-vision-vit-small)Config files are packaged under `lingbot_vision/configs/`\n\nand selected automatically by `load_pretrained_backbone`\n\n.\n\n**Requirements**: Python ≥ 3.10 · PyTorch ≥ 2.0 · CUDA-capable GPU (recommended for large-model inference)\n\n**1. Clone the repository**\n\n```\ngit clone https://github.com/robbyant/lingbot-vision.git\ncd lingbot-vision\n```\n\n**2. Create a conda environment**\n\n```\nconda create -n lingbot-vision python=3.10 -y\nconda activate lingbot-vision\n```\n\n**3. Install lingbot-vision**\n\n```\npython -m pip install -r requirements.txt\npython -m pip install -e .\n```\n\nThe model is automatically downloaded from Hugging Face on first use. This example uses the small model for a lightweight smoke run; `large`\n\nis the default variant, and `giant`\n\nis available as the largest backbone.\n\n``` python\nimport torch\n\nfrom lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ndtype = torch.bfloat16 if device == \"cuda\" else torch.float32\n\n# Downloads model.pt from robbyant/lingbot-vision-vit-small.\nbackbone, embed_dim = load_pretrained_backbone(\n    variant=\"small\",\n    device=device,\n    dtype=dtype,\n)\n\nimg_norm, _, _ = load_image(\n    \"examples/example.png\",\n    size=512,\n    patch_size=backbone.patch_size,\n    mode=\"square\",\n)\npatch_tokens, patch_grid = extract_patch_tokens(backbone, img_norm, device, dtype)\n\nprint(patch_tokens.shape, patch_grid, embed_dim)\n# torch.Size([1, 1024, 384]) (32, 32) 384\n```\n\n`patch_tokens`\n\nhas shape `[B, H * W, C]`\n\n, where `H`\n\nand `W`\n\nare the patch-grid dimensions. `variant`\n\ncan be `giant`\n\n, `large`\n\n, `base`\n\n, or `small`\n\n; if omitted, it defaults to `large`\n\n. You can also pass a local directory or an explicit Hugging Face model repo to `load_pretrained_backbone`\n\n.\n\nDownload a backbone checkpoint from Hugging Face (or ModelScope, once available), then run:\n\n```\n./scripts/run_pca_demo.sh \\\n  --config-file lingbot_vision/configs/lbot_vision_vitl.yaml \\\n  --ckpt /path/to/model.pt \\\n  --input examples/example.png \\\n  --out outputs/pca_demo \\\n  --size 512 \\\n  --mode square \\\n  --dtype bf16\n```\n\nImages are loaded as RGB, resized according to `--size`\n\nand `--mode`\n\n, aligned to the model patch size, and normalized with ImageNet statistics. The demo maps the top three PCA components of the patch tokens to RGB and writes both PCA-only and input/PCA panel visualizations to the output directory. Use `--dtype fp32 --device cpu`\n\nfor CPU-only inference.\n\n**All demo options**\n\n| Parameter | Description |\n|---|---|\n`--config-file` |\nModel config file under `lingbot_vision/configs/` . |\n`--ckpt` |\nLocal path to a pure backbone `.pt` checkpoint. |\n`--input` |\nImage file or directory of images. |\n`--out` |\nOutput directory for PCA visualizations. |\n`--size` |\nTarget input size. For ViT-g/16, 512 gives a 32 x 32 patch grid. |\n`--mode` |\n`square` resizes to `size` x `size` (does not preserve aspect ratio); `shortest` resizes the shortest side to `size` , then center-crops a `size` x `size` square. |\n`--dtype` |\n`bf16` , `fp16` , or `fp32` . |\n`--device` |\nPyTorch device, for example `cuda` or `cpu` . |\n\nReleased checkpoints are `.pt`\n\nfiles containing backbone weights only — no optimizer states, projection heads, or training-time boundary heads. The loader accepts a raw state dict or a dictionary with a `backbone`\n\nentry:\n\n```\nstate_dict\n{\"backbone\": state_dict}\n```\n\nIf checkpoint keys are prefixed with `backbone.`\n\n, the loader strips the prefix automatically.\n\n```\n@article{lingbot-vision2026,\n  title={Vision Pretraining for Dense Spatial Perception},\n  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},\n  journal={arXiv preprint arXiv:2607.05247},\n  year={2026}\n}\n```\n\nThis project is released under the Apache License 2.0. See [LICENSE](/Robbyant/lingbot-vision/blob/main/LICENSE) for details.\n\nLingBot-Vision is part of the LingBot spatial perception effort. We thank [DINOv2](https://github.com/facebookresearch/dinov2) and [DINOv3](https://github.com/facebookresearch/dinov3) for their contributions to self-supervised learning.\n\nFor questions, discussions, or collaborations:\n\n**Issues**: Open an[issue](https://github.com/robbyant/lingbot-vision/issues)on GitHub** Email**: Contact[Zelin Fu](https://github.com/TakuLingFu)([fuzelin.fzl@antgroup.com](mailto:fuzelin.fzl@antgroup.com)) or[Nan Xue](https://xuenan.net)([xuenan@ieee.org](mailto:xuenan@ieee.org))", "url": "https://wpnews.pro/news/ant-group-open-sources-lingbot-vision-for-dense-spatial-perception", "canonical_source": "https://github.com/Robbyant/lingbot-vision", "published_at": "2026-09-03 23:41:18+00:00", "updated_at": "2026-09-03 23:53:01.341495+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "ai-research", "ai-products", "ai-infrastructure"], "entities": ["Ant Group", "LingBot-Vision", "LingBot-Depth 2.0", "Hugging Face", "ModelScope", "ViT-g/16"], "alternates": {"html": "https://wpnews.pro/news/ant-group-open-sources-lingbot-vision-for-dense-spatial-perception", "markdown": "https://wpnews.pro/news/ant-group-open-sources-lingbot-vision-for-dense-spatial-perception.md", "text": "https://wpnews.pro/news/ant-group-open-sources-lingbot-vision-for-dense-spatial-perception.txt", "jsonld": "https://wpnews.pro/news/ant-group-open-sources-lingbot-vision-for-dense-spatial-perception.jsonld"}}