{"slug": "meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1", "title": "Meta SAM 3: Text Prompt Segmentation for Images and Video, Plus SAM 3.1", "summary": "Meta released SAM 3, an open-weight vision model that performs Promptable Concept Segmentation — returning masks and persistent IDs for every instance matching a short text phrase, exemplar box, or click prompt in both images and video — with weights on Hugging Face at facebook/sam3 and code in facebookresearch/sam3. A March 27, 2026 update, SAM 3.1, adds object multiplexing that processes up to 16 objects per video forward pass and raises Meta-reported video speed to about 32 FPS on a single H100, up from roughly 16 FPS. Meta trained the model on a data engine producing on the order of four million unique concept labels and introduced the SA-Co benchmark with about 270K concept labels, citing roughly 75–80% of human performance; Meta says the model is coming to Instagram Edits and Vibes on the Meta AI app.", "body_md": "**Short answer:** [Meta SAM 3](https://ai.meta.com/research/sam3/?ref=explainx) is an open-weight vision model that segments **every instance** of a concept you describe with a short text phrase, an exemplar box, or classic click prompts, in **still images and video**. Weights sit on [Hugging Face `facebook/sam3`](https://huggingface.co/facebook/sam3?ref=explainx) with official code in [`facebookresearch/sam3`](https://github.com/facebookresearch/sam3?ref=explainx). A **March 27, 2026** update, **SAM 3.1**, adds **object multiplexing** for faster multi-object video on one GPU. If you already use SAM 2 for masks, SAM 3 is the upgrade path when you need **open-vocabulary \"find all yellow school buses\"** style queries, not just point-and-track.\n\nSAM 3 sits beside Meta's **3D** line, which we cover in [GenIA on SAM 3D Objects](https://explainx.ai/blog/meta-genia-generative-reconstruction-test-time-input-alignment-sam-3d-objects-2026). SAM 3 is the **2D and video masking** foundation that powers consumer edits and downstream research stacks.\n\n## TL;DR\n\n| Question | Answer | \n|---|---|\n| Core task | **Promptable Concept Segmentation (PCS)** — text or exemplar in, all matching instance masks out | \n| Modalities | Images and video, with tracking identities in video | \n| Open weights | Yes on Hugging Face ( `facebook/sam3` ) after license accept | \n| SAM 3.1 | Drop-in checkpoints; multiplex up to **16 objects** per video forward pass | \n| Meta-reported video speed | About **32 FPS** on one**H100** for medium object counts (was ~16 FPS) | \n| Benchmark hook | **SA-Co** with ~**270K** concept labels; Meta cites ~**75–80%** of human performance | \n| Consumer surfaces | Coming to **Instagram Edits** and**Vibes** on the Meta AI app, per Meta pages | \n| License | Gated on Hugging Face — read Meta terms before shipping a product | \n\n## What problem does SAM 3 solve?\n\nSegmentation models used to split into two camps. **Interactive** models like SAM 1 and SAM 2 excel when a human clicks a point or box and wants a tight mask, including through time in video. **Detector-style** models take a fixed label set and return boxes or masks per class.\n\nMeta's SAM 3 paper frames a middle task: **Promptable Concept Segmentation**. You give a **short noun phrase** (`\"yellow school bus\"`), an **exemplar crop**, or **SAM-style clicks**, and the model should return **segmentation masks and persistent IDs** for **every object instance** that matches the concept, including in cluttered scenes.\n\nThat matters for builders because many pipelines still chain **a detector, a tracker, and a hand-tuned class list**. SAM 3 tries to collapse detection plus instance masks plus tracking behind one prompt interface. Meta also ties it to **creative tools** — pixelate every face, highlight every player, trail every ball — which is why Edits and Vibes show up in the launch story on [Meta's SAM 3 page](https://ai.meta.com/research/sam3/?ref=explainx).\n\nFor comparison with how other labs use segmentation inside agents, see [OpenVLHarness for multimodal agents](https://explainx.ai/blog/openvlharness-uclanlp-visual-agent-harness-multimodal-memory-guide-2026) and [CodeCapBench, which used SAM 3.1 for spatio-temporal IoU](https://explainx.ai/blog/codecap-bench-code-as-video-caption-gpt-6-astra-codex-73-percent-opus-5-2026).\n\nVision object detection lens over a pear, illustrating Meta SAM 3 open-vocabulary segmentation targets\n\n## How SAM 3 is built (without the math wall)\n\nMeta's [publication page](https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/?ref=explainx) describes SAM 3 as one backbone shared by an **image-level detector** and a **memory-based video tracker**. A **presence head** decouples **recognition** from **localization**, which helps when many similar objects appear.\n\nTraining scale is part of the story. Meta says a **data engine** produced on the order of **four million** unique **concept labels**, including **hard negatives**, across images and video. They also shipped the **SA-Co (Segment Anything with Concepts)** benchmark so outsiders can test open-vocabulary prompts instead of only COCO-style classes.\n\nMeta's announcement blog ([November 19, 2025](https://about.fb.com/news/2025/11/new-sam-models-detect-objects-create-3d-reconstructions/?ref=explainx)) claimed roughly a **2× gain** over prior systems on PCS in both images and video, while keeping SAM 2-style click refinement. Treat that multiplier as **vendor-reported**, not independent.\n\n## SAM 3.1: why video builders should care\n\nVideo is where naive multi-object tracking burns GPU hours. Meta's [March 27, 2026 SAM 3.1 note](https://ai.meta.com/blog/segment-anything-model-3/?ref=explainx) says SAM 3 previously ran **one forward pass per tracked object**. **SAM 3.1** introduces **object multiplexing**: up to **16 objects** processed together with **shared memory**, which Meta positions as a **drop-in replacement** for SAM 3 video checkpoints on [Hugging Face `facebook/sam3.1`](https://huggingface.co/facebook/sam3.1?ref=explainx) (` RELEASE_SAM3p1.md` in the GitHub repo).\n\nMeta's headline number: on a single **H100**, throughput for medium object counts moves from about **16 FPS to about 32 FPS**. That is still not phone-real-time for every scene, but it is the difference between **batch post-production** and **near-real-time preview** for creator tools.\n\nIf you evaluate video agents, pair this with our [Viggle Remake open video-to-video agent](https://explainx.ai/blog/viggle-remake-open-weights-agent-mcp-video-to-video-2026) coverage — different task, same question of **whether masks are cheap enough to run inside an agent loop**.\n\n## How to run SAM 3 locally\n\nMeta documents two paths: the **`sam3` Python package** in GitHub and **`transformers`** on Hugging Face.\n\n**Transformers image example** (from the model card on Hugging Face):\n\n``` python\nfrom transformers import Sam3Processor, Sam3Model\nimport torch\nfrom PIL import Image\nimport requests\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nmodel = Sam3Model.from_pretrained(\"facebook/sam3\").to(device)\nprocessor = Sam3Processor.from_pretrained(\"facebook/sam3\")\n\nimage_url = \"http://images.cocodataset.org/val2017/000000077595.jpg\"\nimage = Image.open(requests.get(image_url, stream=True).raw).convert(\"RGB\")\n\ninputs = processor(images=image, text=\"ear\", return_tensors=\"pt\").to(device)\n\nwith torch.no_grad():\n    outputs = model(**inputs)\n\nresults = processor.post_process_instance_segmentation(\n    outputs,\n    threshold=0.5,\n    mask_threshold=0.5,\n    target_sizes=inputs.get(\"original_sizes\").tolist()\n)[0]\n```\n\n**GitHub image path** uses `build_sam3_image_model()` and `Sam3Processor` from the `sam3` package; **video** uses `build_sam3_video_predictor()` with session APIs for `start_session` and `add_prompt`. Read `INSTALL.md` in the repo before you pin versions — SAM 3.1 expects the **latest** `facebookresearch/sam3` commit.\n\n**Hardware reality:** Meta's speed claims assume **datacenter GPUs**. On a laptop CPU you can prototype, but PCS on long video is not pleasant without CUDA. For local vision stacks that avoid cloud APIs, compare [llama.cpp-style local inference guides](https://explainx.ai/blog/llama-modchip-64gb-cpu-deepseek-fractal-gguf-inference-guide-2026) — different modality, same budgeting question.\n\n### License and gating\n\nThe Hugging Face model card requires you to **agree to share information with Meta** and accept their **license** before weights download. That is stricter than a plain Apache repo. If you need **fully permissive** weights for a commercial robot stack, read the text before you bake SAM 3 into a shipping pipeline. Research and internal demos are the easy cases.\n\nA ball tracked across empty video frames, illustrating Meta SAM 3 multi-object video segmentation\n\n## Where SAM 3 fits in Meta's product map\n\nMeta is explicit that SAM 3 is not only a paper. The [SAM 3 research landing page](https://ai.meta.com/research/sam3/?ref=explainx) lists:\n\n- **Segment Anything Playground** for no-code trials on images and video templates.\n- **Instagram Edits** for creator effects on selected people or objects.\n- **Vibes** on the**Meta AI app** for similar generative edits.\n- Partnerships such as **Conservation X Labs** for wildlife monitoring datasets.\n\nFor feed-ranking context on how Meta thinks about **user-controlled AI**, see [Facebook Your Algorithm](https://explainx.ai/blog/facebook-your-algorithm-ai-feed-controls-llm-ranking-instagram-threads-2026). SAM 3 is the computer-vision half; ranking is the distribution half.\n\nDownstream researchers already borrow SAM masks. Our [CodeCapBench](https://explainx.ai/blog/codecap-bench-code-as-video-caption-gpt-6-astra-codex-73-percent-opus-5-2026) post notes **object-level spatio-temporal IoU** with **SAM 3.1** tracking against reference annotations — a pattern you can copy when you build **video eval harnesses**.\n\n## Limitations and honest bench skepticism\n\n**Open vocabulary is not magic.** Meta's **75–80% of human** figure on **SA-Co** is useful for comparing releases, but your warehouse, medical, or factory footage can sit far from Meta's training mixture. Plan a **small gold set** of phrases your product actually uses.\n\n**Concept collisions.** Short phrases like `\"truck\"` or `\"person in red\"` remain ambiguous. SAM 3 supports **exemplar boxes** and **follow-up clicks** precisely because first-pass PCS can miss rare instances or bleed into look-alikes.\n\n**Video ID switches.** Faster multiplexing reduces cost; it does not delete classic tracker failure modes when objects occlude each other. For robotics stacks that need metric depth, pair masking with projects like [NVIDIA MetricTrail](https://explainx.ai/blog/nvidia-metrictrail-camera-motion-metric-depth-long-video-dpvo-moge-2-dinov3-kitti-ate-2026) rather than masks alone.\n\n**3D is a sibling, not a substitute.** Need a mesh from one photo? That is [SAM 3D and GenIA](https://explainx.ai/blog/meta-genia-generative-reconstruction-test-time-input-alignment-sam-3d-objects-2026), not SAM 3. Many pipelines will use **SAM 3 masks to crop** before a 3D model runs.\n\n## What people ask after reading the launch\n\n**Does SAM 3 replace YOLO or Grounding DINO?** Not automatically. Detectors still win when you need **fixed COCO classes** at the lowest latency on edge hardware. SAM 3 wins when the class list is **open-ended** and you want **masks, not boxes**, for every instance of a phrase. Many teams will keep a detector for proposals and use SAM 3 only on hard language-conditioned edits.\n\n**Do I need the Playground?** No for engineering, yes for product discovery. Meta's [Segment Anything Playground](https://segment-anything.com/?ref=explainx) is the fastest way to see whether your phrase triggers false positives before you wire APIs. Templates include privacy-style edits such as **pixelating faces or license plates**, which mirror how Meta describes Instagram Edits.\n\n**What about fine-tuning?** Meta announced a **Roboflow** partnership so teams can **annotate and fine-tune SAM 3** on domain data, per the [November 2025 launch post](https://about.fb.com/news/2025/11/new-sam-models-detect-objects-create-3d-reconstructions/?ref=explainx). Fine-tuning is the realistic path if SA-Co averages hide failure on your SKUs.\n\n**Will SAM 3 run on Apple Silicon?** The Hugging Face card targets **PyTorch with CUDA** for serious video. MPS may run short image demos, but Meta's FPS claims assume **H100-class** hardware. For on-device segmentation trade-offs, watch Apple's separate research line in [tri-modal masked diffusion](https://explainx.ai/blog/apple-tri-modal-masked-diffusion-text-image-audio-from-scratch-neurips-paper-2026), which is generative, not PCS.\n\n**How does this relate to agent tooling?** Agents that call vision tools still need **guardrails on what they blur, crop, or upload**. If your agent pipeline auto-masks people in user uploads, log the **prompt, mask area, and frame hash** alongside model version so compliance can reproduce decisions. That pattern mirrors how we discuss audit trails in [agent sandbox comparisons](https://explainx.ai/blog/agent-sandboxes-coding-agents-compared-e2b-modal-daytona-guide-2026).\n\n## SAM 1, SAM 2, and SAM 3 at a glance\n\n| Generation | Best prompt types | Output style | Video | \n|---|---|---|---|\n| SAM 1 | Points and boxes | One mask per prompt | Limited | \n| SAM 2 | Points, boxes, masks | Masks plus tracking | Strong single-object track | \n| SAM 3 | Text, exemplars, clicks | **All instances** of a concept | PCS plus multiplexed track | \n| SAM 3.1 | Same as SAM 3 | Same quality target | **Faster** multi-object passes | \n\nThe row that matters for builders is the last: SAM 3 is the first in the family where **language is a first-class prompt**, not an afterthought wrapped around a detector.\n\n## What to try this week\n\n1. Accept the license on Hugging Face and run the **`\"ear\"`** COCO val image example to verify your GPU stack.\n2. Pick **ten phrases** your product cares about and log**false merges** (two objects, one mask) and**false splits** .\n3. If you track video, pull **SAM 3.1** checkpoints and compare**FPS and ID stability** against SAM 3 on the same clip.\n4. Wire masks into an agent only with **human-readable overlays** stored alongside prompts — useful for debugging PCS in production.\n\n## Related reading\n\n*Specifications, checkpoint names, and FPS claims are accurate as of November 9, 2026. Check `facebookresearch/sam3` and the Hugging Face model card before you pin versions in production.*", "url": "https://wpnews.pro/news/meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1", "canonical_source": "https://explainx.ai/blog/meta-sam-3-promptable-concept-segmentation-sam-3-1-hugging-face-guide-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-09 00:00:00+00:00", "updated_at": "2026-11-09 02:00:49.973569+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "machine-learning", "ai-research"], "entities": ["Meta", "SAM 3", "SAM 3.1", "Hugging Face", "SA-Co", "Instagram Edits", "Vibes", "Meta AI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1", "markdown": "https://wpnews.pro/news/meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1.md", "text": "https://wpnews.pro/news/meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1.txt", "jsonld": "https://wpnews.pro/news/meta-sam-3-text-prompt-segmentation-for-images-and-video-plus-sam-3-1.jsonld"}}