# Meta SAM 3: Text Prompt Segmentation for Images and Video, Plus SAM 3.1

> Source: <https://explainx.ai/blog/meta-sam-3-promptable-concept-segmentation-sam-3-1-hugging-face-guide-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-09 00:00:00+00:00

**Short answer:** [Meta SAM 3](https://ai.meta.com/research/sam3/?ref=explainx) is an open-weight vision model that segments **every instance** of a concept you describe with a short text phrase, an exemplar box, or classic click prompts, in **still images and video**. Weights sit on [Hugging Face `facebook/sam3`](https://huggingface.co/facebook/sam3?ref=explainx) with official code in [`facebookresearch/sam3`](https://github.com/facebookresearch/sam3?ref=explainx). A **March 27, 2026** update, **SAM 3.1**, adds **object multiplexing** for faster multi-object video on one GPU. If you already use SAM 2 for masks, SAM 3 is the upgrade path when you need **open-vocabulary "find all yellow school buses"** style queries, not just point-and-track.

SAM 3 sits beside Meta's **3D** line, which we cover in [GenIA on SAM 3D Objects](https://explainx.ai/blog/meta-genia-generative-reconstruction-test-time-input-alignment-sam-3d-objects-2026). SAM 3 is the **2D and video masking** foundation that powers consumer edits and downstream research stacks.

## TL;DR

| Question | Answer | 
|---|---|
| Core task | **Promptable Concept Segmentation (PCS)** — text or exemplar in, all matching instance masks out | 
| Modalities | Images and video, with tracking identities in video | 
| Open weights | Yes on Hugging Face ( `facebook/sam3` ) after license accept | 
| SAM 3.1 | Drop-in checkpoints; multiplex up to **16 objects** per video forward pass | 
| Meta-reported video speed | About **32 FPS** on one**H100** for medium object counts (was ~16 FPS) | 
| Benchmark hook | **SA-Co** with ~**270K** concept labels; Meta cites ~**75–80%** of human performance | 
| Consumer surfaces | Coming to **Instagram Edits** and**Vibes** on the Meta AI app, per Meta pages | 
| License | Gated on Hugging Face — read Meta terms before shipping a product | 

## What problem does SAM 3 solve?

Segmentation models used to split into two camps. **Interactive** models like SAM 1 and SAM 2 excel when a human clicks a point or box and wants a tight mask, including through time in video. **Detector-style** models take a fixed label set and return boxes or masks per class.

Meta's SAM 3 paper frames a middle task: **Promptable Concept Segmentation**. You give a **short noun phrase** (`"yellow school bus"`), an **exemplar crop**, or **SAM-style clicks**, and the model should return **segmentation masks and persistent IDs** for **every object instance** that matches the concept, including in cluttered scenes.

That matters for builders because many pipelines still chain **a detector, a tracker, and a hand-tuned class list**. SAM 3 tries to collapse detection plus instance masks plus tracking behind one prompt interface. Meta also ties it to **creative tools** — pixelate every face, highlight every player, trail every ball — which is why Edits and Vibes show up in the launch story on [Meta's SAM 3 page](https://ai.meta.com/research/sam3/?ref=explainx).

For comparison with how other labs use segmentation inside agents, see [OpenVLHarness for multimodal agents](https://explainx.ai/blog/openvlharness-uclanlp-visual-agent-harness-multimodal-memory-guide-2026) and [CodeCapBench, which used SAM 3.1 for spatio-temporal IoU](https://explainx.ai/blog/codecap-bench-code-as-video-caption-gpt-6-astra-codex-73-percent-opus-5-2026).

Vision object detection lens over a pear, illustrating Meta SAM 3 open-vocabulary segmentation targets

## How SAM 3 is built (without the math wall)

Meta's [publication page](https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/?ref=explainx) describes SAM 3 as one backbone shared by an **image-level detector** and a **memory-based video tracker**. A **presence head** decouples **recognition** from **localization**, which helps when many similar objects appear.

Training scale is part of the story. Meta says a **data engine** produced on the order of **four million** unique **concept labels**, including **hard negatives**, across images and video. They also shipped the **SA-Co (Segment Anything with Concepts)** benchmark so outsiders can test open-vocabulary prompts instead of only COCO-style classes.

Meta's announcement blog ([November 19, 2025](https://about.fb.com/news/2025/11/new-sam-models-detect-objects-create-3d-reconstructions/?ref=explainx)) claimed roughly a **2× gain** over prior systems on PCS in both images and video, while keeping SAM 2-style click refinement. Treat that multiplier as **vendor-reported**, not independent.

## SAM 3.1: why video builders should care

Video is where naive multi-object tracking burns GPU hours. Meta's [March 27, 2026 SAM 3.1 note](https://ai.meta.com/blog/segment-anything-model-3/?ref=explainx) says SAM 3 previously ran **one forward pass per tracked object**. **SAM 3.1** introduces **object multiplexing**: up to **16 objects** processed together with **shared memory**, which Meta positions as a **drop-in replacement** for SAM 3 video checkpoints on [Hugging Face `facebook/sam3.1`](https://huggingface.co/facebook/sam3.1?ref=explainx) (` RELEASE_SAM3p1.md` in the GitHub repo).

Meta's headline number: on a single **H100**, throughput for medium object counts moves from about **16 FPS to about 32 FPS**. That is still not phone-real-time for every scene, but it is the difference between **batch post-production** and **near-real-time preview** for creator tools.

If you evaluate video agents, pair this with our [Viggle Remake open video-to-video agent](https://explainx.ai/blog/viggle-remake-open-weights-agent-mcp-video-to-video-2026) coverage — different task, same question of **whether masks are cheap enough to run inside an agent loop**.

## How to run SAM 3 locally

Meta documents two paths: the **`sam3` Python package** in GitHub and **`transformers`** on Hugging Face.

**Transformers image example** (from the model card on Hugging Face):

``` python
from transformers import Sam3Processor, Sam3Model
import torch
from PIL import Image
import requests

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Sam3Model.from_pretrained("facebook/sam3").to(device)
processor = Sam3Processor.from_pretrained("facebook/sam3")

image_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")

inputs = processor(images=image, text="ear", return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]
```

**GitHub image path** uses `build_sam3_image_model()` and `Sam3Processor` from the `sam3` package; **video** uses `build_sam3_video_predictor()` with session APIs for `start_session` and `add_prompt`. Read `INSTALL.md` in the repo before you pin versions — SAM 3.1 expects the **latest** `facebookresearch/sam3` commit.

**Hardware reality:** Meta's speed claims assume **datacenter GPUs**. On a laptop CPU you can prototype, but PCS on long video is not pleasant without CUDA. For local vision stacks that avoid cloud APIs, compare [llama.cpp-style local inference guides](https://explainx.ai/blog/llama-modchip-64gb-cpu-deepseek-fractal-gguf-inference-guide-2026) — different modality, same budgeting question.

### License and gating

The Hugging Face model card requires you to **agree to share information with Meta** and accept their **license** before weights download. That is stricter than a plain Apache repo. If you need **fully permissive** weights for a commercial robot stack, read the text before you bake SAM 3 into a shipping pipeline. Research and internal demos are the easy cases.

A ball tracked across empty video frames, illustrating Meta SAM 3 multi-object video segmentation

## Where SAM 3 fits in Meta's product map

Meta is explicit that SAM 3 is not only a paper. The [SAM 3 research landing page](https://ai.meta.com/research/sam3/?ref=explainx) lists:

- **Segment Anything Playground** for no-code trials on images and video templates.
- **Instagram Edits** for creator effects on selected people or objects.
- **Vibes** on the**Meta AI app** for similar generative edits.
- Partnerships such as **Conservation X Labs** for wildlife monitoring datasets.

For feed-ranking context on how Meta thinks about **user-controlled AI**, see [Facebook Your Algorithm](https://explainx.ai/blog/facebook-your-algorithm-ai-feed-controls-llm-ranking-instagram-threads-2026). SAM 3 is the computer-vision half; ranking is the distribution half.

Downstream researchers already borrow SAM masks. Our [CodeCapBench](https://explainx.ai/blog/codecap-bench-code-as-video-caption-gpt-6-astra-codex-73-percent-opus-5-2026) post notes **object-level spatio-temporal IoU** with **SAM 3.1** tracking against reference annotations — a pattern you can copy when you build **video eval harnesses**.

## Limitations and honest bench skepticism

**Open vocabulary is not magic.** Meta's **75–80% of human** figure on **SA-Co** is useful for comparing releases, but your warehouse, medical, or factory footage can sit far from Meta's training mixture. Plan a **small gold set** of phrases your product actually uses.

**Concept collisions.** Short phrases like `"truck"` or `"person in red"` remain ambiguous. SAM 3 supports **exemplar boxes** and **follow-up clicks** precisely because first-pass PCS can miss rare instances or bleed into look-alikes.

**Video ID switches.** Faster multiplexing reduces cost; it does not delete classic tracker failure modes when objects occlude each other. For robotics stacks that need metric depth, pair masking with projects like [NVIDIA MetricTrail](https://explainx.ai/blog/nvidia-metrictrail-camera-motion-metric-depth-long-video-dpvo-moge-2-dinov3-kitti-ate-2026) rather than masks alone.

**3D is a sibling, not a substitute.** Need a mesh from one photo? That is [SAM 3D and GenIA](https://explainx.ai/blog/meta-genia-generative-reconstruction-test-time-input-alignment-sam-3d-objects-2026), not SAM 3. Many pipelines will use **SAM 3 masks to crop** before a 3D model runs.

## What people ask after reading the launch

**Does SAM 3 replace YOLO or Grounding DINO?** Not automatically. Detectors still win when you need **fixed COCO classes** at the lowest latency on edge hardware. SAM 3 wins when the class list is **open-ended** and you want **masks, not boxes**, for every instance of a phrase. Many teams will keep a detector for proposals and use SAM 3 only on hard language-conditioned edits.

**Do I need the Playground?** No for engineering, yes for product discovery. Meta's [Segment Anything Playground](https://segment-anything.com/?ref=explainx) is the fastest way to see whether your phrase triggers false positives before you wire APIs. Templates include privacy-style edits such as **pixelating faces or license plates**, which mirror how Meta describes Instagram Edits.

**What about fine-tuning?** Meta announced a **Roboflow** partnership so teams can **annotate and fine-tune SAM 3** on domain data, per the [November 2025 launch post](https://about.fb.com/news/2025/11/new-sam-models-detect-objects-create-3d-reconstructions/?ref=explainx). Fine-tuning is the realistic path if SA-Co averages hide failure on your SKUs.

**Will SAM 3 run on Apple Silicon?** The Hugging Face card targets **PyTorch with CUDA** for serious video. MPS may run short image demos, but Meta's FPS claims assume **H100-class** hardware. For on-device segmentation trade-offs, watch Apple's separate research line in [tri-modal masked diffusion](https://explainx.ai/blog/apple-tri-modal-masked-diffusion-text-image-audio-from-scratch-neurips-paper-2026), which is generative, not PCS.

**How does this relate to agent tooling?** Agents that call vision tools still need **guardrails on what they blur, crop, or upload**. If your agent pipeline auto-masks people in user uploads, log the **prompt, mask area, and frame hash** alongside model version so compliance can reproduce decisions. That pattern mirrors how we discuss audit trails in [agent sandbox comparisons](https://explainx.ai/blog/agent-sandboxes-coding-agents-compared-e2b-modal-daytona-guide-2026).

## SAM 1, SAM 2, and SAM 3 at a glance

| Generation | Best prompt types | Output style | Video | 
|---|---|---|---|
| SAM 1 | Points and boxes | One mask per prompt | Limited | 
| SAM 2 | Points, boxes, masks | Masks plus tracking | Strong single-object track | 
| SAM 3 | Text, exemplars, clicks | **All instances** of a concept | PCS plus multiplexed track | 
| SAM 3.1 | Same as SAM 3 | Same quality target | **Faster** multi-object passes | 

The row that matters for builders is the last: SAM 3 is the first in the family where **language is a first-class prompt**, not an afterthought wrapped around a detector.

## What to try this week

1. Accept the license on Hugging Face and run the **`"ear"`** COCO val image example to verify your GPU stack.
2. Pick **ten phrases** your product cares about and log**false merges** (two objects, one mask) and**false splits** .
3. If you track video, pull **SAM 3.1** checkpoints and compare**FPS and ID stability** against SAM 3 on the same clip.
4. Wire masks into an agent only with **human-readable overlays** stored alongside prompts — useful for debugging PCS in production.

## Related reading

*Specifications, checkpoint names, and FPS claims are accurate as of November 9, 2026. Check `facebookresearch/sam3` and the Hugging Face model card before you pin versions in production.*