How to Use the Segment Anything Model (SAM)

Meta AI's Segment Anything Model (SAM) is a promptable image segmentation model trained on over 1 billion masks across 11 million licensed, privacy-respecting images, and its zero-shot performance is often competitive with, or better than, fully supervised models.

In this tutorial, I will show you how to use SAM three ways: generating masks for every object in an image automatically, creating segmentation masks from bounding box prompts, and converting an object detection dataset into segmentation masks.

If your goal is labeling data rather than writing code, Roboflow Annotate uses SAM to power automated polygon labeling in the browser, free to try.

What Is Segment Anything?

Segment Anything (SAM) is an image segmentation model released by Meta AI in April 2023 under an Apache 2.0 license. Given a prompt (a point, a box, or nothing at all), it returns pixel-precise masks for objects it has never been trained to recognize, zero-shot performance that is often competitive with fully supervised models. For the architecture details, read our SAM technical deep dive.

SAM is now a family. SAM 2 (July 2024) extends segmentation to video with a streaming memory that tracks objects across frames, and is roughly 6x more accurate than the original on image segmentation. SAM 3 (November 2025) adds concept prompts: give it plain text like helmet or forklift and it detects, segments, and tracks every matching instance across images and video.

Eager to try SAM 3? Drag and drop an image into our interactive playground below with your own text prompt:

For more information on how SAM works and the model architecture, read our SAM technical deep dive.

Background: Object Detection vs. Segmentation

In object detection tasks, objects are represented by bounding boxes, which are like drawing a rectangle around the object. These rectangles give a general idea of the object's location, but they don't show the exact shape of the object. They may also include parts of the background or other objects inside the rectangle, making it difficult to separate objects from their surroundings.

Segmentation masks, on the other hand, are like drawing a detailed outline around the object, following its exact shape. This allows for a more precise understanding of the object's shape, size, and position.

To use Segment Anything on a local machine, we'll follow these steps:

1. Set up a Python environment
2. Load the Segment Anything Model (SAM)
3. Generate masks automatically with SAM
4. Plot masks onto an image with Supervision
5. Generate bounding boxes from the SAM results

Which SAM Should You Use?

• Use the original SAM (this tutorial's code) for local scripting on single images: automatic mask generation, box-prompted masks, and dataset conversion.
• Use SAM 2 when you need masks tracked across video frames.
• Use SAM 3 when you want to find objects by name: text prompts replace manual clicks, and it handles detection, segmentation, and tracking in one model. SAM 3 is integrated across the Roboflow platform, including zero-shot use in Workflows and dataset auto-labeling.

Setting up Your Python Environment

To get started, open the Roboflow notebook in Google Colab and ensure you have access to a GPU for faster processing. Next, install the required project dependencies and download the necessary files, including SAM weights.

pip install \
'git+https://github.com/facebookresearch/segment-anything.git'
pip install -q roboflow supervision
wget -q \
'https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth'

Loading the Segment Anything Model

Once your environment is set up, load the SAM model into the memory. With multiple modes available for inference, you can use the model to generate masks in various ways. We will explore automated mask generation, generating segmentation masks with bounding boxes, and converting object detection datasets into segmentation masks.

The SAM model can be loaded with 3 different encoders: ViT-B, ViT-L, and ViT-H. ViT-H improves substantially over ViT-B but has only marginal gains over ViT-L. These encoders have different parameter counts, with ViT-B having 91M, ViT-L having 308M, and ViT-H having 636M parameters. This difference in size also influences the speed of inference, so keep that in mind when choosing the encoder for your specific use case.

import torch
from segment_anything import sam_model_registry

DEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
MODEL_TYPE = "vit_h"

sam = sam_model_registry[MODEL_TYPE](checkpoint=CHECKPOINT_PATH)
sam.to(device=DEVICE)

Automated Mask (Instance Segmentation) Generation with SAM

To generate masks automatically, use the SamAutomaticMaskGenerator. This utility generates a list of dictionaries describing individual segmentations. Each dict in the result list has the following format:

• segmentation - [np.ndarray] - the mask with (W, H) shape, and bool type, where W and H are the width and height of the original image, respectively
• area - [int] - the area of the mask in pixels
• bbox - [List[int]] - the boundary box detection in xywh format
• predicted_iou - [float] - the model's own prediction for the quality of the mask
• point_coords - [List[List[float]]] - the sampled input point that generated this mask
• stability_score - [float] - an additional measure of mask quality
• crop_box - List[int] - the crop of the image used to generate this mask in xywh format

To run the code below you will need images. You can use your own, programmatically pull them in from Roboflow, or download one of the over 200k datasets available on Roboflow Universe.

import cv2
from segment_anything import SamAutomaticMaskGenerator

mask_generator = SamAutomaticMaskGenerator(sam)

image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
result = mask_generator.generate(image_rgb)

The supervision package (starting from version 0.5.0) provides native support for SAM, making it easier to annotate segmentations on an image.

import supervision as sv

mask_annotator = sv.MaskAnnotator(color_map = "index")
detections = sv.Detections.from_sam(result)
annotated_image = mask_annotator.annotate(image_bgr, detections)

Generate Segmentation Mask with Bounding Box

Now that you know how to generate a mask for all objects in an image, let’s see how you can use a bounding box to focus SAM on a specific portion of your image.

To extract masks related to specific areas of an image, import the SamPredictor and pass your bounding box through the mask predictor’s predict method. Note that the mask predictor has a different output format than the automated mask generator. The bounding box format for the SAM model should be in the form of [x_min, y_min, x_max, y_max] np.array.

import cv2
from segment_anything import SamPredictor

mask_predictor = SamPredictor(sam)

image_bgr = cv2.imread(IMAGE_PATH)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
mask_predictor.set_image(image_rgb)

box = np.array([70, 247, 626, 926])
masks, scores, logits = mask_predictor.predict(
    box=box,
    multimask_output=True
)

Convert a Detection Dataset into a Segmentation Dataset

If you have an object detection dataset, SAM can upgrade every bounding box into a pixel-precise mask: load the COCO-format annotations, run each box through the predictor, and save the resulting masks.

The companion notebook includes the full conversion code. This turns labeling work you have already done into training data for segmentation models without redrawing a single polygon.

Label Data with SAM in the Browser

For dataset labeling, you do not need to run SAM yourself. Roboflow Annotate's Smart Polygon uses SAM to convert clicks into precise polygons in the browser, and SAM 3 auto-labeling goes further: describe your classes in text and it labels entire datasets, with your review replacing manual drawing.

Run SAM at Scale

Roboflow Inference, the open source inference server, serves the whole family on your own hardware or in the cloud: SAM, SAM 2, and SAM 3. In Roboflow Workflows, SAM 3 runs zero-shot from a text prompt and chains with detection models, filters, tracking, and notifications into full applications.

From SAM Masks to a Production Model

SAM models are large and general; production lines usually want small and specific. The standard pattern pairs them: use SAM 3 to auto-label a segmentation dataset for your classes, then train RF-DETR Seg, Roboflow's real-time segmentation architecture, on the result. You get SAM-quality masks at a fraction of the inference cost, in a model you own under a permissive license, deployable to the edge.

Deploy Segment Anything at Scale

You can run Segment Anything (SAM) on your own hardware, at scale, using Roboflow Inference. Roboflow Inference is an inference server through which you can run fine-tuned models such as RF-DETR, as well as foundation models like SAM and CLIP.

Learn how to get started with the Inference Segment Anything quickstart.

Is Segment Anything free for commercial use?

Yes. SAM's code and weights are released under the Apache 2.0 license. Check the license terms of SAM 2 and SAM 3 individually, as Meta's release terms vary by model.

What is the difference between SAM, SAM 2, and SAM 3?

SAM segments images from point and box prompts. SAM 2 adds video with streaming memory that tracks masks across frames. SAM 3 adds text prompts, finding, segmenting, and tracking every instance of a named concept, which removes the need to click each object.

Does SAM tell me what the objects are?

No. SAM produces masks without class labels. Pair it with a detection model that supplies labeled boxes as prompts, or use SAM 3, where the text prompt itself carries the class.

Can I fine-tune Segment Anything?

SAM 3 can be fine-tuned on your own data in Roboflow (available on paid plans). For most tasks, the faster route is using SAM 3 to label data and training a compact model like RF-DETR Seg on the output.

Do I need a GPU to run SAM?

For interactive local use, a GPU is strongly recommended, especially with the ViT-H encoder. Without one, use the browser-based Smart Polygon labeling or the hosted endpoints through Roboflow Inference.

Cite this Post

Use the following entry to cite this post in your research:

Piotr Skalski. (Jan 22, 2026). How to Use the Segment Anything Model (SAM). Roboflow Blog: https://blog.roboflow.com/how-to-use-segment-anything-model-sam/