Today, we are excited to release support for instance segmentation projects in Roboflow.  

Instance segmentation allows your computer vision model to know the specific outline of an object in an image, unlocking new use cases for Roboflow in your application.

Update: Train & Deploy

Roboflow has now added support for end-to-end Instance Segmentation projects. You can create segmentation datasets, annotate, train, and deploy them to an infinitely scalable API with a single click.

What Is Instance Segmentation?

Instance segmentation, also known as image segmentation, is the computer vision task of recognizing objects in images along with their associated shape. It is an extension of object detection where each prediction also includes a shape as opposed to just a bounding box defined by a center point, width, and height.

With instance segmentation, your application can determine the number of objects in an image, the classifications, and their outline.

It's useful in cases where you need to measure the size of detected objects (eg a tomato leaf), cut them out of their background (eg for background removal), or more accurately detect oblong rotated objects (eg a bridge in a satellite image).

Instance Segmentation Dataset Format

In instance segmentation, the most common format is COCO. The COCO format records the points of the segmentation in a series of x,y pairs.

        {
            "id": 4,
            "image_id": 1,
            "category_id": 2,
            "bbox": [
                273,
                451,
                98.81642999999997,
                119.86870800000003
            ],
            "area": 11844.99779327244,
            "segmentation": [
                [
                    294.526515,
                    451.00638,
                    273.495847,
                    518.689734,
                    372.312277,
                    570.875088,
                    295.413657,
                    452.786753,
                    294.526515,
                    451.00638
                ]
            ],
            "iscrowd": 0
        }

You can now import and export your instance segmentation data from Roboflow in the COCO format.

Instance Segmentation Inference Output

After you have trained a model for instance segmentation, your model will predict both bounding boxes and instance segmentation masks around the detected object.

Example instance segmentation inference from a model trained on the public American Sign Language dataset

Instance Segmentation vs Object Detection

  • Object detection only predicts the object's bounding box, not the outline.
  • Instance segmentation labeling is more laborious.
  • It is harder to train accurate instance segmentation models.
  • Instance segmentation models are typically larger, slower, and less optimized for edge deployment.
  • Instance segmentation models may need bigger datasets to obtain the same accuracy as object detection models.

You should only use instance segmentation if the specificity of the object's outline is required by your application. It's often better to try object detection first because it's easier and cheaper to test, and faster and more accurate to run.

Real world examples:

Instance Segmentation vs Semantic Segmentation

Semantic segmentation labels each pixel in an image with a class label. Similarly to instance segmentation, you can see the contours of objects in an image, but unlike instance segmentation, you can not count or differentiate between separate objects if the objects are overlapping.

How to Label Images for Instance Segmentation

You can now label images for instance segmentation by drawing polygons within Roboflow Annotate.

Annotating the outline of a hand making the letter Q in Roboflow

How to Train an Instance Segmentation Model

Once you have labeled your instance segmentation dataset, it is time to train a model to make predictions on images you have not labeled.

Roboflow offers an automated machine learning solution via Roboflow Train for training and deploying instance segmentation models with a single click. Follow that link to see how you can easily train and deploy instance segmentation into production.

If you place to train a model outside of Roboflow, check out our tutorial: How to Train Detectron2 for Instance Segmentation. You'll see how training works when not using an autoML solution.

Try Instance Segmentation for Free

We are excited to announce the launch of instance segmentation projects in Roboflow today. The new project type extends Roboflow Annotate's capabilities to polygons so you can detect objects with greater specificity 🚀. Instance segmentation and polygon labeling is included in all of our pricing plans.

To get started, log in to your Roboflow account, create a new project, and choose the Instance Segmentation project type.

Happy labeling, happy training, and of course, happy inferencing.