Over the last few years, Roboflow has invested significantly in automated image labeling tools.
We launched model-assisted labeling, which lets you use existing models you have trained to auto-label images. We expanded that feature to allow you to use any of 50,000+ trained models to label your data. We also launched smart polygon powered by Segment Anything, which lets you label images with polygons faster than ever with help from the Segment Anything foundation model.
In June 2023, Roboflow launched Autodistill, an open source framework for auto-labeling images using foundation models. Autodistill integrates foundation models like Grounding DINO, CLIP, and Segment Anything and combines them in ways to allow effective use in labeling image data for a wide range of use cases.
We are excited to announce the launch of automated image labeling powered by Autodistill. With our Autodistill-powered image labeling tool, you can test how multimodal foundation models like Grounding DINO and Grounded SAM will perform on your dataset. You can then retrieve the code you need to auto-label images using any cloud or local compute options.
By accurately auto-labeling data, you can reduce the most time consuming part of building a computer vision application. Instead of managing a team of labelers, you can automate the process, allowing you to get a proof of concept and, later, a production application into use faster than ever.
In this guide, we are going to walk through how to automatically label images with foundation models. This feature offers a powerful automated labeling experience that we have used to auto-label tens of thousands of images. We will also talk about our future plans for Roboflow’s automated labeling tools.
Without further ado, let’s get started!
Auto-Label Images with Roboflow
Foundation models like Grounding DINO, Segment Anything, and models that combine existing models (i.e. Grounded SAM) can effectively identify objects in images. For example, you can use Grounded SAM to identify forklifts in an image and retrieve segmentation masks. You can then use these masks as labels for use in training a model.
Models like Grounding DINO and Grounded SAM take in an arbitrary label – for example, “shipping container” – and aim to identify all instances of that object in an image. Here is Grounding DINO identifying all shipping containers in an image:
Although effective, these models are slow to run, making them impractical to use on the edge or in real-time.
For the last nine months, Roboflow has invested heavily in auto-labeling images with foundation models. We started with the open source framework, Autodistill, which enables you to auto-label vision data. Autodistill now supports using over 15 models to label data.
Autodistill thrives on general objects (i.e. shipping containers, vehicles, food stuffs), but struggles with specific objects (i.e. a particular component or vehicle model).
Now, we are introducing the models with the broadest capabilities into Roboflow so you can evaluate them on a subset of your data. We will then give you all the code you need to auto-label images on your own device and upload the resulting annotations back to Roboflow.
You can auto-label images using two models:
- Grounding DINO (Object Detection)
- Grounded SAM (Segmentation)
We plan to add more models in the future.
How to Auto-Label Computer Vision Data
Suppose you have a dataset of forklifts that you want to label for use in training an industrial safety model. You can auto-label images in the Roboflow application. Let’s walk through how to do this step by step.
Step #1: Create a Project
First, create a free Roboflow account. Then, click “Create Project” on the Roboflow dashboard to create a computer vision project. You can automatically label projects that are created with Object Detection or Segmentation model types.
Step #2: Upload Data
Once you have created a project, you will be asked to upload images. Upload the images you want to use in training your vision model by dragging the images into the Roboflow web application.
Step #3: Open the Automated Labeling Tool
When you have uploaded your images, you will be shown a list of all the images in your dataset.
Do not assign your images to any labelers on your team, since this will put the images in the labeler's queue. Instead, click "Annotate" on the sidebar, then click on the batch of images you uploaded.
On this page, click the “Auto Label Images” button. This button will take you to a page where you can evaluate models for labeling your data.
On this page, you will be asked to choose up to five images to use in evaluating which model performs best on your data. You can do so by selecting images from the page, or by clicking the on-page button to select images at random from your dataset.
You will also be asked to set the classes you want to identify and the corresponding labels. The first value in the table of classes to identify should be the class that you want to send to the model. The second value should be the label that will be saved to your dataset if the class is identified.
For example, if you want to identify shipping containers, you would say you want to find “shipping containers” and set the label name “container”. This would mean:
- You want to identify shipping containers, and;
- You want to save all instances of the class as “container” in your dataset.
You can also choose the confidence below which predictions will be filtered out. You can change this confidence level after running the model.
Once you have configured your automatic labeling job, click “Test”. The five images you selected will be labeled with all supported models.
Step #4: Evaluate the Results
The results from Roboflow’s automated labeling tool will appear in your browser. You can select images from the left-side of the page to see how each model performed on different images in your dataset.
At this stage, you can evaluate the results from the auto labeling tool to see if they are sufficiently accurate to use in training your model. You can adjust the confidence from the model using the on-page confidence slider to see how each model would perform at different confidences.
Here are the results from Grounding DINO, an object detection model:
Here are the results from Grounded SAM, a segmentation model:
Both models perform well at identifying shipping containers. They miss a few containers, but you can always label images with missing annotations manually after running an automated labeling job.
Step #5: Run Autodistill on Your Images
Roboflow's automated labeling tool is powered by Autodistill, an open source framework that lets you automatically label images on your device.
If any of the models used label the test images you selected perform well, you can retrieve a code snippet for use in labeling all of your images. You can retrieve this snippet by clicking “Get Code”.
The code snippet that appears will have everything you need to label your images and upload them back to Roboflow. We recommend running the code snippet on a GPU device for better performance, although you can run the snippet on CPU devices.
If you do not have a GPU-enabled device, you can run the snippet in a free Google Colab notebook for strong performance.
To run this code snippet, first run the command that appears the top of the box. This will start with
pip install roboflow autodistill, followed by the model packages that you'll need.
Then, copy then code from the window into a new file. The script will download your images from Roboflow, label them, then upload the results to your Roboflow dashboard.
When you run the code snippet, images will be labeled on your device. The resulting annotations are uploaded to Roboflow one-by-one. You can see the results in the Annotate tab in your Roboflow project as images are uploaded.
The Future of Automated Labeling
Roboflow’s automated labeling tool, powered by Autodistill, enables you to automatically label image data with foundation models. You can use the resulting labels to train a fine-tuned model that performs well at your specific use case.
Today, we are launching the first version of our automated labeling solution. In the coming months, we plan to announce support for more using models to auto-label data, support for auto-labeling classification models, and a hosted version of our automated labeling tool. Our hosted solution will allow you to pay to run an automated labeling job on your entire dataset in the cloud, hosted on compute optimized for auto-labeling.
If the Roboflow automated labeling tool cannot help with your use case at the moment, check out the Roboflow Outsourced Labeling solution. With our expert outsourced labelers, you can label large datasets without having to manage a team of labelers internally. To learn more about outsourced labeling, fill out the Roboflow Outsourced Labeling intake form.