Launch: Model Prompting for Automated Labeling with Autodistill

Over the last few years, Roboflow has invested significantly in automated image labeling tools and now offer a fully hosted automated labeling product, Auto Label.

We launched model-assisted labeling in Roboflow Annotate, which lets you use existing models you have trained to images during the manual annotation process. We expanded that feature to allow you to use any of 50,000+ trained models to label your data.

We also launched Smart Polygon powered by Segment Anything, which lets you label images with polygons faster than ever with help from the Segment Anything foundation model.

In June 2023, Roboflow launched Autodistill, an open source framework for auto-labeling images using foundation models. Autodistill integrates foundation models like Grounding DINO, CLIP, and Segment Anything and combines them in ways to allow effective use in labeling image data for a wide range of use cases.

We are excited to announce the launch of an interface to help you calibrate automated labeling prompts. With our Autodistill-powered image labeling tool, you can test how multimodal foundation models will perform on your dataset.

By accurately auto-labeling data, you can reduce the most time consuming part of building a computer vision application. Instead of managing a team of labelers, you can automate the process, allowing you to get a proof of concept and, later, a production application into use faster than ever.

In this guide, we are going to walk through how to configure Autodistill for labeling images with foundation models.

Without further ado, let’s get started!

Auto-Label Images with Roboflow

Foundation models like Grounding DINO, Segment Anything, and models that combine existing models (i.e. Grounded SAM) can effectively identify objects in images. For example, you can use Grounded SAM to identify forklifts in an image and retrieve segmentation masks. You can then use these masks as labels for use in training a model.

Models like Grounding DINO and Grounded SAM take in an arbitrary label – for example, “shipping container” – and aim to identify all instances of that object in an image. Although effective, these models are slow to run, making them impractical to use on the edge or in real-time.

For the last nine months, Roboflow has invested heavily in auto-labeling images with foundation models. We started with the open source framework, Autodistill, which enables you to auto-label vision data. Autodistill now supports using over 15 models to label data.

How to Configure Autodistill Prompts in Roboflow

Suppose you have a dataset of forklifts that you want to label for use in training an industrial safety model. You can auto-label images in the Roboflow application. Let’s walk through how to do this step by step.

Step #1: Create a Project

First, create a free Roboflow account. Then, click “Create Project” on the Roboflow dashboard to create a computer vision project. You can automatically label projects that are created with Object Detection or Segmentation model types.

Step #2: Upload Data

Once you have created a project, you will be asked to upload images. Upload the images you want to use in training your vision model by dragging the images into the Roboflow web application.

Step #3: Open the Prompting Tool

When you have uploaded your images, you will be shown a list of all the images in your dataset. Click "Annotate" on the sidebar, then click on the batch of images you uploaded.

On this page, click the “Auto Label Images” button. This button will take you to a page where you can evaluate models for labeling your data.

On this page, you will be asked to choose up to five images to use in evaluating which model performs best on your data. You can do so by selecting images from the page, or by clicking the on-page button to select images at random from your dataset.

You will also be asked to set the classes you want to identify and the corresponding labels. The first value in the table of classes to identify should be the class that you want to send to the model. The second value should be the label that will be saved to your dataset if the class is identified.

For example, if you want to identify shipping containers, you would say you want to find “shipping containers” and set the label name “container”. This would mean:

1. You want to identify shipping containers, and;
2. You want to save all instances of the class as “container” in your dataset.

You can also choose the confidence below which predictions will be filtered out. You can change this confidence level after running the model.

Once you have configured your prompts and classes, click “Test”. The five images you selected will be labeled with all supported models.

Step #4: Evaluate the Results

The results will appear in the browser. You can select images from the left-side of the page to see how each model performed on different images in your dataset.

At this stage, you can evaluate the results to see if they are sufficiently accurate to use in training your model. You can adjust the confidence from the model using the on-page confidence slider to see how each model would perform at different confidences.

Foundation models might miss a few objects, but you can always label images with missing annotations manually after running an automated labeling job.

Step #5: Run Autodistill on Your Images

Autodistill is an open source framework that lets you automatically label images on your device. If any of the models used label the test images you selected perform well, you can use the prompts for use in labeling all of your images.

Conclusion

You are now ready to use the prompts and classes in Autodistill to run your own labeling projects.

Our hosted solution, Auto Label, will allow you to run an automated labeling job on your entire dataset in the cloud, hosted on Roboflow.

If the Roboflow automated labeling tool cannot help with your use case at the moment, check out the Roboflow Outsourced Labeling solution. With our expert outsourced labelers, you can label large datasets without having to manage a team of labelers internally. To learn more about outsourced labeling, fill out the Roboflow Outsourced Labeling intake form.