How to Label Images for Computer Vision Models

Creating a high quality dataset for computer vision is essential to having strong model performance.

In addition to collecting images that are as similar to your deployed conditions as possible, labeling images carefully and accurately is essential.

But how do you label images? What should you keep in mind to label images well? We're going to answer those questions in this post. By the end of this post, you'll have a few tips to keep in mind so that your image annotations are correct and helpful a model.

What is image labeling?

Image labelling is when you annotate specific objects or features in an image. Image labels teach computer vision models how to identify a particular object in an image. For example, in a set of aerial view images, you might annotate all of the trees. The labels will help a model understand what a tree is.

Image labelling can be done with a variety of annotation tools. These annotation tools let you draw specific boundaries around objects. These boundaries are called "bounding boxes". Each bounding box is given a label so that the model can differentiate different objects. For example, all trees might be labeled as "tree" and all houses might be annotated as "house".

The quality of your image labels will directly affect the accuracy of a trained model. Using the right labeling and annotation strategy, you can produce a high-quality dataset that will help a model better learn how to identify the objects you have labelled.

Caveat: Labeling instructions depend on your task

While the below best practices are generally true, it is important to note that labeling instructions are highly dependent on the nature of the task at hand.

Moreover, images labeled for one task, may not be suitable for another task – re-labeling is not uncommon. It is best to think of a dataset and its labels as something alive: constantly changing and improving to fit the task at hand.

How to label images for computer vision tasks

With that in mind, lets walkthrough a few tips on how to effectively label images.

Unsure which images to label first? Consider how to use active learning in computer vision.

1. Label Every Object of Interest in Every Image

Computer vision models are built to learn what patterns of pixels correspond to an object of interest.

Because of this, if we're training a model to identify an object, we need to label every appearance of that object in our images. If we do not label the object in some images, we will be introducing false negatives to our model.

For example, in a chess piece dataset, we need to label the appearance of every single piece on the board – we would not label only some of, say, the white pawns.

2. Label the Entirety of an Object

Our bounding boxes should enclose the entirety of an object of interest. Labeling only a portion of the object confuses our model for what a full object constitutes.

In our chess dataset, for example, notice how each piece is fully enclosed in a bounding box.

3. Label Occluded Objects

Occlusion is when an object is partially out of view in an image due to something blocking it in a photo. It is best to label even occluded objects.

Moreover, it is commonly best practice to label the occluded object as if it were fully visible – rather than drawing a bounding box for only the partially visible portion of the object.

For example, in the chess dataset, one piece will regularly occlude the view of another. Both objects should be labeled, even if the boxes overlap. (It is a common misconception that boxes cannot overlap.)

4. Create Tight Bounding Boxes

Bounding boxes should be tight around the objects of interest. (But, you should never have a box so tight that it is cutting off a portion of the object.) Tight bounding boxes are critical to helping our model learn, precisely, which pixels constitute an object of interest vs irrelevant portions of an image.

5. Create Specific Label Names

When determining a given object's label name, it is better to error on the side of being more specific rather than less. It is always easier to remap label classes to be more general, whereas being more specific requires relabeling.

For example, imagine you are building a dog detector. While every object of interest is a dog, it may be wise to create a class for labrador and poodle. In initial model building, our labels could be combined to be dog. But, if we had started with dog and later realized having individual breeds is important, we would have to relabel our dataset altogether.

In our chess dataset, for example, we have white-pawn and black-pawn. We could always combine these to be pawn, or even combine all classes to be piece.  

6. Maintain Clear Labeling Instructions

Inevitably, we will need to add more data to our dataset – this is a key element to model improvement. Tactics like active learning ensure that we spend our time labeling intelligently. As such, having clear, shareable, and repeatable labeling instructions is essential for both our future selves and coworkers to create and maintain high quality datasets.

Many of the techniques we've discussed here should be included: label the entirety of an object, make labels tight, label all objects, etc. It is always better to err on the side of more specificity than less.

7. Use These Labeling Tools

Now that we know how to label effectively, how should we label our data? If we're labeling ourselves, we can use tools like Roboflow Annotate to add and maintain labels.

When we're ready to scale up our labeling efforts either with teammates or an outsourced workforce, we manage large scale labeling projects with team workflows and label-only users.

As always, happy building!