Creating a high quality dataset for computer vision is essential to having strong model performance. In addition to collecting images that are as similar to your deployed conditions as possible, labeling images carefully and accurately is essential.

Check out the video version of this article on our YouTube channel.

In order to label images effectively, the below post walks through tactics to ensure your dataset is as high quality as possible. While the below best practices are generally true, it is important to note that labeling instructions are highly dependent on the nature of the task at hand. Moreover, images labeled for one task, may not be suitable for another task – re-labeling is not uncommon. It is best to think of a dataset and its labels as something alive: constantly changing and improving to fit the task at hand.

With that in mind, lets walkthrough a few tips.

Unsure which images to label first? Consider how to use active learning in computer vision.

1. Label Every Object of Interest in Every Image

Computer vision models are built to learn what patterns of pixels correspond to an object of interest. Because of this, if we're training a model to identify an object, we need to label every appearance of that object in our images. If we do not label the object in some images, we will be introducing false negatives to our model.

For example, in a chess piece dataset, we need to label the appearance of every single piece on the board – we would not label only some of, say, the white pawns.

Chess dataset labeled for object detection.
Label every occurrence of our objects of interest. 

2. Label the Entirety of an Object

Our bounding boxes should enclose the entirety of an object of interest. Labeling only a portion of the object confuses our model for what a full object constitutes.

In our chess dataset, for example, notice how each piece is fully enclosed in a bounding box.

Chess dataset with bounding boxes completely around the pieces.
The entirety of a piece is labeled. 

3. Label Occluded Objects

Occlusion is when an object is partially out of view in an image due to something blocking it in a photo. It is best to label even occluded objects. Moreover, it is commonly best practice to label the occluded object as if it were fully visible – rather than drawing a bounding box for only the partially visible portion of the object.

For example, in the chess dataset, one piece will regularly occlude the view of another. Both objects should be labeled, even if the boxes overlap. (It is a common misconception that boxes cannot overlap.)

Label objects completely even if they are occluded.
Even if object is blocking view of another, it is best to label them both as if they were both fully visible.

4. Create Tight Bounding Boxes

Bounding boxes should be tight around the objects of interest. (But, you should never have a box so tight that it is cutting off a portion of the object.) Tight bounding boxes are critical to helping our model learn, precisely, which pixels constitute an object of interest vs irrelevant portions of an image.

Object detection bounding boxes should be tight around an object.
Bounding boxes should be tight. Here, we're making existing bounding boxes even tighter.

5. Create Specific Label Names

When determining a given object's label name, it is better to error on the side of being more specific rather than less. It is always easier to remap label classes to be more general, whereas being more specific requires relabeling. For example, imagine you are building a dog detector. While every object of interest is a dog, it may be wise to create a class for labrador and poodle. In initial model building, our labels could be combined to be dog. But, if we had started with dog and later realized having individual breeds is important, we would have to relabel our dataset altogether.

In our chess dataset, for example, we have white-pawn and black-pawn. We could always combine these to be pawn, or even combine all classes to be piece.  

Create specific class names for labels, like in chess, we have specific pieces.
Class names should be specific, like black-pawn rather than pawn or piece.

6. Maintain Clear Labeling Instructions

Inevitably, we will need to add more data to our dataset – this is a key element to model improvement. Tactics like active learning ensure that we spend our time labeling intelligently. As such, having clear, shareable, and repeatable labeling instructions is essential for both our future selves and coworkers to create and maintain high quality datasets.

Many of the techniques we've discussed here should be included: label the entirety of an object, make labels tight, label all objects, etc. It is always better to err on the side of more specificity than less.

7. Use These Labeling Tools

Now that we know how to label effectively, how should we label our data? If we're labeling ourselves, we can use tools like CVAT, LabelImg, RectLabel (for Mac), and now Roboflow to add and maintain labels.

When we're ready to scale up our labeling efforts either with teammates or an outsourced workforce, we can pick from a marketplace of labeling services like AWS, Scale, and others. (Reach out to us if you'd like to setup a test to see which service is best for your data.)

As always, happy building!