Creating a high quality dataset for computer vision is essential to having strong model performance. In addition to collecting images that are as similar to your deployed conditions as possible, labeling images carefully and accurately is essential.
In order to label images effectively, the below post walks through tactics to ensure your dataset is as high quality as possible. While the below best practices are generally true, it is important to note that labeling instructions are highly dependent on the nature of the task at hand. Moreover, images labeled for one task, may not be suitable for another task – re-labeling is not uncommon. It is best to think of a dataset and its labels as something alive: constantly changing and improving to fit the task at hand.
With that in mind, lets walkthrough a few tips.
Unsure which images to label first? Consider how to use active learning in computer vision.
1. Label Every Object of Interest in Every Image
Computer vision models are built to learn what patterns of pixels correspond to an object of interest. Because of this, if we're training a model to identify an object, we need to label every appearance of that object in our images. If we do not label the object in some images, we will be introducing false negatives to our model.
For example, in a chess piece dataset, we need to label the appearance of every single piece on the board – we would not label only some of, say, the white pawns.
2. Label the Entirety of an Object
Our bounding boxes should enclose the entirety of an object of interest. Labeling only a portion of the object confuses our model for what a full object constitutes.
In our chess dataset, for example, notice how each piece is fully enclosed in a bounding box.
3. Label Occluded Objects
Occlusion is when an object is partially out of view in an image due to something blocking it in a photo. It is best to label even occluded objects. Moreover, it is commonly best practice to label the occluded object as if it were fully visible – rather than drawing a bounding box for only the partially visible portion of the object.
For example, in the chess dataset, one piece will regularly occlude the view of another. Both objects should be labeled, even if the boxes overlap. (It is a common misconception that boxes cannot overlap.)
4. Create Tight Bounding Boxes
Bounding boxes should be tight around the objects of interest. (But, you should never have a box so tight that it is cutting off a portion of the object.) Tight bounding boxes are critical to helping our model learn, precisely, which pixels constitute an object of interest vs irrelevant portions of an image.
5. Create Specific Label Names
When determining a given object's label name, it is better to error on the side of being more specific rather than less. It is always easier to remap label classes to be more general, whereas being more specific requires relabeling. For example, imagine you are building a dog detector. While every object of interest is a dog, it may be wise to create a class for
poodle. In initial model building, our labels could be combined to be
dog. But, if we had started with
dog and later realized having individual breeds is important, we would have to relabel our dataset altogether.
In our chess dataset, for example, we have
black-pawn. We could always combine these to be
pawn, or even combine all classes to be
6. Maintain Clear Labeling Instructions
Inevitably, we will need to add more data to our dataset – this is a key element to model improvement. Tactics like active learning ensure that we spend our time labeling intelligently. As such, having clear, shareable, and repeatable labeling instructions is essential for both our future selves and coworkers to create and maintain high quality datasets.
Many of the techniques we've discussed here should be included: label the entirety of an object, make labels tight, label all objects, etc. It is always better to err on the side of more specificity than less.
7. Use These Labeling Tools
Now that we know how to label effectively, how should we label our data? If we're labeling ourselves, we can use tools like CVAT, LabelImg, RectLabel (for Mac), and now Roboflow to add and maintain labels.
When we're ready to scale up our labeling efforts either with teammates or an outsourced workforce, we can pick from a marketplace of labeling services like AWS, Scale, and others. (Reach out to us if you'd like to setup a test to see which service is best for your data.)
As always, happy building!