A model can only be as good as the labels it learns from. Yet "99% accurate" means very little unless you know what was measured and how. This guide explains the quality metrics that matter when you buy or run data annotation, and how to check them yourself.
1. Label accuracy against a gold set
The simplest and most useful check. Build a gold set: a few hundred items labelled carefully by your own experts. Mix them, unmarked, into the work your annotation team receives, then compare.
- Classification: the share of items with the correct class.
- Attributes: accuracy per attribute (occluded, truncated, colour), since errors often hide here.
- Missed and extra objects: count objects the annotator skipped and objects that do not exist.
Report accuracy per class, not just overall. A dataset that is 99% accurate on cars and 80% on cyclists will fail where it matters most.
2. Intersection over Union (IoU) for geometry
For bounding boxes, polygons, masks and 3D cuboids, "correct" also means "in the right place". IoU divides the overlap between the annotator's shape and the reference shape by their combined area.
| Label type | Typical IoU check | Notes |
|---|---|---|
| 2D bounding boxes | IoU ≥ 0.5 to 0.7 counts as a match | Tighten for small objects and safety-critical classes. |
| Polygons and masks | Mask IoU or boundary F-score | Boundary metrics catch sloppy edges that area-based IoU forgives. |
| 3D cuboids (LiDAR) | 3D IoU plus heading error | Check yaw separately: a box can overlap well and still face the wrong way. |
| Keypoints | Distance error, often normalised by object size | Visibility flags need their own accuracy check. |
Agree the threshold in your guidelines before the work starts, so pass or fail is never a matter of opinion.
3. Precision and recall of labels
Once matches are defined by IoU, you can borrow two familiar ideas from model evaluation:
- Precision: of the objects the annotator labelled, how many are real and correctly classed? Low precision means false positives.
- Recall: of the real objects, how many did the annotator label? Low recall means missed objects, the most common and most damaging annotation error in perception datasets.
4. Inter-annotator agreement
When two trained people label the same item, how often do they agree? Agreement shows whether your guidelines are clear, not just whether one person is careful.
- Cohen's kappa for two annotators on categories; it corrects for agreement by chance.
- Fleiss' kappa or Krippendorff's alpha for three or more annotators, or missing data.
- Mean IoU between annotators for geometry.
Low agreement on one class is a signal to rewrite that part of the guideline and add examples, not just to retrain the team.
5. QA sampling: how much to check
Reviewing 100% of labels doubles the cost. Reviewing too little hides problems. Common practice:
- Review a high share (often all) of the pilot and first batches while guidelines settle.
- Move to random sampling per batch once quality is stable, with a clear accept/reject threshold.
- Keep full review for rare, safety-critical or high-disagreement classes.
- Reject and rework a batch that fails the sample, rather than fixing only the sampled items.
6. Questions to ask an annotation vendor
- What exactly does your accuracy figure measure, and against what reference?
- Do you report accuracy per class and per attribute?
- What IoU threshold do you use for a match?
- Who does QA: the same team, or an independent reviewer?
- What share of each batch is reviewed, and what happens when a sample fails?
- Can we run a free pilot and check it against our own gold set?
How Annova measures quality
Every Annova project runs through a 3-layer quality shield: expert annotation, AI-assisted pre-labelling and blind cross-verification by a second, independent QA team, targeting 99%+ accuracy. Start with a free pilot and measure it with the checks above, or read more about our data annotation services in India.
