Annotation quality metrics: how to measure training data accuracy

Label accuracy, IoU, precision and recall, inter-annotator agreement and QA sampling, explained for teams buying or running data annotation.

A model can only be as good as the labels it learns from. Yet "99% accurate" means very little unless you know what was measured and how. This guide explains the quality metrics that matter when you buy or run data annotation, and how to check them yourself.

1. Label accuracy against a gold set

The simplest and most useful check. Build a gold set: a few hundred items labelled carefully by your own experts. Mix them, unmarked, into the work your annotation team receives, then compare.

  • Classification: the share of items with the correct class.
  • Attributes: accuracy per attribute (occluded, truncated, colour), since errors often hide here.
  • Missed and extra objects: count objects the annotator skipped and objects that do not exist.

Report accuracy per class, not just overall. A dataset that is 99% accurate on cars and 80% on cyclists will fail where it matters most.

2. Intersection over Union (IoU) for geometry

For bounding boxes, polygons, masks and 3D cuboids, "correct" also means "in the right place". IoU divides the overlap between the annotator's shape and the reference shape by their combined area.

Label typeTypical IoU checkNotes
2D bounding boxesIoU ≥ 0.5 to 0.7 counts as a matchTighten for small objects and safety-critical classes.
Polygons and masksMask IoU or boundary F-scoreBoundary metrics catch sloppy edges that area-based IoU forgives.
3D cuboids (LiDAR)3D IoU plus heading errorCheck yaw separately: a box can overlap well and still face the wrong way.
KeypointsDistance error, often normalised by object sizeVisibility flags need their own accuracy check.

Agree the threshold in your guidelines before the work starts, so pass or fail is never a matter of opinion.

3. Precision and recall of labels

Once matches are defined by IoU, you can borrow two familiar ideas from model evaluation:

  • Precision: of the objects the annotator labelled, how many are real and correctly classed? Low precision means false positives.
  • Recall: of the real objects, how many did the annotator label? Low recall means missed objects, the most common and most damaging annotation error in perception datasets.

4. Inter-annotator agreement

When two trained people label the same item, how often do they agree? Agreement shows whether your guidelines are clear, not just whether one person is careful.

  • Cohen's kappa for two annotators on categories; it corrects for agreement by chance.
  • Fleiss' kappa or Krippendorff's alpha for three or more annotators, or missing data.
  • Mean IoU between annotators for geometry.

Low agreement on one class is a signal to rewrite that part of the guideline and add examples, not just to retrain the team.

5. QA sampling: how much to check

Reviewing 100% of labels doubles the cost. Reviewing too little hides problems. Common practice:

  • Review a high share (often all) of the pilot and first batches while guidelines settle.
  • Move to random sampling per batch once quality is stable, with a clear accept/reject threshold.
  • Keep full review for rare, safety-critical or high-disagreement classes.
  • Reject and rework a batch that fails the sample, rather than fixing only the sampled items.

6. Questions to ask an annotation vendor

  • What exactly does your accuracy figure measure, and against what reference?
  • Do you report accuracy per class and per attribute?
  • What IoU threshold do you use for a match?
  • Who does QA: the same team, or an independent reviewer?
  • What share of each batch is reviewed, and what happens when a sample fails?
  • Can we run a free pilot and check it against our own gold set?

How Annova measures quality

Every Annova project runs through a 3-layer quality shield: expert annotation, AI-assisted pre-labelling and blind cross-verification by a second, independent QA team, targeting 99%+ accuracy. Start with a free pilot and measure it with the checks above, or read more about our data annotation services in India.

Related services: Data annotation services in India · LiDAR & 3D point cloud annotation · Annova FAQ

See our quality on your own data

Send a sample and your guidelines. We'll label a free pilot batch you can check before you commit.

Request a free pilot