Bounding box vs polygon vs segmentation: which annotation type do you need?

A side-by-side comparison of the main image annotation types, with a simple decision guide so you do not pay for precision your model does not need.

The label type you choose sets both your model's ceiling and your annotation budget. Too simple and the model cannot learn the shapes it needs; too detailed and you pay for precision you never use. Here is how the main image annotation types compare.

The main types at a glance

TypeWhat you drawPrecisionRelative effortGood for
ClassificationOne tag per imageNone (no location)LowestSorting, content moderation, quality pass/fail
Bounding boxA rectangle around each objectRoughLowObject detection, counting, retail, traffic
PolygonPoints tracing the object's outlineGoodMediumIrregular shapes: buildings, fields, damage areas
Semantic segmentationA class for every pixelVery highHighDrivable area, land cover, medical regions
Instance segmentationA separate mask for each objectVery highHighestRobot picking, cell counting, overlapping objects
KeypointsNamed points (joints, landmarks)Point-levelMediumPose estimation, faces, product measurements
PolylinesOpen linesLine-levelLow to mediumLanes, roads, power lines, pipelines
3D cuboidsA 3D box with headingSpatialMedium to highDepth and orientation; see the LiDAR guide

Bounding boxes: fast and often enough

Boxes are quick to draw and to review, and most detection models are built for them. Their weakness is that they include background: a box around a diagonal pole or a bent arm is mostly empty space. Use boxes when you need to know what and roughly where.

Polygons: the middle ground

Polygons follow the outline, so they capture shape and exclude most background. They suit objects with irregular edges, and they are the usual choice for aerial and satellite imagery, such as building footprints and field boundaries.

Segmentation: when every pixel matters

Semantic segmentation tells the model which class each pixel belongs to, but does not separate two touching cars. Instance segmentation separates them. Both cost more, though AI-assisted tools (promptable segmentation models that turn a click into a mask) have cut the effort a lot; annotators then refine the edges.

A simple decision guide

  1. Do you only need to know if something is in the image? Use classification.
  2. Do you need where, and rough location is enough? Use bounding boxes.
  3. Does the shape matter, or do boxes include too much background? Use polygons.
  4. Do you need exact area or boundaries? Use semantic segmentation; if objects touch or overlap and must be counted, use instance segmentation.
  5. Is the target a pose or a line? Use keypoints or polylines.

You can mix types in one project: boxes for most classes and masks only for the classes where shape matters. That alone can cut cost a lot; see how annotation is priced.

Test before you commit

Label a small sample two ways, train quick baselines and compare. If boxes get you close to your accuracy target, the cheaper option wins. A free Annova pilot is a good place to run that test, and our image and video annotation team supports every type above.

Related services: Image & video annotation · Autonomous vehicle annotation · How annotation is priced

See our quality on your own data

Send a sample and your guidelines. We'll label a free pilot batch you can check before you commit.

Request a free pilot