Inter-annotator agreement is the number most teams check first, and it is the number most likely to mislead them. A 2026 survey of agreement metrics for annotation and evaluation makes the point directly: a homogeneous group of annotators can reach high agreement precisely because its members share the same interpretive frame — including where that […]
