Data from: Measuring agreement among experts in classifying camera images of similar species

Here, we measure agreement among experts in distinguishing between images of two similar congeneric species, bobcats (Lynx rufus) and Canada lynx (L. canadensis). We asked experts to classify the species in selected images to test whether the season, background habitat, time of day, and the visible features of each animal (e.g., face, legs, tail) affected agreement among experts about the species in each image. Overall, experts had moderate agreement (Fleiss’ kappa = 0.64), but experts had varying levels of agreement depending on these image characteristics.

Most images (71%) had ≥1 expert classification of ‘unknown’, and many images (39%) had some experts classify the image as ‘bobcat’ while others classified it as ‘lynx’. Further, experts were inconsistent even with themselves, changing their classifications of numerous images when they were asked to reclassify the same images months later. These results suggest that classification of images by a single expert is unreliable for similar-looking species.

Most of the images did obtain a clear majority classification from the experts, although we emphasize that even majority classifications may be incorrect. We recommend that researchers using wildlife images consult multiple species experts to increase confidence in their image classifications of similar sympatric species. Still, when the presence of a species with similar sympatrics must be conclusive, physical or genetic evidence should be required.

Usage notes

Image classifications Table shows the list of images used in our experiment (image ID), and for each image the trial and category that is was used in, the total number of experts that classified it as 'bobcat', 'lynx', and 'unknown', and the individual classifications of each expert (1 through 27).