Pigeon 3D Tracking

3D-POP Dataset

Using a large motion tracking system and pigeons with attached head and backpack markers, we tracked the fine-scale position and orientation of pigeon individuals. By finding the relative position of markers to keypoints of interest (e.g eyes, beak) by annotating 5-10 frames, we propagated keypoints using motion tracking information, allowing us to produce annotations for bounding box, individual trajectories, 2D and 3D keypoints for groups of 1, 2,5 and 10 pigeons.

3D-Muppet: 3D Multi-Pigeon Pose Estimation and Tracking

Using the 3D-POP dataset, we trained 2D keypoint detectors and triangulated 2D postures into 3D. The approach worked well and yielded high accuracy (median 6.9mm keypoint error). We also show that training a model with single pigeon data also works with multiple pigeons, for future biologists who can potentially avoid annotating multi-animal data, which is much more labour intensive than labelling single individuals.

Pigeons Everywhere: 2D postures of pigeons in all environments

Finally, we show that a model trained using data collected in captivity (3D-POP) can also work in pigeons in the wild, allowing 2D and 3D posture estimation of multiple individuals in the field. Using the segment anything model, we trained a 2D keypoint detection model of masked pigeons to remove the influence of the background. During inference, we first used a pre-trained MaskRCNN model to get pigeon masks, and managed to get 2D keypoint detection without additional annotations for training.