FishTrack is an experimental, open-source Python desktop application for detecting, segmenting and tracking fish in recorded underwater video. It combines AI-based object detection and segmentation with conventional computer-vision techniques to explore the challenges of following individual fish through complex underwater environments.
The project grew out of a trip to the Maldives that produced around 50 GB of GoPro footage. Following the development of ReefFrame, which dehazes, enhances and stabilises underwater video, FishTrack was a natural next step: Rather than simply extracting interesting frames, could computer vision be used to find individual fish and follow their movements through the footage?
This is a surprisingly difficult problem. Underwater scenes contain complex backgrounds, changing illumination, suspended particles and moving vegetation. Camera movement introduces further complications, particularly when foreground and background objects appear to move at different rates. Fish themselves may change direction, overlap, disappear behind coral or become difficult to distinguish from their surroundings.
FishTrack provides an interactive workflow for experimenting with these challenges:
- Fish detection: Use Grounding DINO to propose candidate fish regions from a starting frame, or select regions manually
- Fish tracking: Choose between conventional computer-vision tracking and SAM 2, allowing different approaches to be explored and compared
- Segmentation: Use SAM 2 to generate pixel-level masks representing the predicted extent of tracked fish
- Video annotation: Generate a separate annotated MP4 showing tracking results, without modifying the original footage
- Interactive configuration: Use the desktop interface to select video footage, review candidate regions, configure processing options and inspect results
FishTrack is intended as a tool for experimentation, exploration and visual annotation, rather than a validated system for species identification, population assessment or behavioural measurement. Detection and tracking results are algorithmic predictions and may be incorrect, particularly in challenging underwater scenes.
The project is also an opportunity to investigate how modern AI techniques such as Grounding DINO and SAM 2 compare with more traditional computer-vision approaches when applied to real-world underwater footage.
Please see the Wiki for the user guide, controls and further details.
- Dave Walker - Initial work
To report an issue or suggest an improvement, please use the project's GitHub Issues page.
This project is licensed under the MIT License — see the LICENSE file for details.