Every fish, every frame, in a spreadsheet.

FishAI turns video of a tank filmed from above into data: every fish in every frame, with its species, its position and a track number. From that come the count, where each species swims, how fast, and how diverse the tank is. It is for labs and research groups that film fish. We trained the model ourselves, on our own footage and annotations.

Eight seconds of the run used throughout this page: 48 fish of four species at once, each with its track number.

Tested against a known tank.

We filmed a tank whose stock we knew: 20 corydoras, 15 mollies, 9 zebrafish (cebritas) and 4 guppies, 48 fish. Everything below comes from that one run, unedited: 8 minutes 18 seconds of 1080p video.

A chart of the eight-minute run: the frame-by-frame count jumps between about 40 and 55 fish, while the stabilised count stays mostly between 46 and 49, around the 48 in the tank.

frames analysed: 29,880
of frames with the count within two fish of the 48: 88 %

In orange, the frame-by-frame count: in a full tank there is always one fish hiding another. In black, the count FishAI reports, the median of the last 181 frames (three seconds in this video).

The count of each species over the same run, smoothed over three seconds, against the real stock as dashed lines: corydoras stay around 20, mollies around 15 after the first minute, guppies around 4, and cebritas mostly between 7 and 9, below their 9.

The count of each species against the real stock, dashed. On average: 20.2 corydoras, 15.0 mollies and 4.2 guppies. Zebrafish come up short: 7.6 of 9.

Human review: In a 15-fish tank, a person reviewed 30 frames and FishAI had every one right. In the 48-fish tank no single frame comes out perfect, which is why the numbers on this page are averages and stabilised counts.
Model: mAP50-95 of 0.93 and mAP50 of 0.995 on frames held out from training. They come from the same recordings, so the figure is optimistic.
Beyond training: The model never saw a tank this crowded: it was trained on tanks with fewer fish. This run is the hardest test we have.

Where each species swims.

With every fish's position in every frame, the tank looks different depending on who is using it.

Four heatmaps of the same tank, one per species: corydoras show many small bright spots spread over the floor, mollies are brightest along the walls and in the corners, guppies in the corners and a few spots, and cebritas fill the open middle of the tank.

One heatmap per species, each on its own scale. Mollies and guppies spend half their detections near a wall (52% and 47%, within 120 pixels); corydoras and zebrafish far less (25% and 18%). The corydora spots are fish resting on the bottom.

The ten longest tracks of each species drawn over the empty tank: corydora tracks wander slowly with tight knots where the fish rested, molly and guppy tracks run along the walls, and cebrita tracks cut straight across the open water.

The ten longest tracks of each species. One corydora kept the same number for 2 minutes 19 seconds. Zebrafish are the fastest, a median 169 pixels a second against 70 for corydoras, and their longest tracks last 12 to 18 seconds.

Shannon diversity over the same run, in five-second windows: after a dip to about 1.09 in the first minute it stays between about 1.20 and 1.29, around the dashed line of the real stock at 1.25.

Shannon diversity (H′) in five-second windows, against the real stock's 1.25. The measured median is 1.24, and 87% of windows are within 0.05.

From footage to spreadsheet.

A video goes in, and out comes everything above, in plain files.

VideoOne file, a batch of files or the live camera.
DetectEvery fish, with its species and its confidence.
TrackOne number per fish for as long as it can be followed.
CountTotal and per species, stabilised.
CSVOne row per frame and one per detected fish.

summary.csvone row per frame

frame	count	stabilized_count	mean_conf	species_counts
14040	46	48	0.826	cebrita:7\|corydora:21\|guppy:4\|molly:14

tracks.csvone row per detected fish, two of each species

frame	track_id	species	x	y	w	h	conf
14040	17637	corydora	492.8	803.0	100.8	118.0	0.772
14040	18283	corydora	749.4	525.7	189.2	74.8	0.903
14040	18591	molly	1145.0	789.6	123.5	66.0	0.933
14040	16577	molly	495.1	135.3	134.8	46.3	0.82
14040	18999	guppy	1465.4	841.5	99.0	137.3	0.963
14040	19070	guppy	1470.1	265.5	71.2	109.3	0.557
14040	18900	cebrita	810.9	630.1	111.8	104.7	0.907
14040	18910	cebrita	1234.8	353.8	41.7	157.0	0.758

Real rows from the same run: frame 14,040, at 3:54, inside the eight seconds of the video above. summary.csv's time and active-number columns are left out for space. Fish 17637 and 18283 already had those numbers four seconds earlier.

With the live camera, someone else can watch the processed tank from their own browser.

Per frame: Time, frame number, count, stabilised count, active track numbers, mean confidence and count per species.
Per fish: Frame, track number, species, position, box width and height, and confidence.
Charts: Ten from any run: count, heatmap, trajectories, speed, residence time, direction, diversity and confidence, among others.
Validation: You mark what is in the tank by hand (or bring MOT, YOLO or COCO annotations) and FishAI is scored against it: precision and recall, MOTA, MOTP, IDF1 and ID switches, with a confusion matrix, Bland-Altman and calibration.
API: A REST API for cameras, detections, tracks, statistics and export.
Your species: Today it tells corydora, molly, guppy and zebrafish apart. For others it is trained on your footage: the machine proposes the boxes and a person corrects the species and the mistakes before a frame goes into training.
Speed: On an ordinary graphics card it processes 1080p video at about 34 frames a second, with 48 fish in the tank.

What it does not do yet.

So you know what to measure with FishAI and what not to.

Identity: In a crowded tank a fish gets a new number when it crosses another or is hidden: this run used 17,618 numbers for 48 fish. It serves for tracks of seconds to minutes, not for following one individual through a whole session.
Units: Positions and speeds are in pixels. There is no calibration to centimetres yet.
Depth: One camera from above: it sees the plan of the tank, not how high each fish swims.

Do you study fish?

Tell us which species you keep, how you film and what you want to measure. Send us a video and we will send back its data.

Write to us WhatsApp

← Back to home