Draft

Tennis tracker

Broadcast tennis video goes in. The ball, both players and the 14 court keypoints come out, plus a top-down minimap and markers for racket hits. One small network does all three tasks.

Model: TennisNet-lite, 3.45M parameters · about 5 ms per frame (≈200 fps) on an M2 Max laptop

Result on a match the model never saw

Test match game10. Left: detections on the video. Right: the top-down minimap.

What the project does

The interesting part is how we got the training data. The public dataset (TrackNet) labels only the ball. We made the player and court labels automatically, on a laptop, with two large models. Then a small, fast model learns all three tasks from these labels.

This is a form of distillation: large, slow models make the labels one time. A small, fast model learns to copy them.

Architecture

frame t frame t−1 frame t−2 grayscale, 512 × 288stacked as 3 channels TennisNet-lite · 3.45M params ResNet-18stem + layers 1–3(layer 4 removed) U-Net decoderskip connectionsoutput at 1/2 size 1 ball heatmap256 × 144 14 court heatmapsone per keypoint 1 player heatmap+ box width, height peak → ball (x, y)trail, racket-hit rule peaks → homographycourt lines + minimap 2 peaks → 2 boxesfeet onto the minimap one forward pass gives 16 heatmaps + player sizes; simple rules turn them into points and boxes

Key ideas

  • Heatmaps, not boxes, for small objects. The ball is only a few pixels. The model does not predict coordinates. It draws a heatmap with a bright spot on the ball. Finding the peak of a heatmap is much easier to learn than direct regression of the numbers. Players are also heatmaps: a peak at the box center, plus the box width and height at that peak (as in CenterNet).
  • Time as channels. In one frame, the ball looks like any white dot. In three frames, the ball moves and the dots do not. So we put 3 grayscale frames into the 3 input channels of a normal image network.
  • A smaller ResNet-18. Layer 4 of ResNet-18 has 8M of its 11M weights. Heatmaps need fine detail, not the deepest features, so we remove layer 4. A U-Net decoder brings the features back to half the input size.
  • Homography for the minimap. The 14 court points match the 14 points of a standard court diagram. These pairs give a 3 × 3 matrix that maps any ground pixel onto the minimap. We map each player by the bottom of the box (the feet), because only points on the ground map correctly.
  • Racket hit. A hit is a frame where the vertical direction of the ball changes while the ball is near a player.

Training

Step 1: make labels with large models

TrackNet datasetframes + human ball labels Grounding DINO tinyprompt: "a person." TennisCourtDetector14 keypoints per frame keep the 2 playersdrop judges, crowd, edges one homography per clipRANSAC, camera does not move tennis.jsonl19,835 frames, 3 tasks TennisNet-litelearns all three
The large models run one time to make labels. Only the small model runs on video.
  1. Players. Grounding DINO finds every person in every frame. A filter keeps the two players: it removes boxes that never move (line judges, crowd), boxes at the edge of the image and boxes with low scores.
  2. Court. A pretrained court detector finds 14 keypoints per frame. The camera does not move inside a clip, so we fit one homography for each clip with RANSAC. RANSAC ignores the frames where the detector is wrong.
  3. Difficult camera. One match (game4) has a low camera, and the court detector fails on it. We warp each frame to look like a normal broadcast view, detect the points, and warp them back.
  4. Merge. Ball, player and court labels go into one file. We split by match, never by frame: game9 is validation, game10 is test.

Step 2: train the small model

SettingValue
Input3 consecutive grayscale frames, 512 × 288
TargetsGaussian heatmaps: 1 ball, 14 court points, 1 player center; plus player width and height
LossCenterNet focal loss on every heatmap, L1 on player sizes. Court loss only where the court label is reliable.
AugmentationFlip, rotate, scale, brightness, blur
Hard framesFrames near a racket hit, or with a ball that is hard to see, are shown 3 times more often
OptimizerAdamW, learning rate 1e-3, one-cycle schedule, batch 16
LengthPlanned for 20 epochs. We stopped at epoch 10 because the test score stopped improving. About 6 minutes per epoch on an M2 Max.
bash tennis/setup.sh                          # dataset + court detector
bash label_all.sh                             # auto-label players + court (1-2 h)
.venv/bin/python lite/train.py                # train
.venv/bin/python lite/evaluate.py             # results table
.venv/bin/python lite/demo.py --clip game10/Clip1

Results on the test match (game10)

BallValue
Precision / recall / F10.997 / 0.876 / 0.932
Median position error1.6 px (in 1280 × 720)
Clearly visible ball found91%
Faint or blurred ball found57%
Ball at a racket hit found56%
Far half / near half of the court84% / 93%

On the validation match: court keypoints have a median error of about 1.3 px, and more than 99% of players are found (IoU above 0.5). Racket-hit markers have about 90% precision and 54% recall, and they come 4 frames late.

What fails. The model almost never sees a ball that is not there. It fails when it misses a small, faint ball: far from the camera or at the racket. Showing hard frames more often helped a little at hits (48% → 56%), but not for faint and far balls. A far ball is about 2 pixels on the 256 × 144 heatmap, so higher input resolution is the next step to try.
A short clip from the test match.

Papers and links