Tennis tracker
Broadcast tennis video goes in. The ball, both players and the 14 court keypoints come out, plus a top-down minimap and markers for racket hits. One small network does all three tasks.
Model: TennisNet-lite, 3.45M parameters · about 5 ms per frame (≈200 fps) on an M2 Max laptop
Result on a match the model never saw
Test match game10. Left: detections on the video. Right: the top-down minimap.
What the project does
The interesting part is how we got the training data. The public dataset (TrackNet) labels only the ball. We made the player and court labels automatically, on a laptop, with two large models. Then a small, fast model learns all three tasks from these labels.
This is a form of distillation: large, slow models make the labels one time. A small, fast model learns to copy them.
Architecture
Key ideas
- Heatmaps, not boxes, for small objects. The ball is only a few pixels. The model does not predict coordinates. It draws a heatmap with a bright spot on the ball. Finding the peak of a heatmap is much easier to learn than direct regression of the numbers. Players are also heatmaps: a peak at the box center, plus the box width and height at that peak (as in CenterNet).
- Time as channels. In one frame, the ball looks like any white dot. In three frames, the ball moves and the dots do not. So we put 3 grayscale frames into the 3 input channels of a normal image network.
- A smaller ResNet-18. Layer 4 of ResNet-18 has 8M of its 11M weights. Heatmaps need fine detail, not the deepest features, so we remove layer 4. A U-Net decoder brings the features back to half the input size.
- Homography for the minimap. The 14 court points match the 14 points of a standard court diagram. These pairs give a 3 × 3 matrix that maps any ground pixel onto the minimap. We map each player by the bottom of the box (the feet), because only points on the ground map correctly.
- Racket hit. A hit is a frame where the vertical direction of the ball changes while the ball is near a player.
Training
Step 1: make labels with large models
- Players. Grounding DINO finds every person in every frame. A filter keeps the two players: it removes boxes that never move (line judges, crowd), boxes at the edge of the image and boxes with low scores.
- Court. A pretrained court detector finds 14 keypoints per frame. The camera does not move inside a clip, so we fit one homography for each clip with RANSAC. RANSAC ignores the frames where the detector is wrong.
- Difficult camera. One match (
game4) has a low camera, and the court detector fails on it. We warp each frame to look like a normal broadcast view, detect the points, and warp them back. - Merge. Ball, player and court labels go into one file. We split by match, never by frame:
game9is validation,game10is test.
Step 2: train the small model
| Setting | Value |
|---|---|
| Input | 3 consecutive grayscale frames, 512 × 288 |
| Targets | Gaussian heatmaps: 1 ball, 14 court points, 1 player center; plus player width and height |
| Loss | CenterNet focal loss on every heatmap, L1 on player sizes. Court loss only where the court label is reliable. |
| Augmentation | Flip, rotate, scale, brightness, blur |
| Hard frames | Frames near a racket hit, or with a ball that is hard to see, are shown 3 times more often |
| Optimizer | AdamW, learning rate 1e-3, one-cycle schedule, batch 16 |
| Length | Planned for 20 epochs. We stopped at epoch 10 because the test score stopped improving. About 6 minutes per epoch on an M2 Max. |
bash tennis/setup.sh # dataset + court detector bash label_all.sh # auto-label players + court (1-2 h) .venv/bin/python lite/train.py # train .venv/bin/python lite/evaluate.py # results table .venv/bin/python lite/demo.py --clip game10/Clip1
Results on the test match (game10)
| Ball | Value |
|---|---|
| Precision / recall / F1 | 0.997 / 0.876 / 0.932 |
| Median position error | 1.6 px (in 1280 × 720) |
| Clearly visible ball found | 91% |
| Faint or blurred ball found | 57% |
| Ball at a racket hit found | 56% |
| Far half / near half of the court | 84% / 93% |
On the validation match: court keypoints have a median error of about 1.3 px, and more than 99% of players are found (IoU above 0.5). Racket-hit markers have about 90% precision and 54% recall, and they come 4 frames late.
Papers and links
- TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications (Huang et al., 2019). The ball dataset, and the idea of heatmaps from stacked frames.
- Objects as Points (CenterNet) (Zhou et al., 2019). Heatmap centers + sizes, and the focal loss we use.
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (Liu et al., 2023). Player labels from a text prompt.
- Deep Residual Learning for Image Recognition (ResNet) (He et al., 2015) and U-Net (Ronneberger et al., 2015). The encoder and the decoder.
- Distilling the Knowledge in a Neural Network (Hinton et al., 2015). Large models teach a small model.
- TennisCourtDetector and TrackNet (PyTorch) by yastrebksv. Court labels and the dataset download.