TCAM: Track and Caption Any Motion

Query-free motion discovery and description in videos. · Accepted at WACV 2027

Resource	Link
Paper (arXiv)	https://arxiv.org/abs/2512.10607 · PDF
Project page	https://ostadabbas.github.io/TCAM.io/
Code	https://github.com/ostadabbas/TCAM-Track-and-Caption-Any-Motion
Model (this repo)	https://huggingface.co/bishoygaloaa/tcam

Same drawer, opposite motions. TCAM tracks what moves and tells them apart: opening (orange, out) vs. closing (blue, in).

Give TCAM a video and it tells you what moves, when, and where: timestamped captions for each motion event, each grounded to the point tracks of the subject performing it. No query needed.

Quick start

git lfs install
git clone https://huggingface.co/bishoygaloaa/tcam && cd tcam

pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

python infer.py --video my_clip.mp4 --vis

=== my_clip.mp4
  [  0.00s -   4.20s] A boy is skating on a zebra crossing.
  [  4.68s -   7.52s] He is skating on a zebra crossing.
  -> tcam_out/my_clip.json

--vis also writes tcam_out/my_clip_vis.mp4, which overlays each event's grounded points on the video. The TCAM weights download automatically, and so does CoTracker3, from Meta's official facebook/cotracker3 repo.

Useful flags: --chunk_sec 10 for long videos, --start_sec/--end_sec for a segment, --sample --seed 1 for more varied event windows, --model <local dir> to run offline.

Python

from tcam import TCAMPipeline

pipe = TCAMPipeline.from_pretrained("bishoygaloaa/tcam")
res = pipe("my_clip.mp4")

for e in res.events:
    print(e.start_sec, e.end_sec, e.caption)
    xy = res.tracks[:, e.top_tracks(40)]   # grounded points, (16 frames, 40, 2), normalised

mask = res.masks(event=0)                  # (16, 224, 224) subject mask
res.save_json("my_clip.json")

Fine-tuning

Describe your videos in a train.json (times in seconds; box is optional and normalised to [0, 1]. Add it to also train grounding):

[{"video": "clips/drawer.mp4",
  "events": [{"start": 0.0, "end": 3.5, "caption": "a hand opens the drawer", "box": [0.2, 0.3, 0.8, 0.7]},
             {"start": 3.5, "end": 7.0, "caption": "the hand closes the drawer"}]}]

python finetune.py --data train.json --out_dir my_tcam --steps 1000
python infer.py --model my_tcam --video new_clip.mp4

This uses the same objective as the original training. Keep --lr low (default 2e-5).

Notes

• TCAM reads 16 frames per call. Use --chunk_sec for anything longer than a short clip.
• Grounding selects the subject's tracks; masks are drawn from those tracks.
• Tested on a CUDA GPU with Python 3.10. Keep the pinned versions in requirements.txt.

License

Weights and code: CC BY-NC 4.0 (non-commercial). FLAN-T5, CLIP and CoTracker3 keep their own licenses.

Citation

@article{galoaa2025tcam,
  title   = {Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation},
  author  = {Galoaa, Bishoy and Ostadabbas, Sarah},
  journal = {arXiv preprint arXiv:2512.10607},
  year    = {2025},
  url     = {https://arxiv.org/abs/2512.10607}
}

@inproceedings{galoaa2027tcam,
  title={Track and Caption Any Motion: Query-Free Motion Discovery and Description in Videos},
  author={Galoaa, Bishoy and Ostadabbas, Sarah},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2027}
}