Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

Researchers introduced Prompting-MammAlps, the first camera-trap text-to-video retrieval benchmark, and a fine-grained method that uses a vision transformer for spatiotemporal action localization and an LLM-based coding agent to parse structured text. Their approach achieved a 34% F1-score on 135 queries and 775 videos, outperforming the best zero-shot VLM's 18% while offering interpretability.

arXiv:2607.09876v1 Announce Type: new Abstract: Automatically retrieving videos from large camera-trap datasets remains challenging. Text-to-Video retrieval TVR methods based on large video-language models VLMs have potential to retrieve events of interest by describing them with simple text queries. However, current methods often lack spatiotemporal understanding and do not generalize well to ecological data. In this work, we introduce Prompting-MammAlps, the first camera-trap TVR benchmark, and propose a fine-grained and interpretable TVR method. Specifically, we trained a vision transformer to perform spatiotemporal action localization, and convert its output to structured text, describing each video. Independently, ethology-inspired queries are processed by a Large-Language Model LLM based coding agent to parse the structured text per video and retrieve videos accordingly. We harnessed the LLM to use functions from a custom parsing library to minimize the risk of LLM hallucinations and to improve method interpretability. This retrieval approach applied on the Prompting-MammAlps benchmark achieved a set-based F1-score of 34\% on a test set of 135 ecologically-relevant queries and 775 candidate videos. In comparison the best zero-shot VLM achieved a F1-score of 18\%, while also lacking interpretability. Project page: https://cnai.epfl.ch/prompting-mammalps