Guess where? Actor-supervision for spatiotemporal action localization

Victor Escorcia, Cuong D. Dao, Mihir Jain, Bernard Ghanem, Cees Snoek

Research output: Contribution to journalArticlepeer-review

13 Scopus citations


This paper addresses the problem of spatiotemporal localization of actions in videos. Compared to leading approaches, which all learn to localize based on carefully annotated boxes on training video frames, we adhere to a solution only requiring video class labels. We introduce an actor-supervised architecture that exploits the inherent compositionality of actions in terms of actor transformations, to localize actions. We make two contributions. First, we propose actor proposals derived from a detector for human and non-human actors intended for images, which are linked over time by Siamese similarity matching to account for actor deformations. Second, we propose an actor-based attention mechanism enabling localization from action class labels and actor proposals. It exploits a new actor pooling operation and is end-to-end trainable. Experiments on four action datasets show actor supervision is state-of-the-art for action localization from video class labels and is even competitive to some box-supervised alternatives.
Original languageEnglish (US)
Pages (from-to)102886
JournalComputer Vision and Image Understanding
StatePublished - Dec 9 2019


Dive into the research topics of 'Guess where? Actor-supervision for spatiotemporal action localization'. Together they form a unique fingerprint.

Cite this