Online Multi-modal Person Search in Videos
European Conference on Computer Vision (ECCV) 2020
Overview
The task of searching certain people in videos has seen increasing potential in real-world applications,
such as video organization and editing. Most existing approaches are devised to work in an offline manner,
where identifies can only be inferred after an entire video is examined. This working manner precludes such methods from
being applied to online services or those applications that require real-time responses.
In this paper, we propose an online person search framework, which can recognize people in a video on the fly.
This framework maintains a multi-modal memory bank at its heart as the basis for person recognition, and updates it dynamically
with a policy obtained by reinforcement learning. Our experiments on a large movie dataset show that the proposed method is effective,
not only achieving remarkable improvements over strong online schemes but also outperforming offline methods.
Online Multi-modal Searching Machine
Given the portraits of a list of casts, our goal is to search them in a sequential movie with an online fashion following the human behaviors. To tackle this
challenging problem, we propose a novel online multi-modal searching machine
(OMS).
There are four key components in OMS, i.e. multimodal feature representations (MFR), a dynamic memory bank (DMB), an uncertain instance cache (UIC) and a controller. Each instance, represented by multi-modal features, is compared with the exemplars stored in the memory bank to judge its identity. The controller then determines whether this instance should be used to update memory or put into the uncertain instance cache for later comparisons. The memory bank and the uncertain instance cache are dynamically updated over time, with a strategy operated by the controller. All these components together build an “intelligent machine” to watch a movie and gradually recognize the characters like humans do.
There are four key components in OMS, i.e. multimodal feature representations (MFR), a dynamic memory bank (DMB), an uncertain instance cache (UIC) and a controller. Each instance, represented by multi-modal features, is compared with the exemplars stored in the memory bank to judge its identity. The controller then determines whether this instance should be used to update memory or put into the uncertain instance cache for later comparisons. The memory bank and the uncertain instance cache are dynamically updated over time, with a strategy operated by the controller. All these components together build an “intelligent machine” to watch a movie and gradually recognize the characters like humans do.
Citation
@inproceedings{xia2020online,
title={Online Multi-modal Person Search in Videos},
author={Xia, Jiangyue and Rao, Anyi and Xu, Linning and Huang, Qingqiu and Wen, Jiangtao and Lin, Dahua},
booktitle = {The European Conference on Computer Vision (ECCV)},
year={2020}
}