A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Anyi Rao1     Jiaze Wang1     Linning Xu1     Xuekun Jiang2     Qingqiu Huang1     Bolei Zhou1     Dahua Lin1
European Conference on Computer Vision (ECCV) 2020

Overview


Shots are key narrative elements of various videos, e.g. movies and user-generated videos that are thriving over the Internet. The types of shots greatly influence how the underlying ideas, emotions, and messages are expressed. The techniques to analyze shot types is important to the understanding of videos, which has seen increasing demand in real-world applications in this era. Classifying shot type is challenging due to the additional information required beyond the video content, such as the spatial composition of a frame and camera movement. To address these issues, we propose a learning framework Subject Guidance Network (SGNet) for shot type recognition. SGNet separates the subject and background of a shot into two streams, serving as maps to guide scale and movement type classification respectively. To facilitate shot type analysis and model evaluations, we build a large-scale dataset MovieShots, which contains 46K shots from 7K movie trailers with annotations of their scale and movement types. Experiments show that our framework is able to recognize these two attributes of shot accurately, outperforming all the previous methods.

Dataset


Shot scale has five categories: 1) long shot (LS) is taken from a long distance, sometimes as far as a quarter of a mile away; 2) full shot (FS) barely includes the human body in full; 3) medium shot (MS) contains a figure from the knees or waist up; 4) close-up shot (CS) concentrates on a relatively small object, showing the face of the hand of a person; (5) extreme close-up shot (ECS) shows even smaller parts such as the image of an eye or a mouth.
Shot movement has four categories: 1) in static shot, the camera is fixed but the subject is flexible to move; 2) for motion shot, the camera moves or rotates; 3) the camera zooms in for push shot, and 4) zooms out for pull shot. While all the four movement types are widely used in movies, the use of push and pull shots only takes a very small portion. The usage of different shots usually depends on the movie genres and the preferences of the filmmakers.

Prototype


Application: Shot Type Change


Introduction Video





Citation

@inproceedings{rao2020unified,
title={A Unified Framework for Shot Type Classification Based on Subject Centric Lens},
author={Rao, Anyi and Wang, Jiaze and Xu, Linning and Jiang, Xuekun and Huang, Qingqiu and Zhou, Bolei and Lin, Dahua},
booktitle = {The European Conference on Computer Vision (ECCV)}, 
year={2020}
}