Eye movement and strategic placement of the visual field onto the retina, gives
animals increased resolution of the scene and suppresses distracting
information. This fundamental system has been missing from video understanding
with deep networks, typically limited to 224 by 224 pixel content locked to the
camera frame. We propose a simple idea, WorldFeatures, where each feature at
every layer has a spatial transformation, and the feature map is only
transformed as needed. We show that a network built with these WorldFeatures,
can be used to model eye movements, such as saccades, fixation, and smooth
pursuit, even in a batch setting on pre-recorded video. That is, the network
can for example use all 224 by 224 pixels to look at a small detail one moment,
and the whole scene the next. We show that typical building blocks, such as
convolutions and pooling, can be adapted to support WorldFeatures using
available tools. Experiments are presented on the Charades, Olympic Sports, and
Caltech-UCSD Birds-200-2011 datasets, exploring action recognition,
fine-grained recognition, and video stabilization.