Clustering appearance and shape by
learning jigsaws
J. Winn, A. Kannan, C. Rother
Appearance models of fixed size image patches are widely used for object recognition. Such models have problems when the object part is not the same shape as the patch, due to the variation in background appearance. We present the learned jigsaw model, a generative image model where the shape, size and appearance of patches are learned automatically from repeated structures in the training images. By learning such irregularly shaped 'jigsaw pieces', we are able to discover both the shape and the appearance of object parts in images without supervision. When applied to face images, for example, the learned jigsaw pieces are surprisingly strongly associated with face parts of different shapes and scales such as eyes, noses, eyebrows, cheeks and so on. We conclude that learning the shape of the patch not only improves the accuracy of appearance-based part detection but also allows for shape-based part detection. This enables parts of similar appearance but different shapes to be distinguished; for instance, foreheads and cheeks which are both skin colored but have markedly different shapes. Jigsaws are also robust when object parts are partially occluded. We will discuss some applications of jigsaws to object recognition.