Interactive Acquisition and Rendering of Humans

Abstract

We present a distributed model-based system for interactive acquisition and rendering of free viewpoint video of humans. All current model-based systems are offline. Online systems which can be used for free viewpoint video of humans use few low resolution cameras or use only silhouette information while the method presented here can cope with high resolution images and uses both color and silhouette information. A person is captured with multiple synchronized and calibrated digital video cameras attached to a cluster of pcs. At interactive rates, the shape of the person is estimated and the person can be rendered from novel viewpoints. First the pose of the person is obtained using a real-time motion capturing algorithm. A generic mesh of a human is transformed into this pose to obtain a first approximation of the shape of the recorded person. This mesh is refined to match the input images as closely as possible by translating its vertices along their normals. The refined mesh is rendered using a distributed version of the unstructured lumigraph algorithm.