Traditional stereo algorithms implicitly use the frontal parallel plane assumption when exploiting contextual information, since the smoothness prior biases towards constant disparity (depth) over a neighborhood. For curved surfaces these algorithms introduce systematic errors to the matching process. These errors are non-negligible for detailed geometric modeling of natural objects (e.g. a human face). We propose to use contextual information geometrically. In particular, we perform a differential geometric study of smooth surfaces and argue that geometric contextual information should be encoded in Cartan’s moving frame model over local quadratic approximations of the smooth surfaces. The result enforces geometric consistency for both depth and surface normal. We develop a simple stereo algorithm to illustrate the importance of using such geometric contextual information and demonstrate its power on images of the human face.