This paper describes the design and implementation of an active surface reconstruction algorithm for two-frame image sequences using passive imaging. A novel strategy based on the statistical grouping of image gradient features is used. It is shown that the gradient of the intensity in an image can successfully be used to drive the direction of the viewer’s motion. As such, an increased efficiency in the accumulation of information is demonstrated through a significant increase in the convergence rate of the depth estimator (3 to 4 times for the presented results) over traditional passive depth-from-motion. The viewer is considered to be restricted to a short baseline. A maximal-estimation framework is adopted to provide a simple approach for propagating information in a bottom-up fashion in the system. A Kalman filtering scheme is used for accumulating information temporally. The paper provides results for real-textured data to support the findings.