In this paper, efficient and generic tools for calibration and 3D reconstruction are presented. These tools exploit geometric constraints frequently present in man-made environments and allow camera calibration as well as scene structure to be estimated with a small amount of user interactions and little a priori knowledge. The proposed approach is based on primitives that naturally characterize rigidity constraints: parallelepipeds. It has been shown previously that the intrinsic metric characteristics of a parallelepiped are dual to the intrinsic characteristics of a perspective camera. Here, we generalize this idea by taking into account additional redundancies between multiple images of multiple parallelepipeds. We propose a method for the estimation of camera and scene parameters that bears strong similarities with some self-calibration approaches. Taking into account prior knowledge on scene primitives or cameras, leads to simpler equations than for standard self-calibration, and is expected to improve results, as well as to allow structure and motion recovery in situations that are otherwise under-constrained. These principles are illustrated by experimental calibration results and several reconstructions from uncalibrated images.