Real world manipulation tasks vary in their demands for precision and freedoms controlled. In particular, during any one task the complexity may vary with time. For a robotic hand-eye system, precise tracking and control of full pose is computationally expensive and less robust than rough tracking of a subset of the pose parameters (e.g. just translation). We present an integrated vision and control system in which the vision component provides (1) the continuous, local feedback at the required complexity for robot manipulation and (2) the discrete state information needed to switch between control modes of differing complexity.