A computational framework for recognizing an object of interest in a complex visual environment is described. Arising from the problem of finding the destination address block on a mail piece, a general framework for coordinating a collection of specialized image-analysis tools is described. The resulting system is capable of dealing with a wide range of environments—from those having a high degree of global spatial structure (e.g., letter mail envelopes that conform to specifications) to those with no structure (e.g., magazines with randomly pasted address labels). The problem-solving architecture accounts for uncertainty in the imaging environment by using the blackboard model. This paper discusses systematic derivation of a set of object recognition heuristics (knowledge base), specialized image analysis tools for extracting those features that are called for by the heuristics, and a control structure for integrating evidence and managing tools. Experimental results with a database of difficult cases demonstrating the promise of the methodology are presented.