In this paper, we present a new multiple regions and their spatial relationship-based image retrieval method. In this method, a semantic object is integrated as a set of related regions based on their spatial relationships and visual features. In contrast to other ROI (Region-of-Interest) or multiple region-based algorithms, we use the Hausdorff Distance (HD) to estimate spatial relationships between regions. By our proposed HD, we can simplify matching process between complex spatial relationships and admit spatial variations of regions, such as translation, rotation, insertion, and deletion. Furthermore, to solve the weight adjust problem automatically and to reflect user’s perceptual subjectivity to the system, we incorporate relevance feedback mechanism into our similarity measure process.