Evaluation of retrieval performance is a crucial problem in exhaustive web-based image retrieval. The challenge is to find reliable techniques for estimating the full set of matching answers. In this paper, we present an automatic image-gathering method based on keyword-based web search engines. User’s interaction is involved to build an initial reference image set and verify the result images. By filtering images based on the Earth Mover’s Distance on color feature and combining the results from several search engines, our approach achieves high accuracy in collecting images relevant to user’s information need. We have compared the result of our approach to the performance of three commercial web image search engines.