In this paper, we present a semantic retrieval and semi-automatic annotation system for movies, based on the regional features of video images. The system uses a 5-dimensional GBD-tree structure to organize the low-level features: the color, area, and minimal bounding rectangle coordinates of each region that is a segment of a key frame. We propose a regionally based “semantic” object retrieval method that compares color, area, and spatial relationships between selected regions to distinguish them from background information. Using this method, movie information can be retrieved for video data containing the same objects based upon object semantics. In addition, a semi-automatic annotation method is proposed for annotating the matched “semantic” objects for further use. A retrieval system has been implemented that includes semantic retrieval and semi-automatic annotation functions.