Motion information is the basic element for analyzing video. It represents the change of video on the time-axis and plays an important role in describing the video content. In this paper, a robust motion-based, video retrieval system is proposed. At first, shot boundary detection is achieved by analyzing luminance information, and motion information of video is abstracted and analyzed. Then rough set theory is introduced to classify the shots into two classes, global motions and local motions. Finally, shots of these two types are respectively retrieved according to the motion types of submitted shots. Experiments show that it’s effective to distinguish shots with global motions from those with local motions in various types of video, and in this situation motion-information-based video retrieval are more accurate.