This paper describes an aspect of a developing system named ASSAVID which will provide an automatic and semantic annotation of sports video. This annotation process segments the sports video into semantic categories (e.g. type of sport) and permits the user to formulate queries to retrieve events that are significant to that particular sport (e.g. goal, foul). The system relies upon the concept of “cues” which attach semantic meaning to low-level features computed on the video. In this paper we adopt the multiple classifier system approach to fusing the outputs of multiple cue detectors using Behaviour Knowledge Space fusion. Using this technique, unknown sports video can be classified into the type of sport being played. Experimental results on sports video provided by the BBC demonstrate that this method is working well.