This paper describes a general framework for the detection and tracking of traffic and road signs from image sequences using only color information. The approach consists of two independent parts. In the first we use a set of Gaussian distributions that model each color for detecting road and traffic signs. In the second part we track the targets detected in the first step over time. Our approach is tested using image sequences with high clutter that contain targets with the presence of rotation and partial occlusion. Experimental results show that the proposed system detects on average 97% of the targets in the scene in near real-time with an average of 2 false detections per sequence.