The edge and motion are the main features that human visual system (HVS) perceives intensively. This paper proposes an algorithm for the segmentation of the moving object with accurate boundary using color and motion focusing on the HVS perception in the general image sequence. The proposed algorithm is composed of three parts: color segmentation, motion analysis, and region refinement and merging part. In the color segmentation phase, K-Means algorithm is used in consideration of the sensitivity of the human color perception to get the boundaries that HVS perceives. The global and local motion estimation are performed in parallel with color analysis. After that, Bayesian clustering using color and motion provides more accurate boundary. In the final stage, regions are merged taking into account their motion. The experimental results of the proposed algorithm show the accurate moving object boundary coinciding with the boundary that HVS perceives.