In this paper, a computer interface for handicapped people is proposed, where input signals are given by eye movement of the handicapped people. Eye movement is detected by neural network (NN)-based texture classifier, which enables our system to be not obliged to constrained environment. To be robust the natural motion of a user, we first detect a user’s face using skin-color information, and then detect her or his eyes using neural network (NN)-based texture classifier. After detection of eye movements, the tracking is performed using mean-shift algorithms. We use this eye-tracking system as an interface to control the surrounding system such as audio, TV, light, phone, and so on. The experimental results verify the feasibility and validity of the proposed eye-tracking system to be applicable as an interface for the handicapped people.