In this paper, we propose an approach based on HMM and linguistics for the Vietnamese recognition problem, including handwritten and speech recognition. The main contribution is that our method could be used to model all Vietnamese isolated words by a small number of HMMs. The method is not only used for handwritten recognition but also for speech recognition. Furthermore, it could be integrated with language models to improve the accuracy. Experimental results show that our approach is robust and considerable.