In this paper we present a method of Multi-branch two dimensional HMM (hidden markov model ) for handwritten numeral recognition and another method of Neural network for handwrittern numeral recognition and then integrated the two method into a totally unconstrained handwritten numeral recognition system. The system is composed of a horizontal super state multi-branch two dimensional HMM, a vertical super state multi-branch two dimensional HMM and a Neural network. The integrated recognition system recognition rate is higher than the three subsystem. The data base of handwriting digits used in this paper was collected at Beijing Postal Center, the digits was scanned from letters zip code, altogether 4000 hand writing digits, 2000 are used for training set, 2000 are used for testing set.The training set recognition rate is 99.85%, the testing set recognition rate is 98.05%. If we used more complex decision strategy the system performance will be better.