Block motion estimation can be regarded as a function minimization problem in a finite-dimensional space. Therefore, fast block motion estimation can be achieved by using an efficient function minimization algorithm instead of a predefined search pattern, such as diamond search. Downhill simplex search is an efficient derivative-free function minimization algorithm. In this paper, we proposed a fast block motion estimation algorithm based on applying the downhill simplex search for function minimization. Several enhanced schemes are proposed to improve the efficiency and accuracy, including a new initialization process, a special rounding scheme, and an early-stop error function evaluation procedure. We also extend the downhill simplex search for the multi-reference frame motion estimation problem. Experimental results show superior performance of the proposed algorithm over some existing fast block matching methods on several benchmarking video sequences.