Non-rigid image registration (NIR) is an essential tool for morphologic comparisons in the presence of intra- and inter-individual anatomic variations. Many NIR methods have been developed, but are especially difficult to evaluate since point-wise inter-image correspondence is usually unknown, i.e., there is no “Gold Standard” to evaluate performance. The Non-rigid Image Registration Evaluation Project (NIREP) has been started to develop, establish, maintain, and endorse a standardized set of relevant benchmarks and metrics for performance evaluation of nonrigid image registration algorithms. This paper describes the basic framework of the project.