As large-scale, detailed network modeling projects are flourishing in the field of computational neuroscience, it is more and more important to design single neuron models that not only capture qualitative features of real neurons but are quantitatively accurate in silico representations of those. Recent years have seen substantial effort being put in the development of algorithms for the systematic evaluation and optimization of neuron models with respect to electrophysiological data. It is however difficult to compare these methods because of the lack of appropriate benchmark tests. Here, we describe one such effort of providing the community with a standardized set of tests to quantify the performances of single neuron models. Our effort takes the form of a yearly challenge similar to the ones which have been present in the machine learning community for some time. This paper gives an account of the first two challenges which took place in 2007 and 2008 and discusses future directions. The results of the competition suggest that best performance on data obtained from single or double electrode current or conductance injection is achieved by models that combine features of standard leaky integrate-and-fire models with a second variable reflecting adaptation, refractoriness, or a dynamic threshold.