This article investigates systematically the utility of performance measures in non-stationary environments. Three characteristics for describing the goals of a dynamic adaptation process are proposed: accuracy, stability, and recovery. This examination underpins the usage of the best fitness value as a basis for measuring the three characteristics in scenarios with moderate changes of the best fitness value. However, for dynamic problems without coordinate transformations all considered fitness based measures exhibit severe problems. In case of the recovery, a newly proposed window based performance measure is shown to be best as long as the accuracy level of the optimization is rather high.