In this paper, we present a combined classification approach called the ‘virtual test sample method’. Contrary to classifier combination, where the outputs of a number of classifiers are used to come to a combined decision for a given observation, we use multiple instances generated from the original observation and a single classifier to compute a combined decision. In our experiments, the virtual test sample method is used to improve the performance of a statistical classifier based on Gaussian mixture densities. We show that this approach has some desirable theoretical properties and performs very well, especially when combined with the use of invariant distance measures. In the experiments conducted throughout this work, we obtained an excellent error rate of 2.2% on the original US Postal Service task.