We propose to use systematic simulation studies as opposed to the use of real-world benchmark datasets to better understand
the behaviour, strengths and weaknesses of machine learning algorithms. Simulated data sets allow much better control and
understanding of the nature of the learning problem than empirical benchmark data sets.
To demonstrate the value of our proposed research methodology, we describe in this paper the results of our studies concerning
the problem of learning multiple classes. We derived the following hypothesis: “Learning classification functions using decision tree learners can be helped by providing additional subclass labels.” To illustrate, for learning a two class problem “car is OK/car needs service” it can be helpful to provide a finer-grained
classification in the training data such as “car OK”, “faulty brakes”, “faulty engine”, “faulty lights”, etc.
This hypothesis was corroborated using a number of ‘real-world’ multi-class data sets from the UCIMLrepository. Our empirical
studies demonstrate the usefulness of the proposed research methodology using artificial data sets as an important methodological
complement to using real-world datasets.