This paper describes a task-based evaluation methodology appropriate for dialogue systems such as the TRAINS-95 system, where a human and a computer interact and collaborate to solve a given problem. In task-based evaluations, techniques are measured in terms of their effect on task performance measures such as how long it takes to develop a solution using the system, and the quality of the final plan produced. We report recent experiment results which explore the effect of word recognition accuracy on task performance.
Funding was gratefully received from NSF under Grant IRI-90-13160 and from ONR/DARPA under Grant N00014-92-J-1512. Many thanks to George Ferguson for developing the on-line tutorial, Eric Ringger for compiling the word recognition accuracy figures, Amon Seagull for advice on statistical measures, and Peter Heeman for numerous helpful comments. Thanks also to Mike Tanenhaus and Joy Hanna for their suggestions on the experimental design.