A comprehensive methodology for evaluating a user model presents challenges in choosing metrics and in assessing usefulness from both user and system perspectives. In this paper, we describe such a methodology and use it to assess the effectiveness of an adaptive user model embedded in a medical information retrieval. We demonstrate that the user model helps to improve the retrieval quality without degrading the system performance and identify usability problems overlooked in the user model architecture. Empirical data help us in analyzing drawbacks in our user model and develop solutions.