Judgments made by the assessors observing consultations are widely used in the assessment of medical students. The aim of this research was to study judgment accuracy and confidence and the relationship between these. Assessors watched recordings of consultations, scoring the students on: a checklist of items; attributes of consultation; a passmark scale and lastly their confidence in this last judgment. Then they were interviewed using stimulated recall to explain their rationale for scoring and confidence. Twenty-three staff assessors watched two consultations. Assessor confidence and accuracy were least for those student performances perceived near the passmark standard. The difference between confidence and accuracy, over-confidence, was greatest at this level, although at the extremes under-confidence was found. In the interviews the assessors were aware of a variety of factors that they perceived affected their scoring and confidence. As in other contexts confidence and accuracy vary and over-confidence increases the more difficult the judgments. However, this study also demonstrated under-confidence for less difficult judgments. The perception that more information would help, may improve confidence but not necessarily accuracy, so increasing over-confidence.