Like other cross language tasks, we show that the quality of the translation resource, among other factors, has an effect on retrieval performance. Using data from the ImageCLEF test collection, we investigate the relationship between translation quality and retrieval performance when using Systran, a machine translation (MT) system, as a translation resource. The quality of translation is assessed manually by comparing the original ImageCLEF topics with the output from Systran and rated by assessors based on their semantic content. Quality is also measured using an automatic score derived from the mteval MT evaluation tool, and compared to the manual assessment score. Like other MT tasks, we find that measures based on the automatic score are correlated with the manual assessments for this CLIR task. The results from this short study formed our entry to ImageCLEF 2003.