Previous experiments have shown that human attention is influenced by high level task demands. In this paper, we propose an architecture to estimate the task-relevance of attended locations in a scene.We maintain a task graph and compute relevance of fixations using an ontology that contains a description of real world entities and their relationships. Our model guides attention according to a topographic attention guidance map that encodes the bottom-up salience and task-relevance of all locations in the scene.We have demonstrated that our model detects entities that are salient and relevant to the task even on natural cluttered scenes and arbitrary tasks.