A method for image-based queries and search is proposed which is based on the generation of object outlines in images by using the pen, e.g., on color pen computers. The rationale of the approach is based on a survey on user needs, as well as on considerations from the point of view of pattern recognition and machine learning. By exploiting the actual presence of the human users with their perceptual-motor abilities and by storing textually annotated queries, an incrementally learning image retrieval system can be developed. As an initial test domain, sets of photographs of motor bicycles were used. Classification performances are given for outline and bitmap-derived feature sets, based on nearest-neighbour matching, with promising results. The benefit of the approach will be a user-based multimodal annotation of an image database, yielding a gradual improvement in precision and recall over time.