Image classification could be treated as an effective solution to enable keyword-based semantic image retrieval. In this paper, we propose a novel image classification framework by learning semantic concepts of image categories. To choose representative features for an image category and meanwhile reduce noisy features, a three-step salient feature selection strategy is proposed. In the feature selection stage, salient patches are first detected and clustered. Then the region of dominance and salient entropy measures are calculated to reduce non-common salient patches for the category. Based on the selected visual keywords, SVM and keyword frequency model categorization method are applied to classification, respectively. The experimental results on Corel image database demonstrate that the proposed salient feature selection approach is very effective in image classification and visual concept learning.