In this paper, a novel face recognition system is presented in which not only a classifier but also a feature space is learned incrementally to adapt to a chunk of incoming training samples. A distinctive feature of the proposed system is that the selection of useful features and the learning of an optimal decision boundary are conducted in an online fashion. In the proposed system, Chunk Incremental Principal Component Analysis (CIPCA) and Resource Allocating Network with Long-Term Memory are effectively combined. In the experiments, the proposed face recognition system is evaluated for a self-compiled face image database. The experimental results demonstrate that the test performance of the proposed system is consistently improved over the learning stages, and that the learning speed of a feature space is greatly enhanced by CIPCA.