The composition of training samples set is an important influence factor of spectral reflectance reconstruction process. Representative color samples selection for learning-based spectral reflectance reconstruction is discussed in this paper. A method based on Principal Component Analysis (PCA) is proposed to perform sample selection. First of all, a part of samples are selected according to the minimum Euclidean distance criteria in terms of camera response value from a large number of samples, which aim to ensure the similarity between training samples and target samples. Then the PCA data processing method is applied to these samples after removing the duplicate samples. The samples with larger principal component loadings are regarded as the representative color samples. Different thresholds for each principal component are used to make decision whether the loading of sample is large enough. In order to validate the proposed method, the selected samples are used as training samples to recover the spectral reflectance of color patches. A real multi-channel imaging system by loading broadband color filters in front of lens is used in the experiment to acquire the multi-channel image dataset. In this paper the pseudo-inverse method is employed to reconstruct spectral reflectance of target color patches. It is shown that the proposed method is superior to the previous methods in spectral reconstruction accuracy and can meet the requirements of high precision color reproduction.