Spectral reflectance reconstruction, also referred to as spectral characterization, aims to recover accurate spectral reflectance of object surface by employing standard color charts. As there are always a large number of color samples on a color chart, spectral characterization becomes a time-consuming process for practical application. Some methods have been presented to selected representative color samples based on the redundancy of the colors on a chart. However, these methods only consider the distribution of spectral reflectance, and thus the selected colors may not be optimal for a specific imaging system. To deal with this problem, the present paper proposes a sequential method for the selection of most representative colors, which consists of two steps. In the first step, a part of representative colors are selected according to the minimization of mean spectral root-mean-square error, by assuming a virtual imaging system. The spectral responsivity of the real imaging system is then calculated based on these selected samples. In the second step, additional representative colors are selected based on the characteristics of the real imaging system. Two quite different systems, i.e., an 11-channel narrowband multispectral imaging system and a 3-channel broadband color scanner, were used in the experiment. It was shown that the proposed method significantly outperforms the previous method in terms of both spectral and colorimetric accuracy.