Chang’E-3 panoramic camera is designed for color imaging, and a color calibration has to be performed before the launch of the detector. The first step when calibrating the camera is to obtain the ground truth color values of the samples. The color values measured by the traditional method have some significant deviations with the actual values and differ at the color difference and human visual perception. When the values are used to perform calibration, the color differences are not ideal, and the images have a negative effect on human perception. In this study, the white reference coordinates adopted in transformation matrix between XYZ and sRGB color space are redefined by using the relative spectral power distribution of D65 calibration light source. After that, based on the CIE color-computing formula and H. Grassmann color mixing law, the transformation matrix to recalculate RGB color values from XYZ color space is modified. In addition, by means of calculating spectral tristimulus values in sRGB with modified matrix, we give analysis to the factors that caused color distortions. The results of the ground experiment showed that the color differences DE76 for cameras A and B after calibration reduced 0.8 and 0.73 respectively compared with those achieved using traditional method. And when the calibrated matrix is used to correct images taken on lunar surface, color difference reduced 26.50% and 34.47% respectively, and the images have a positive effect on human perception.