A novel photometric calibration framework is presented for a projector-camera (ProCam) display system, which is currently under booming development. Firstly, a piecewise bilinear model and five 5-ary color coding images are used to construct the homography between the image planes of a projector and a camera. Secondly, a photometric model is proposed to describe the data flow of the ProCam display system for displaying color images on colored surface in a general way. An efficient self-calibration algorithm is correspondingly put forward to recover the model parameters. Aiming to adapt this algorithm to different types of ProCam display system robustly, a 3\times7 masking coupling matrix and a patches image with 1024 color samples are adopted to fit the complex channel interference function of the display system. Finally, the experimental results demonstrate the validity and superiority of this calibration algorithm for the ProCam display system.