There are some problems in the simple optical system, such as blurring and distortion of the image due to the serious aberration. The imaging characteristics of the simple optical system are analyzed. A point spread function model with aberration and an image restoration model are established with the prior knowledge that the focal length and pupil aperture of the optical system are known. Only the focal plane and defocus surface images and the precise defocus amount are required, the image obtained by the simple optical system is reconstructed by the phase diversity method without the prior information of the point spread function. The experimental results show that the image quality evaluation index improvement rates of single lens, double lens and three lens system are 24.96%, 24.80% and 42.04%, respectively, and the restored image quality has been improved obviously.