Terahertz passive imaging technology has many unique advantages and has become an important research direction in the field of security inspection. In order to further improve the imaging performance and practicability of the terahertz passive imaging system, the performance of a terahertz passive imaging system based on opto-mechanical scanning developed by our laboratory was studied. By exploring the influence of different system parameters and experimental conditions including its imaging speed, distance and the status of detector on the performance of imaging system such as the image resolution and background noise, the optimized system parameters were obtained. At the same time, the performance of imaging system to detect the human body hiding the metal objects was explored. It was shown that the system was suitable for the security inspection of human body. The experimental results show that the imaging distance of the terahertz passive imaging system is within the range of 1.5–2.5 m in front of imager, that is, its depth of field is about 1 m. The imaging speed of system is 2–4 seconds per frame. Moreover, the operating level and gain of detector have a great impact on image sharpness and system noise. There exists an optimized operating state of detector. Under the optimized condition, the clear terahertz images can be achieved and the resolution of target was 1.5–2 cm. Our imaging system can clearly detect metal objects hidden under human body’s clothes and substances with strong absorption on terahertz waves.