Aiming at the problem of poor detection of vehicles and pedestrians by a single sensor in complex road conditions and bad weather conditions, a set of visible light, visible polarization, short-wave infrared and long-wave infrared multimodal data acquisition system was built to construct a multimodal dataset, and a multimodal pedestrian detection algorithm for vehicle pedestrians was proposesd. Firstly, a heterologous image registration algorithm based on the multi-scale partial intensity invariant features of improved SIFT feature points was proposed. Then, for target detection, a multimodal data object detection network based on YOLOv5 was proposed. Finally, the average accuracy is improved by 1.0% in the daytime dataset and 10.9% in the daytime and nighttime mixed dataset.