Odometry is an important component of simultaneous localization and mapping (SLAM) technology. Existing odometry algorithms mainly rely on visual or laser sensors, failing to fully exploit the advantages of multimodal sensors and exhibiting insufficient robustness in feature-deprived scenarios and complex environments. To address this issue, this paper utilizes data from multimodal sensors including lidar, color camera, and inertial measurement unit, and proposes a multimodal fusion deep network, MLVIO-Net, which collaborates with an error state Kalman filter (ESKF) to form a multimodal fusion odometry system. MLVIO-Net consists of a feature pyramid network, multi-layer bidirectional long-short term memory (Bi-LSTM) network, pose estimation network, and pose optimization network, achieving close integration of multimodal data. The feature pyramid network performs hierarchical feature extraction on lidar point clouds, while the LSTM network effectively learns the temporal features of inertial measurement data. The pose estimation and optimization networks iteratively refine the predicted results. The ESKF predicts poses using the kinematic model of the inertial measurement unit and corrects poses using the predictions from MLVIO-Net, thereby improving prediction accuracy and significantly enhancing the output frame rate of the odometry. Experimental results on the open dataset KITTI demonstrate that the proposed multimodal fusion odometry exhibits higher accuracy and robustness compared to other common algorithms.