As a typical porous media material, the diffusivities of concrete are closely related to its pore structure. It is necessary to propose a model for predicting the diffusivities based on the time-varying pore structure since the pore structure can continuously change with the hydration process. In this work, the parameters of the pore structure, the moisture distribution in the unsaturated state, and the prediction and validation of the relative diffusion coefficients of cement paste at different saturation levels were analyzed based on the reconstructed cement paste microstructure. It is indicated that the predicted relative diffusion coefficients are related to the extracted pore parameters (i.e., hydration degree α, porosity ρ and dimensionless peak pore size B*), and the relationships between α and time t, ρ and α, and B* and ρ were systematically analyzed, and a diffusivity prediction model containing the time-varying parameters f (α(t), ρ(t), B*(t)) was proposed. The proposed model can fully consider the time-varying process of concrete pore structure affected at different water-cement ratios and hydration time, thus providing a microscopic perspective approach for analyzing the diffusivities of unsaturated concrete from the perspective of time-varying pore structure.