A two-dimensional particle flow program based on the Fish language of particle flow code (PFC) was compiled to construct a two-dimensional concrete fine-view model, and the calibrated PB model parameters were used to analyse the damage crack evolution and damage mechanism of concrete with different sizes. The results of the simulations show that the uniaxial compressive failure of concrete is caused by the expansion and evolution of cracks at the cement mortar interface and at the intersection of aggregate and cement mortar. Different sizes and random distribution of aggregates have certain influences on the peak stress and failure mode of concrete. Cracks in uniaxial compression specimens initiate at the interface between cement mortar and aggregate, and then propagate primarily along the interface between aggregate and cement mortar. As the load increases, cracks extend into the interior of the concrete. After reaching 90% of the peak compressive strength, the extension and increase speed of cracks enter a sudden increase stage. The interaction between the main crack and secondary cracks at the interface between cement mortar cause the concrete specimens to fail in a perforated manner. As the size of concrete and the quantity of aggregates increase, the number of cracks also increases. Moreover, with the decrease of concrete size, the ratio of sudden crack increase speed and crack quantity become larger.