Polycrystalline materials are extensively employed in industry. Its surface roughness significantly affects the working performance. Material defects, particularly grain boundaries, have a great impact on the achieved surface roughness of polycrystalline materials. However, it is difficult to establish a purely theoretical model for surface roughness with consideration of the grain boundary effect using conventional analytical methods. In this work, a theoretical and deep learning hybrid model for predicting the surface roughness of diamond-turned polycrystalline materials is proposed. The kinematic–dynamic roughness component in relation to the tool profile duplication effect, work material plastic side flow, relative vibration between the diamond tool and workpiece, etc, is theoretically calculated. The material-defect roughness component is modeled with a cascade forward neural network. In the neural network, the ratio of maximum undeformed chip thickness to cutting edge radius RTS, work material properties (misorientation angle θg and grain size dg), and spindle rotation speed ns are configured as input variables. The material-defect roughness component is set as the output variable. To validate the developed model, polycrystalline copper with a gradient distribution of grains prepared by friction stir processing is machined with various processing parameters and different diamond tools. Compared with the previously developed model, obvious improvement in the prediction accuracy is observed with this hybrid prediction model. Based on this model, the influences of different factors on the surface roughness of polycrystalline materials are discussed. The influencing mechanism of the misorientation angle and grain size is quantitatively analyzed. Two fracture modes, including transcrystalline and intercrystalline fractures at different RTS values, are observed. Meanwhile, optimal processing parameters are obtained with a simulated annealing algorithm. Cutting experiments are performed with the optimal parameters, and a flat surface finish with Sa 1.314 nm is finally achieved. The developed model and corresponding new findings in this work are beneficial for accurately predicting the surface roughness of polycrystalline materials and understanding the impacting mechanism of material defects in diamond turning.