Most machine vision-based roughness measurement methods either build a prediction model based on roughness correlation indices or build an index-free prediction model using deep learning networks. However, both these models have disadvantages. The artificial designed index has a complicated calculation process, which is not conducive to inline detection. In comparison, deep learning models rely heavily on big data. It is difficult to train an effective model when the amount of data is insufficient. To address the above problems, this study proposes a graph neural network-based method for measuring the roughness of milling surfaces. This proposed approach acquired the ability to learn autonomously during the training phase. Thus, only a few milling samples were required to measure the roughness of the milling workpieces. The experimental results show that the proposed method can automatically extract features on roughness measurement of milling workpieces with high accuracy and good robustness of lighting environment.