With the emergence of Optical Coherence Tomography (OCT) technology and the rapid development of computer hardware, researchers have been attempting to utilize computer-aided to identify breast tissue OCT images. This study proposes a Convolutional Neural Network (CNN) model based on residual network enhancement for auxiliary diagnosis of OCT breast tissue images. The proposed method employs ResNet-18 framework as the basis to prevent gradient disappearance or explosion. In addition, to enhance the efficiency and regularization of the model, the original 7×7 convolutional layer was substituted with a series of three cascading layers, each having a dimension of 3×3. This design choice allows for a nonlinear decomposition of the original 7×7 layer while preserving the same receptive field size. As a result, it accomplishes two advantages: a reduction in computational cost associated with model parameters and acting as an implicit regularization technique. Subsequently, a Convolutional Block Attention Module (CBAM) was introduced after each set of cascaded small convolutional layers and the final convolutional layer. This module integrates a spatial attention module, which focuses on capturing spatial dependencies, and a channel attention module, which emphasizes informative channels, thereby serving as the initial stage of filtering and enhancing the discriminative capabilities of the network. At the same time, Octave Convolution (OctConv) is employed to substitute the 3×3 convolutional layers in the original model. The convolutional kernel of OctConv has the capability to partition the input image sample data into four parts based on the low-frequency dimension ratio parameter within its structure. This functionality allows the network to dynamically balance high and low-frequency components during the process of extracting image features as the secondary filtering stage. After this, the Global Average Pooling (GAP) layer is used instead of the fully connected layer to reduce the computation of network parameters, and the structure of the network is regularized to prevent overfitting of the models. Ultimately, a residual network model “Double Filtering” ResNet (DF-ResNet) is constructed. The “double filtering” structure can not only reduce the overall parameter computation of the model, but also focus on high-frequency components with rich structural information during feature extraction. By decreasing the ratio of low-frequency constituents, the reduction of informative duplication is attained, resulting in an enhancement of the model's proficiency towards categorization and identification of images manifesting proportional configurations. The proposed DF-ResNet model is employed to train and classify the OCT image dataset of three breast tissue types. It conducts multiple tuning tests and optimization techniques such as data augmentation and batch normalization, achieving an overall classification accuracy of 96.88%. After conducting three comparative experiments, the performance of the DF-ResNet model has been validated. In the first experiment, the DF-ResNet model was compared with the ResNet-28 model with an equivalent number of layers. The experimental results showed that the replacement of some convolutional layers with OctConv allows the model to focus on high-frequency components during the image feature extraction process. This led to an improvement in the model's ability to classify and recognize images with similar structures and ultimately culminating in an overall increase in classification accuracy. Additionally, it is important to note that the use of OctConv did not negatively impact the overall convergence speed of the model. In the second experiment, the DF-ResNet model was compared with the Oct_ResNet-28 model, which incorporated OctConv as a modification to improve its performance. The experimental results validated that the DF-ResNet model effectively filtered out low-frequency information. In the third experiment, the performance of the DF-ResNet model was assessed against established CNN models such as DenseNet-169 and VGG-19. The DF-ResNet model not only boasts a reduced parameter count but also demonstrated superior classification accuracy compared to several traditional CNN models. For the classification of OCT images of breast tissue, the DF-ResNet model displayed exceptional performance, robustness, and real-time processing capabilities. As a result, it is well-suited for providing technical support for real-time margin diagnosis in the clinical applications of breast cancer.