Traffic sign detection is an essential function of autonomous driving systems, and most modern traffic sign detectors are anchor-based, traversing potential object locations based on anchors. To solve the problems of heavy computing costs and the need to set several hyperparameters in anchor-based models, we propose an anchor-free traffic sign detection algorithm based on an encoder-decoder structure. We introduce a residual augmentation branch in the decoder module in this study to improve feature expression ability during the decoding process. To improve the ability to detect multiscale traffic signs, we propose a multiscale feature fusion subnetwork to effectively extract and use multiscale features. A Ghost lightweight module is adopted by the multiscale feature extraction module, which indistinctively increases the computational cost. On the Tsinghua-Tencent 100 K dataset, our approach achieved a recall of 92.5% and an accuracy of 90.3%, while the model's parameter amount and model size are approximately 1.61×10⁷ and 64.4 Mbit, respectively. The experimental results show that the proposed algorithm outperforms the mainstream object detection algorithms in terms of precision, computing cost, and overall performance.