The integration of endoscopy has significantly propelled the diagnosis and treatment of gastrointestinal diseases, with colonoscopy establishing itself as the primary method for early diagnosis and preventive care in colorectal cancer (CRC). Although deep learning holds promise in mitigating missed polyp rates, modern endoscopy examinations pose additional challenges, such as image blurring and atomizing. This study explores lightweight yet powerful attention mechanisms, introducing the spatial-channel transformer (SCT), an innovative approach that leverages spatial channel relationships for attention weight calculation. The method utilizes rotation operations for inter-dimensional dependencies, followed by residual transformation, encoding inter-channel and spatial information with minimal computational overhead. Extensive experiments on the CVC-ClinicDB polyp detection dataset, addressing endoscopy pitfalls, underscore the superiority of our SCT over other state-of-the-art methods. The proposed model maintains high performance, even in challenging scenarios.