A method for detecting the tower components from far to near is proposed based on the histogram of the gradient (HOG) along the tower gradient direction by analyzing the structural characteristics of hollowing-out of a tower. The multi-layer perceptron (MLP) is first trained by using the HOG feature of the tower under different orientations to obtain a trained classification model, then the aerial image is input into this classification model to identify the orientation of the tower, and the detection of a local target is finally realized. Compared with that of a deep learning neural network, the classification feature of the proposed method is clear and representative. The experimental results show that the detection accuracy of the proposed method is 27.9% higher than that of the Faster RCNN (Regions with Convolutional Neural Networks) method, but the computation time is 70.9% lower than the latter. The proposed method is suitable for the accurate detection of the tower orientation and its local parts by the unmanned aerial vehicle in an open environment.