Most of the existing anomaly detection algorithms for hyperspectral image only focus on the spectrum differences between the target and background while ignoring the spatial structure differences, which leads to poor detection results. Aiming at this issue, we propose a novel algorithm based on the combination of spectral and spatial information for anomaly detection (SSAD). To preserve the spatial structure information of the image, we detect anomalies band by band. The dual windows are established to calculate the luminance differences between the pixel under test (PUT) and background, and the spectral anomaly degree of PUT is measured. Then the inner window is regarded as the spatial structure window of PUT, and the most similar spatial structure window with the spatial structure window of PUT is searched from the background. The differences between the two is calculated to measure the spatial structure anomaly degree of PUT. Thus, the anomaly index of the PUT is obtained by the measurement of spectral and spatial anomaly degree. Going through the whole image, the detection result of the algorithm is acquired by summing up the anomaly index of each band correspondingly. Experimental results on three hyperspectral data show that, compared with existing anomaly detection algorithms, the proposed algorithm can significantly reduce the false alarm rate and has good robust to noise.