Hyperspectral images contain abundant spectral information of ground objects and have great development prospects in the field of remote sensing images. Anomaly detection of hyperspectral images can detect abnormal targets in images without any prior spectral information. Therefore, it is widely used in national, military, and civil fields, and it is a research hotspot in hyperspectral image processing at present. However, hyperspectral images are characterized by complex data, strong redundancy, unlabeled and small number of samples, which brings great challenges to anomaly detection of hyperspectral images. Especially in deep learning, large image data is often needed as training samples, which is difficult to obtain hyperspectral images. Aiming at the problems that most existing algorithms are not adaptive to hyperspectral images and lack of space-spectral information utilization, a hyperspectral anomaly detection algorithm based on 3D convolution autoencoder network is proposed, which can effectively utilize hyperspectral image information, learn more discriminative feature expression, and improve detection accuracy under the premise of a small amount of training data. Firstly, the 3D convolution network is designed through 3D convolution, 3D pooling and 3D normalization, and then the spatial-spectral structure features of hyperspectral images are extracted. Then, the 3D convolution network and the 3D deconvolution network are embedded into the auto and decoder of the autoencoder network, respectively. background reconstruction is carried out by minimizing the reconstruction error combining the mean square error and the spectral angular distance. Finally, the Mahalanobis distance between the original hyperspectral image and the reconstructed background image is used for anomaly detection. This algorithm can automatically train all parameters in the network without prior information, learn the effective features of hyperspectral images and carry out background reconstruction in an unsupervised way. It is performed using the nine images from three sets of real high spectral data sets and is compared with the five algorithms of RX, SRX, CRD, UNRS, and LRASR. The experimental results show that this algorithm maintains a high detection effect and accuracy in the context of high spectrum images compared to existing algorithms.