It’s significant to study the algorithm of endmember extraction, which is the key for pixel unmixing, in the fields of feature identification, abundance inversion, quantitative remote sensing and so on. Based on the theory of shannon entropy and Gaussian distribution function, a new algorithm, named spectral minimum shannon entropy (SMSE) method for extracting endmembers of hyperspectral images, is proposed in the present paper after analyzing the characteristics of spectra of the hyperspectral images. This algorithm was applied to extract the endmembers of an AVRIRS hyperspectral image, it was found that these extracted endmember spectra have higher precision by matching with the spectral library of United States Geological Survey(USGS). At the same time, it was also found that the SMSE algorithm has better efficiency and accuracy for extracting endmember spectra through comparing and analyzing comprehensively the results of endmember extraction of the experimental data by using the methods of SMSE, pixel purity index(PPI), sequential maximum angle convex cone(SMACC) and so on. In addition, the SMACC and SMSE are used to extract the endmembers in a Hyperion hyperspectral image, and it is concluded that the results of the SMSE is better than the SMACC’s. Thus, the SMSE algorithm can be thought to have a certain degree of universal applicability.