Applying adaptive optics (AO) system to correct aberrations is an effective technical way to improve the performance of optical systems. In order to ensure the long-term, safe and stable operation of the AO system, it is necessary to monitor the operating data of the AO system and identify the instability state of the system to provide decision making suggestions. Based on the above purpose, a set of 127 units AO system instability data simulation platform was established. The abnormal data frames were inserted into the closed-loop operation of the simulation platform, and the abnormal data sets under four kinds of closed-loop instability were obtained. Once the deformable mirror, the core component of AO system, worked abnormally, it will threaten the safety of the system. Based on the deformable mirror control voltage rms index, three machine learning methods were used: Kmeans clustering, K-NN classification and ARIMA prediction for recognition and detection. The detection results of the three methods in different types of abnormal data are different, indicating that the three anomaly detection methods have certain effects and scope of application for system instability detection. In actual application, one or a combination of multiple methods should be selected for testing.