Process noise and photoelectric measurement noise seriously affect the sensitivity when detecting the displacement of the microsphere in the actual measurement. Aiming at this problem, a method of detecting the displacement of the microsphere based on Kalman filter is proposed. In order to obtain the displacement of the microsphere with high sensitivity and high signal-to-noise ratio, the harmonic oscillator model which interprets the motion of the microsphere is transformed into the form of state transition matrix for Kalman filter. Simulated results show that the root-mean-square error of measured displacement is reduced from 1 nm to 0.27 nm after filtered at 101 kPa. Experimental results show that the proposed method reduces the detection root-mean-square error of measured displacement from 2.8 nm to 1.1 nm at 293 K and 101 kPa. Other parameters remaining unchanged, the root mean square error is reduced from 5.2 nm to 2.1 nm at 293 K and 1 Pa. The proposed method can be applied to the laser cooling feedback scheme of the center-of-mass motion of a micro-particle optically trapped in high vacuum.