The vibration signal of the gearbox contains all the characteristic information during gear operation. The current running state of the equipment can be mastered by collecting and analyzing vibration signals. There are many methods for collecting gearbox vibration signals. The noncontact measurement method, based on the principle of laser self-mixing interference, has extremely unique advantages. First, this method can collect the early characteristic signals of gearbox fault and take corresponding measures in time. Second, the method can obtain the required characteristic information without damaging the gearbox’s surface. Third, compared with the installation of an acceleration sensor, the influence of this method on the reflector attached to the gearbox is very small and can be ignored. The planetary gear wear fault is artificially made, and the fault characteristic signal is collected during the experiment. Furthermore, the fault vibration signals of planetary gears are extracted and the intrinsic mode function (IMF) components of each order and Hilbert spectrum are analyzed using the Hilbert-Huang transform (HHT) signal processing method. The Hilbert spectrum can be analyzed during the period of gear wear fault. The results are in good agreement with the theoretical values, showing that this method can effectively detect gearbox vibration faults.