Sensitive detection of remote vibrations at nanometer scale owns promising potential applications such as geological exploration, architecture, and public security. Nevertheless, how to detect remote vibration information with high sensitivity and anti-disturbance has become a major challenge. Reported current non-contact measurement methods are difficult to simultaneously possess characteristics of high light intensity sensitivity, long working distance, high vibration response sensitivity, and anti-disturbance of ambient light. Here, we propose a polarization-modulated laser frequency-shifted feedback interferometry method with the above characteristics, to obtain remote vibration information. The method can directly measure non-cooperative targets without the need for any cooperative markers. In each interference cycle, the energy as low as 2.3 photons can be effectively responded to, and the vibration amplitude sensitivity at 300 m can reach $0.72\mathrm{nm}/{\mathrm{Hz}}^{1/2}$ at 1 kHz. This approach provides a strategy for the ultrasensitive detection of remote vibration that is immune to electromagnetic interference.