Temperature sensing is essential for human health monitoring. High-sensitivity ($>1\mathrm{nm}/°\mathrm{C}$) fiber sensors always require long interference paths and temperature-sensitive materials, leading to a long sensor and thus slow response (6–14 s). To date, it is still challenging for a fiber optic temperature sensor to have an ultrafast ($\sim \mathrm{ms}$) response simultaneously with high sensitivity. Here, a side-polished single-mode/hollow/single-mode fiber (SP-SHSF) structure is proposed to meet the challenge by using the length-independent sensitivity of an anti-resonant reflecting optical waveguide mechanism. With a polydimethylsiloxane filled sub-nanoliter volume cavity in the SP-SHSF, the SP-SHSF exhibits a high temperature sensitivity of 4.223 nm/°C with a compact length of 1.6 mm, allowing an ultrafast response (16 ms) and fast recovery time (176 ms). The figure of merit (FOM), defined as the absolute ratio of sensitivity to response time, is proposed to assess the comprehensive performance of the sensor. The FOM of the proposed sensor reaches up to $263.94(\text{nm/}°\mathrm{C})/\mathrm{s}$, which is more than two to three orders of magnitude higher than those of other temperature fiber optic sensors reported previously. Additionally, a three-month cycle test shows that the sensor is highly robust, with excellent reversibility and accuracy, allowing it to be incorporated with a wearable face mask for detecting temperature changes during human breathing. The high FOM and high stability of the proposed sensing fiber structure provide an excellent opportunity to develop both ultrafast and highly sensitive fiber optic sensors for wearable respiratory monitoring and contactless in vitro detection.