In recent years, progress in the field of physics has led to many useful applications^[1–3]. Especially, optically pumped atomic magnetometers have achieved ultrahigh sensitivity and precision^[4,5]. They have been widely used in various applications such as medical diagnosis^[6,7], mineral detection^[8], space exploration^[9], and fundamental physics research^[10–12]. Among various atomic magnetometers, the spin-exchange relaxation-free (SERF) atomic magnetometer has the advantages of small volume and high sensitivity. Therefore, the SERF atomic magnetometer has been increasingly applied to biomagnetic measurements such as magnetocardiography (MCG)^[13,14], magnetoencephalography (MEG)^[15–17], and magnetopneumography (MPG)^[18]. Conventional single- or double-axis SERF atomic magnetometers typically detect the magnetic fields along the direction perpendicular to the pump beam. However, in practical applications, such as biomagnetic field detection, the three-axis magnetic field measurement is necessary and has been proven to effectively improve the ability to suppress the interference of magnetic noise and movement^[19,20].

Several methods of three-axis magnetic field measurement have been proposed. Seltzer and Romalis reported a three-axis SERF atomic magnetometer on the quasi-static condition by applying two low-frequency modulation fields in a laboratory environment without magnetic shields using a feedback system^[21]. Huang et al. developed a single-beam atomic magnetometer for three-axis magnetic measurement based on the quasi-static evolution of spin polarization^[22]. Limited by the quasi-static operation condition, these two methods using low-frequency modulation have limited bandwidths. Zheng et al. proposed a method for measuring the three-axis magnetic field using an elliptically polarized laser that was reflected after passing through an alkali–metal vapor cell^[23]. Tang et al. demonstrated a single-beam three-axis magnetometer by applying bias magnetic fields and modulation fields^[24]. Lu et al. reported a nonmodulation three-axis atomic magnetometer with a pump–probe orthogonal configuration^[25]. The three components of the magnetic field can only be obtained in turn. Xiao et al. realized the three-axis measurement of the magnetic fields with three orthogonal modulation fields and one pump laser beam reflected at 90° in alkali–metal vapor^[26]. The sensitivity of the magnetometer is limited due to the attenuation of the reflected light. Boto et al. presented a triaxial optically pumped magnetometer that used two orthogonal circularly polarized lights and three modulation magnetic fields with different frequencies or phases^[20]. This atomic magnetometer was based on the optical absorption configuration, which was easily affected by the light-intensity noise.

In this study, a high-sensitivity three-axis SERF atomic magnetometer with two orthogonal elliptically polarized laser beams is proposed. The circularly polarized component of the laser pumps the alkali–metal atoms, and the linearly polarized component simultaneously probes the spin polarization by the detection of the optical rotation angle. The light–intensity noise can be suppressed through polarization balance measurement. Every laser beam can detect the magnetic fields along two orthogonal axes by applying three modulation magnetic fields. Three-axis sensitivities are achieved at $12.5{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $x$ axis, $9.5{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $y$ axis, and $22{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $z$ axis simultaneously. Furthermore, a closed-loop system is implemented to maintain the magnetic field measured by atoms near zero. The closed-loop atomic magnetometer obtained a wider dynamic range of $\pm 150\mathrm{nT}$, a higher bandwidth of over 200 Hz, and lower cross talk between axes.

The evolution of the spin polarization of the alkali–metal atom spin ensemble in the SERF regime can be expressed by the Bloch equation^[27,28]$$\frac{\mathrm{d}\mathbf{P}}{\mathrm{d}t}=\frac{1}{q}[\gamma \mathbf{B}\times \mathbf{P}+R_{\text{op}}(s\widehat{z}-\mathbf{P})-R_{\mathrm{tot}}\mathbf{P}],$$(1)where $\mathbf{P}$ is the electron spin polarization vector, $q$ is the slowing-down factor, $\gamma$ is the electron gyromagnetic ratio, $\mathbf{B}$ is the magnetic field vector, $R_{\text{op}}$ is the optical pumping rate along the $z$ axis, $s$ is the photon spin of the pump laser, and $R_{\mathrm{tot}}$ is the total spin relaxation rate.

For the atomic magnetometer with an elliptically polarized laser beam along the $z$ axis, the small optical rotation $\theta$ of the linearly polarized component is proportional to the polarization projection component $P_z$^[29], $$\theta =\frac{1}{2}l{r}_{\mathrm{e}}cn{P}_z{f}_{\mathrm{D}1}\frac{v-v_{\mathrm{D}1}}{{(v-v_{\mathrm{D}1})}^2+{({\mathrm{\Gamma }}_{\mathrm{D}1}/2)}^2},$$(2)where $l$ is the length of the vapor cell, $n$ is the number density of the alkali–metal atom, $r_{\mathrm{e}}$ is the radius of the electron, $c$ is the speed of light, $f_{\mathrm{D}1}$ is the oscillator strength, $v$ is the laser frequency, $v_{\mathrm{D}1}$ is the resonance frequency for the D1 transition, and ${\mathrm{\Gamma }}_{\mathrm{D}1}$ is the broadened linewidth due to the pressure broadening. The optical rotation angle $\theta$ is detected via balanced polarimetry as the output signal.

The solution of the Bloch equation indicates the spin polarization along the $z$ axis is nearly linear with the magnetic field^[30–32], $$P_z\alpha \frac{2s{R}_{\text{op}}{J}_0(u)J_1(u)}{{(R_{\text{op}}+R_{\mathrm{tot}})}^2}{B}_y\sin ({\omega }_{\mathrm{m}}t),$$(3)where $B_y$ is a small-amplitude pending-measured magnetic field, $B_{\mathit{y}}\cos ({\omega }_{\mathrm{m}}t)$ is a high-frequency modulation field along the $y$ axis, $u=\gamma B_y/(q{\omega }_{\mathrm{m}})$ is the modulation index, $J_0(u)$ and $J_1(u)$ are Bessel functions of the first kind and the subscript number represents the order of the function. The atomic magnetometer response to the pending-measured magnetic field $B_y$ is determined by demodulating the output signal with a reference frequency ${\omega }_{\mathrm{m}}$.

However, it should be noted that the atomic magnetometer can only detect the magnetic field perpendicular to the pumping laser direction. To solve this difficulty, we introduced two orthogonal elliptically polarized laser beams. The propagation directions of the two laser beams in the atomic magnetometer are orthogonal. As shown in Fig. 1, the two laser beams propagated through the alkali–metal vapor along the $x$ and $z$ axes, respectively. Simultaneously, the triaxial modulation magnetic fields $B_{x\mathrm{m}}\sin ({\omega }_{x}t)\widehat{x}+B_{y\mathrm{m}}\cos ({\omega }_{y}t)\widehat{y}+B_{z\mathrm{m}}\sin ({\omega }_{z}t)\widehat{z}$ are employed to manipulate the alkali–metal atoms. As mentioned above, the atoms polarized along the $x$ axis are nonsensitive to the fields $B_{x\mathrm{m}}\sin ({\omega }_{x\mathrm{m}}t)\widehat{x}$, which are only modulated by $B_{y\mathrm{m}}\cos ({\omega }_{y}t)\widehat{y}+B_{z\mathrm{m}}\sin ({\omega }_{z}t)\widehat{z}$. The output signal for the magnetic field at the $y$ and $z$ axes can be demodulated with reference frequency ${\omega }_y$ and ${\omega }_z$, respectively. Moreover, the dual-axis measurements can be also realized through quadrature demodulation techniques when the modulation frequency is set to be the same along the $y$ and $z$ axes^[33]. Similarly, the alkali–metal atoms polarized along the $z$ axis can detect the magnetic fields along the $x$ and $y$ axes with modulation fields $B_{x\mathrm{m}}\sin ({\omega }_{x}t)\widehat{x}+B_{y\mathrm{m}}\cos ({\omega }_{y}t)\widehat{y}$. Therefore, the real-time three-axis measurement of the magnetic fields can be realized with three-axis modulation fields based on this double laser beam structure.

The conventional SERF atomic magnetometer has a limited dynamic range (generally less than $\pm 10\mathrm{nT}$) and bandwidth (usually $\sim 100\mathrm{Hz}$)^[5]. To solve the difficulty, we introduce a closed-loop control method in order to improve its performance. As shown in Fig. 2, the proportional-integration (PI) controller is employed to null the magnetic field detected by the atomic magnetometer with a magnetic coil. The feedback signal is also collected by the data acquisition equipment (DAQ) as the output of the atomic magnetometer. The transfer function of the open-loop atomic magnetometer is $H(s)=G_0/(1+s/{\omega }_c)$. $G_0$ is the DC response and ${\omega }_c$ is the cutoff angular frequency. The transfer function of the PI controller is $G_{\mathrm{PI}}(s)=K_{\mathrm{p}}+K_{\mathrm{i}}/s$. $K_{\mathrm{p}}$ and $K_{\mathrm{i}}$ are the coefficients of the proportional and integral gains, respectively. The transfer function of the closed-loop atomic magnetometer is given by^[34]$$G_{\mathrm{closed}}(s)=\frac{K_{\mathrm{coil}}{G}_0(K_{\mathrm{p}}s+K_{\mathrm{i}})}{s^2/{\omega }_c+(1+K_{\mathrm{coil}}{G}_0{K}_{\mathrm{p}})s+K_{\mathrm{coil}}{G}_0{K}_{\mathrm{i}}},$$(4)where $K_{\text{coil}}$ denotes the scaling coefficient of the magnetic coil.

Figure 3 illustrates the structure of the three-axis atomic magnetometer. The cube-shaped vapor cell with an internal 8 mm side length contained a proper amount of metal ${}^{87}\mathrm{Rb}$, approximately 2100 Torr ${}^4\mathrm{He}$, and 70 Torr ${\mathrm{N}}_2$. A temperature control system including a computer program, two heating films, and a temperature sensor was applied to heat the cell. The cell was heated to approximately 443 K using a pair of twisted-pair heating films. To further eliminate the harmful magnetic field generated by the heating current, the films were driven by a high-frequency current (500 kHz). The micro-nonmagnetic sensor is mounted close to the cell to monitor the temperature.

The atomic magnetometer was operated using a four-layer cylindrical $\mathrm{\mu }$-metal magnetic shield. A cylindrical triaxial coil was mounted to generate the magnetic field, which was composed of two saddle-shaped coils (along the $x$ and $y$ axes) and a solenoid coil (along the $z$ axis).

An external cavity laser (Toptica DLpro) provided laser beams introduced into the magnetometer by polarization-maintaining fibers (PMFs). The two laser beams with a 3 mm diameter were collimated by a collimating lens. A $\lambda /4$ wave plate was fixed at $\pi /8$ relative to the polarization axis. The two laser beams had an ellipticity of $\pi /8$ through the $\lambda /4$ wave plates. Two laser beams propagated distinctly along the $x$ and $z$ axes in the cell. To decrease the cross talk effect, the beams had a distance of about 0.8 mm along the $y$ axis. The optical rotation angle was detected using a balanced photodiode. The output signal was demodulated by a lock-in amplifier (Zurich Instrument MFLI). Three PI controllers received the magnetometer response signal and transmitted the three-axis feedback signal to the triaxial coil under the closed-loop mode so the atoms of alkali metals can experience a near-zero magnetic field. Three demodulation signals under the open-loop regime and triaxial feedback signals under the closed-loop regime were collected by DAQ to further analyze the atomic magnetometer performance.

In the following sections, the laser and modulation parameters were optimized to improve the scale factor of the atomic magnetometer. Then, the single-axis and three-axis sensitivities were separately investigated. Subsequently, a closed-loop control method was applied to counteract the pending-measured three-axis magnetic fields experienced by the alkali–metal atoms. Finally, the performance, including cross talk between axes, dynamic range, and bandwidth, was investigated and compared under the open-loop and closed-loop regimes.

The scale factor of the atomic magnetometer can be improved by optimizing the laser parameters and modulation index. For simplicity, the method to optimize the scale factor along the $y$ axis is only described; it is the same for the $x$ and $z$ axes. First, the light intensity is set to $30\mathrm{mW}/{\mathrm{cm}}^2$, and the scale factor is measured and optimized by tuning the laser frequency. As shown in Fig. 4(a), the maximum scale factor is achieved when the laser frequency is detuned by 117 GHz from the Rb D1 line. When the laser frequency is tuned close to the D1 line, the alkali–metal atoms would absorb most of the light, thereby reducing the signal strength. By adjusting the laser off-resonance, a spin polarization of 50% is obtained to maximize the sensitivity of the atomic magnetometer^[29]. Second, the laser intensity is also optimized when the laser frequency is tuned by 117 GHz from the D1 line. Figure 4(b) indicates that the scale factor of the magnetometer increases with the laser intensity. However, when the laser intensity is too high, the intensity noise could lead to a loss of magnetometer sensitivity.

The modulation parameter is also investigated and optimized to improve the scale factor of the atomic magnetometer along the $y$ axis. The scale factor of the atomic magnetometer is measured at different modulation amplitudes with a modulation frequency of 1 kHz, as shown in Fig. 5, which is the function of the modulation index $u$. The scale factor of the atomic magnetometer achieves a maximum when the modulation amplitude is approximately 100 nT rms.

With the optimized laser and modulation parameters, the single-axis sensitivity is measured. The DAQ collects the atomic magnetometer’s response signal when a calibration magnetic field with an amplitude of 100 pT rms and a frequency of 30.5 Hz is applied along the $y$ axis.

The response signal of the atomic magnetometer is collected by the DAQ when a 30.5 Hz calibration magnetic field with an amplitude of 100 pT rms is applied along the $y$ axis. Then the noise spectral density of the signal is normalized by dividing the frequency response. As shown in Fig. 6, the calibration field at 30.5 Hz is used to evaluate the sensitivity of the atomic magnetometer. The single-axis measurement exhibits a high sensitivity of approximately $8{\mathrm{fT}/\mathrm{Hz}}^{1/2}$.

When three-axis modulation magnetic fields are applied, the three components of the magnetic field can be detected simultaneously. The three-axis modulation frequency is set to 1 kHz. To obtain consistency in the triaxial response, the same modulation amplitude is applied along the three axes. The three-axis modulation amplitude is reoptimized using the above method. The modulation field increases the spin-exchange relaxation; thus the three-axis modulation amplitude should be adjusted to be smaller than that of the single-axis measurement. The modulation amplitude is optimized to 30 nT rms using the above optimization method. The laser beam along the $z$ axis is used to detect the magnetic fields along the $x$ and $y$ axes, and another laser beam detects the magnetic field along the $z$ axis. The three-axis sensitivity measurements are performed, as shown in Fig. 7. The three-axis sensitivities are $12.5{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $x$ axis, $9.5{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $y$ axis, and $22{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $z$ axis. Because of the polarization balance measurement, the sensitivity in this study is better than that in our prior work^[34]. The sensitivity may be improved further using the power differential method^[35].

Three-axis sensitivity decreases due to two factors compared to single-axis sensitivity. First, the spin-exchange relaxation increases because of the three-axis modulation field^[36]. The larger spin relaxation rate leads to a lower sensitivity. Second, the light-shift effect has a significant impact on sensitivity, which creates a virtual magnetic field $B_{\mathrm{LS}}$ experienced by Rb atoms along the propagation direction of the laser^[37], $$B_{\mathrm{LS}}=\frac{s{r}_{\mathrm{e}}c{f}_{\mathrm{D}1}\varphi }{\gamma }\frac{v-v_{\mathrm{D}1}}{{(v-v_{\mathrm{D}1})}^2+{({\mathrm{\Gamma }}_{\mathrm{D}1}/2)}^2},$$(5)where $\varphi$ is the photon flux.

The virtual magnetic field can be compensated for by using triaxial magnetic coils or by tuning the laser frequency^[38,39]. However, the pump laser with detuning frequency for the SERF atomic magnetometer is needed to avoid the extreme gradients of the optical pumping rate. Furthermore, there are two parts of alkali–metal atoms polarized along the $x$ and $z$ axes, respectively. The direction of the virtual magnetic field experienced by these two parts of atoms is also different. Therefore, the light-shift effect along the $x$ and $z$ axes cannot be canceled using triaxial magnetic coils simultaneously.

The system performance, including bandwidth, dynamic range, and cross talk effect, was measured and analyzed in open-loop and closed-loop regimes.

The atomic magnetometer acquired a wider bandwidth with a closed-loop control system. White noise with an amplitude of 100 pT rms and a bandwidth of 800 Hz is applied along the $x$ axis. The Fourier spectrum of the response is used to estimate the frequency response^[40]. The measurements along the $y$ and $z$ axes were performed similarly. As indicated in Fig. 8, the bandwidths of the atomic magnetometer under the closed-loop regime are 240, 200, and 280 Hz along the $x$, $y$, and $z$ axes, respectively, larger than those of 150, 90, and 180 Hz under the open-loop regime. The bandwidth in the open-loop mode was affected by the spin-relaxation rate and optical pump rate^[5]. According to Eq. (4), the bandwidth of the atomic magnetometer can be increased by tuning $K_{\mathrm{p}}$ and $K_{\mathrm{i}}$ in the closed-loop regime.

The cross talk between the axes of the atomic magnetometer should be considered. For example, there is the cross talk signal from $y$ or $z$ axis in the response signal of the pending-measured magnetic field along the $x$ axis. A calibration magnetic field at 30.5 Hz is added along the three orthogonal axes ($x$, $y$, and $z$ axes) in turn to measure the cross talk signal. Under the condition of open-loop control, the atomic magnetometer receives 14% and 15% leakage into $B_y$ and $B_z$ when a calibration field is added along the $x$ axis, 7% and 14% leakage into $B_x$ and $B_z$ when a calibration field is added along the $y$ axis, and 10% and 30% leakage into $B_x$ and $B_y$ when a calibration field is added along the $z$ axis. The main influencing factor is the light-shift effect. During the experiment, alkali–metal atoms were exposed to a fictitious magnetic field. Furthermore, the cross talk effect is also influenced by the installation errors of the magnetic coils and laser.

The closed-loop operation can reduce the cross talk between axes. The same method is used to measure the cross talk under the condition of the closed-loop control. The closed-loop atomic magnetometer obtains 7% and 6% leakage into $B_y$ and $B_z$ when applying a calibration field along the $x$ axis, as shown in Fig. 9(a). Figure 9(a) indicates that the feedback signal of the closed-loop atomic magnetometer is applied immediately when the magnetic field changes along the $x$ axis. The magnetic fields are, respectively, applied along the $y$ and $z$ axes to measure the three-axis response. As shown in Figs. 9(b) and 9(c), the closed-loop atomic magnetometer obtains 2% and 5% leakage into B_x and B_z when applying a calibration field along the $y$ axis, and 4% and 12% leakage into B_x and B_y when applying a calibration field along the $z$ axis, respectively. In addition, the dynamic range under the closed-loop regime is extended to $\pm 150\mathrm{nT}$, which is much larger than that under the open-loop regime.

We introduced a high-sensitivity three-axis atomic magnetometer with two orthogonal elliptically polarized lights that pump alkali–metal atoms and probe the optical rotation signal simultaneously.

After parameter optimization, the average sensitivities are $14{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $x$ axis, $11$$\mathrm{fT}/{\mathrm{Hz}}^{1/2}$ along the $y$ axis, and $25{\mathrm{fT}/\mathrm{Hz}}^{1/2}$ along the $z$ axis. When the PI modules were employed, the atomic magnetometer achieved low cross talk (in the range of 2% to 12%), the dynamic range increased to $\pm 150\mathrm{nT}$, and the bandwidth was enhanced to 240, 200, and 280 Hz along the three orthogonal axes; the lower bound of the bandwidth is 1 Hz.

The three-axis atomic magnetometer presented in this study not only offers high sensitivity but also can operate with large dynamic range and wide bandwidth under the condition of the closed-loop control. Therefore, the high-sensitivity atomic magnetometer is expected to be applied in the field of biomagnetic measurements in a challenging environment.