Lithium niobate (${\mathrm{LiNbO}}_3$, LN) is a key crystal material in photonics, which has comprehensive functional properties, such as optical harmonic generation, electro-optic, acousto-optic, and photorefractive properties^[1,2]. Particularly, thin-film LN on insulator (LNOI) has emerged in recent years as a new feasible platform for various integrated LN photonic applications, such as microcavity frequency comb^[3–5], waveguiding modulators^[6–12], ultracompact quantum light sources^[13,14], and on-chip spectrometers^[15]. Furthermore, as a typical artificial nano-structured material, the metasurfaces that are formed by engineered subwavelength building blocks known as meta-atoms have demonstrated the ability to control optical waves with unprecedented flexibility, providing a new promising solution for shrinking optical devices down to subwavelength dimensions. In the recent years, studies of LN metasurfaces have captured a growing interest, and various intriguing optical phenomena have been demonstrated, including structural color^[16], engineerable second harmonic generation^[17–21], enhanced electro-optic modulations^[22–24], and generation of entangled photon pairs^[25].

Until now, there are two main technical routes developed to fabricate the LN metasurfaces. One method is the focused ion beam (FIB) milling^[7,8,16,19], and the other is based on electron beam lithography (EBL) combined with the dry plasma etching (PE) process^[5,17,26,27].

The FIB milling process, which realizes the nano-structures by direct milling through the LN film using the ion beam (such as gallium ion, ${\mathrm{Ga}}^{+}$), has several advantages. For example, the FIB milling technique has greater flexibility and lower cost compared with the combined EBL-PE processes. Moreover, the surface and sidewalls of the nano-structures fabricated by the FIB are generally smooth. However, several problems also arise from FIB milling, such as a round top in nano-structures and crystal lattice damage. The first problem could be attributed to angle divergence and scattering of the ion beams and also the redeposition of sputtered atoms during FIB milling. Furthermore, in order to remove the surface lattice damage, amorphization by the high-energy ion beam bombardment and unwanted ${\mathrm{Ga}}^{+}$ ion implantation, a subsequent wet treatment by dilute hydrofluoric acid (HF) is usually adopted^[28–30]. However, this process causes serious lateral etching of the silicon dioxide (${\mathrm{SiO}}_2$) bonding layer under the LN film, which not only reduces the mechanical strength of the final nano-structures, but also leads to significant structural deviations from the original design.

On the other hand, the combined EBL-PE process involves first patterning a high-precision electron beam resist (e-beam resist) mask by EBL and then further transferring it to the LN layer by PE. However, LN is chemically inert and therefore difficult to etch. The e-beam resist mask usually has low selectivity to LN, so thick resist is normally required, however, this will greatly limit fabrication resolution and pattern density. Alternatively, multilayer masking schemes have been developed, such as stacking multiple e-beam resist layers or adding hard mask layers [e.g., nickel^[31,32], chromium (Cr)^[33,34], silicon^[35,36], or ${\mathrm{SiO}}_2$^[37,38]]. Although these methods provide the required selectivity, the multiple patterning, cleaning, and etching steps greatly increase the complexity of the process. In addition, due to mask errors^[39,40] or large grains in metal masks, the quality of the final structures will be reduced, for example, deteriorating the sidewall steepness and causing surface roughness.

In this paper, to overcome the respective constraints of the above individual technique, we demonstrate a technical route that combines the FIB direct milling and the PE procedure (inductively coupled plasma reactive ion etching, ICP-RIE) to fabricate LN metasurfaces. Unlike the round shaped ridges fabricated by FIB alone, the cross sections obtained here are trapezoids with sidewall angles of larger than 80°. The transmission spectral properties of the metasurfaces are studied both numerically and experimentally. Quasi-bound states in the continuum (QBIC) are steadily observed in experiments, confirming the high quality of the fabricated samples. Our study opens up a new route for fabricating high-$Q$-factor LN metasurfaces for applications in optical communications, on-chip photonics, laser physics, and sensing.

The structural design of the LN nano-grating metasurfaces is illustrated in Fig. 1. The LNOI wafer (NANOLN Co., Ltd.) adopted here includes a quartz substrate and a 230-nm-thick LN film on top. The geometry of the nano-grating array is defined by the ridge width ($w$) and the ridge height ($h$). Furthermore, the lattice constant is P_d. Nano-grating ridges are oriented parallel to the optical axis of the LN crystal, denoted here as the $y(\mathrm{e})$ axis.

The process to fabricate the metasurfaces is illustrated in Fig. 2(a). In order to remove particles and organic impurities on the surface, the LNOI wafer was first washed with acetone and then rinsed with isopropanol. The entire cleaning process was conducted in an ultrasonic bath. A 40-nm-thick Cr layer, acting as a conducting layer for FIB milling, was further deposited on the LN surface by electron beam evaporation. The good electronic conductivity of the Cr layer ensures a controllable and precise FIB fabrication. Both the Cr and LN layers were milled through and patterned into the nano-grating arrays by the ${\mathrm{Ga}}^{+}$ ion beam with a current of 80 pA at an acceleration voltage of 30 kV. The patterned Cr layer served as a protective hard mask in the subsequent ICP-RIE process. This PE process employed a reactive sulfur hexafluoride (${\mathrm{SF}}_6$) gas mixed with physical bombardment argon (Ar) gas. Since the etching rate of the Cr layer is much lower than that of the LN layer, the sidewalls of the LN nano-grating ridges that were unprotected by the Cr mask were selectively etched, and the LN damaged layer and redeposited layer caused by the ${\mathrm{Ga}}^{+}$ ion beam were removed as well. The sample was further wet treated using the standard clean 1 (SC-1; ${\mathrm{NH}}_4\mathrm{OH}\text{:}{\mathrm{H}}_2{\mathrm{O}}_2\text{:}{\mathrm{H}}_2\mathrm{O}=1\text{:}1\text{:}5$ by volume) for 5 min at a temperature of 80°C to remove lithium fluoride (LiF), which is a chemical by-product of fluorine-based gas and LN generated during the PE process^[41]. Finally, the residual Cr layer was removed completely by chemical wet etching using a standard Cr etchant (Alfa Aesar GmbH) at room temperature for 20 min.

Figure 2(b) gives representative scanning electron microscopy (SEM) images of the metasurface with P_d = 400 nm, where the grating duty cycle defined by w/P_d was fixed at 62.5%. In our experiment, a series of tests were performed, and fabricating parameters were determined carefully to ensure the desired duty cycle. The footprint of the metasurface array was $20\mathrm{\mu m}\times 20\mathrm{\mu m}$. The SEM image in the right panel demonstrates the cross section of the LN nano-ridge, which is an isosceles trapezoid with a top width of 211 nm and a bottom width of 275 nm, hence a slant angle of about 82°, which is larger than that of previously reported metasurfaces^[20,23], implying the high fabrication quality of the samples.

Such nano-grating structures could support the bound states in the continuum (BIC) mode when the incident is polarized along the ridge orientation^[42]. The BIC modes represent the confined states in an open system with an infinite $Q$-factor that cannot interact extensively with radiation channels owing to symmetry incompatibility^[43]. Fundamentally, symmetry protected BIC (SP-BIC) is the inherent property of a wave system, and the discrete mode in the Brillouin zone near the $\mathrm{\Gamma }$ point. However, when structure symmetry is broken or illumination is oblique, the SP-BIC mode would convert into QBIC modes with finite spectral widths and large $Q$-factors, thus observable^[44]. Such QBIC modes are regarded as potential candidates for boosting nonlinear light–matter interactions^[45], electro-optic effects, and usefulness in various applications such as bio-sensing and optical tweezing.

We next study the BIC properties of our nano-grating metasurfaces. We start with the numerical studies. The numerical calculations were conducted using a finite element method (COMSOL Multiphysics software). The actual cross-section profiles of the fabricated ridges depicted in Fig. 2(b) were considered in the simulation modeling. The birefringent refractive indices of LN were taken from ellipsometry measurements, and the optical constant of the ${\mathrm{SiO}}_2$ substrate was set as 1.45. The simulated transmission spectral color map of the metasurface with P_d = 400 nm as a function of the incident angle under $y$-polarized ($y\text{-}\mathrm{pol}$) incidence is presented in Fig. 3(a). Sharp transmission resonances with narrow linewidths are observed around 700 nm, and a nearly vanishing linewidth is observed at the SP-BIC point denoted by the solid white circle at normal incidence. In our experiments, the transmission spectra of the metasurface for $y\text{-}\mathrm{pol}$ incident light were measured in the visible and near-infrared wavelength range using a commercial microspectrophotometer (IdeaOptics Technologies). The experimental and simulated transmission spectra obtained for incident angles of 0°, 5°, 10°, 15°, and 20° are presented in Fig. 3(b). The transmission spectra at normal incidence exhibit a stepwise shape and become relatively flat with nearly 100% transmission in the long wavelength range. This high transmission originates from both the high transparency of LN and the structural design of the grating, which includes geometric parameters satisfying the impedance matching condition for free space, leading to near-zero light reflectance. Here, the photonic SP-BIC mode cannot be directly observed under normal incidence. Agreeing with the simulation prediction depicted by blue dotted line, the experimentally obtained QBIC resonance dips (red solid line) with a non-zero spectral width can be observed for oblique incidence^[22], as indicated by the vertical line. To further elaborate the existence and evolution between the BIC and QBIC modes, we calculated the field maps of the $y$ component of the electric field (${\mathit{E}}_y$) distributed in the $xoz$ plane of the metasurface for different incident angles, as shown in the right panels of Fig. 3(b). The ${\mathit{E}}_y$ distribution at normal incidence exhibits an anti-symmetric profile along the $x$ direction with a node at the center, corresponding to an odd mode parity symmetry. The electromagnetic fields are tightly confined within the LN ridge, exhibiting a minimal radiation loss. However, when the incident angle increases, the confinement of the mode is weakened, and leakage radiation occurs, which corresponds to a gradually broadened spectral width, indicating a declining $Q$-factor. Obviously, both the resonance positions and the line shapes of the spectra exhibit high sensitivity to the incident angle. The peak positions and transmission values of the QBIC resonances as a function of the incident angle are presented in Fig. 3(c). The QBIC resonances of the metasurface shift to longer wavelength and exhibit better intensity contrast for larger incident angles. The measured spectra match reasonably well with the simulated ones, confirming the high quality of the metasurfaces. It is worth pointing out that the spectral discrepancy at QBIC resonance is relatively obvious for the small incident angle. This can be explained by the fact that the resonance is too sharp to be resolved by the spectrometer used. Furthermore, the $Q$-factor value for each resonance is also given within parentheses under each transmission value point. The $Q$-factors are calculated by the ratio between the resonance wavelength $\lambda$ and full width at half-maximum (FWHM, $\mathrm{\Delta }\lambda$) of the resonances following the formula of $Q=\lambda /\mathrm{\Delta }\lambda$. The QBIC resonances gradually broaden as the incident angle increases, leading to decreasing $Q$-factor values. This is caused by the coupling of the SP-BIC mode with the radiative channels of the metasurface, in other words, due to electromagnetic mode leakage, as discussed above.

In conclusion, we have developed a recipe for fabricating the LN metasurfaces using FIB direct milling followed by ICP-RIE. The fabricated ridge cross sections are trapezoidal, and the sidewall angles are larger than 80°. The sharp QBIC spectral resonances are observed under oblique incidence in experiments, which confirms the high quality of the fabricated samples. Our methods can be applied to fabricate the LN metasurfaces with sharp resonances for various applications including optical steering, nonlinear optics, and sensing.