Rapid advancements in artificial intelligence, particularly in deep learning^[1], have positioned face recognition^[2–4] as a pivotal technology in biometric identification^[5], widely utilized in security surveillance, mobile device authentication, and financial services. This technology offers a swift and non-invasive mode of identity verification through facial feature analysis. However, the widespread deployment of face recognition systems has raised significant privacy concerns. If leaked or misused, the sensitive biometric data processed by these systems could pose severe threats to individual privacy.

Conventional lens-based face recognition systems provide high-resolution images for accurate recognition while heightening the risk of privacy breaches by inadvertently storing detailed facial images. Therefore, developing face recognition technologies that effectively safeguard personal privacy is crucial, though balancing privacy protection with recognition performance presents considerable challenges.

Privacy-preserving face recognition technologies are categorized into software-level and hardware-level protections. Software-level methods typically employ disturbance, encryption, and other data manipulation techniques. For instance, PEEP^[6] employs differential privacy by first projecting original images onto eigenfaces and then adding noise to enhance privacy. This approach efficiently maintains computational simplicity and inference speed but significantly reduces the accuracy of face recognition. Homomorphic encryption^[7,8] and secure multiparty computation (SMC)^[9,10], despite offering robust privacy protection, tend to incur high latency and substantial computational demands. Recent advancements include methods that subtract residual features^[11], balancing privacy with performance by diminishing sensitive information while preserving identity traits. Although certain software-level methods can offer a degree of privacy protection without compromising image recognition accuracy, they inherently suffer from the risk of data leakage before protection can be applied if the data are captured and stored.

Hardware-level privacy protection avoids the capture of high-resolution images through innovative device designs and imaging techniques, thus mitigating privacy risks at the source. Typical methods include lensless imaging^[12–15] technologies, such as multi-aperture and coded aperture imaging^[16], as well as single-pixel imaging^[17–20]—technologies that eschew traditional optical lenses and employ optical encoding to obscure spatial information, complicating data recovery by potential attackers. In addition to the advantages of privacy protection, lensless imaging excels in the presence of challenging constraints on weight, size, scale, and form factor^[21]. For example, a multi-pinhole camera system^[22] uses artificially designed pinhole masks to capture blurred images for facial recognition. FlatCam^[13,23] employs coded aperture techniques with a flexible binary coded mask to disrupt spatial correlations, thus protecting image privacy. The LOEN network^[24] performs convolution operations in the optical domain with end-to-end optimized coded masks^[25,26], ensuring data privacy while extracting pertinent features. These hardware approaches not only prevent the acquisition of clear high-resolution images but also integrate certain computational capabilities, reducing reliance on subsequent data processing and encryption.

Despite the promising aspects of lensless imaging for privacy protection, these technologies often lag behind lens-based systems in terms of recognition performance, spatial resolution, and light throughput. To address this limitation, we have innovatively integrated a microlens array (MLA) into the LOEN architecture, which is commonly used to concentrate light in a variety of optical imaging systems^[27–31], like light field cameras for 3D reconstruction^[30], resulting in a novel mask-encoded MLA face recognition system, termed the MEM-FR system. At the same time, we have developed a dual cosine similarity-based privacy loss function for our end-to-end training approach to finely balance privacy protection with recognition performance. Compared to the LOEN system, the MEM-FR system attains a 4.0% improvement of recognition accuracy in simulation and approximately 5.0% in physical experiments, which are 95.0% and 92.3%, respectively. Additionally, experimental measurements reveal a 6.5° increase in the field of view, a 4.4-fold enhancement in light throughput, and a 12.2-fold improvement in spatial resolution. Collectively, these findings demonstrate notable enhancements in key performance metrics while ensuring robust privacy protection. By varying the focal length and feature size of the MLA and adjusting the mask distribution, we can achieve different spatial resolutions and perform optical convolutions with varied parameters, enhancing system design flexibility and application generalizability. This approach also opens avenues for other visual privacy protection applications.

In our configuration, optical signals emitted from the scene undergo amplitude modulation by a mask and an MLA before being captured by an image sensor. We conceptualize the system as a linear space-invariant model where the scene decomposes into a series of point light sources, with the image recorded by the sensor representing the linear superposition of these sources’ point spread function (PSF). Mathematically, this convolution process is expressed as g(x,y)=f(x,y)∗h(x,y)+n(x,y),(1)where g(x,y), f(x,y), *, h(x,y), and n(x,y) denote the recorded image’s light intensity distribution, scene’s light intensity distribution, convolution operation, system’s point spread function, and noise term, respectively. Overall, we can characterize the imaging process as an optical convolution between the scene and PSF of the mask-encoded MLA.

After conceptualizing the imaging process as the convolution of the scene with the system’s PSF, the focus shifts to the PSF’s design and implementation. In our previous work, as illustrated in Fig. 1(d) (left), the implementation of the PSF in the LOEN system is based on the principle of light propagation in straight lines, where the CMOS sensor is adjacent to the binary-coded mask, allowing the mask’s pattern to be projected as the system’s PSF. In our proposed method, as illustrated in Fig. 1, an MLA is introduced behind the mask in the LOEN system. Each microlens unit aligns with the mask aperture, facilitating binary encoding of the MLA (either transmitting or blocking light). The PSF captured by the sensor represents a form of dilated convolution^[32–36], which is a convolution operation where the kernel is expanded by inserting spaces (zeros) between its elements, effectively enlarging the receptive field without increasing the number of parameters. Figure 2(b) illustrates the fundamental differences between the optical standard convolution (LOEN system) and dilated convolution (MEM-FR system). Here, the standard convolution is applied with a dilation rate of 1, meaning no spaces between the kernel elements, unlike the spaced elements in the dilated convolution of the MEM-FR system. Contrary to the dilation rate of 2 shown in the figure, the actual dilation rate is 8, determined by the ratio of the microlens unit aperture to the characteristic size of the PSF.

We now discuss the physical mechanisms by which the MEM-FR system achieves higher spatial resolution and greater light throughput compared to the LOEN system. The left region of Fig. 1(d) illustrates the optical path of the LOEN system, where point sources A and B are modulated by the mask and form PSF A′ and B′ on the CMOS sensor. Optical convolution is achieved by sequentially superimposing the PSFs of all point sources according to the convolution step size, which in optical convolution systems determines the spatial resolution. On the image plane, the distance Δ′ between A′ and B′ represents the convolution step size, and thus the spatial resolution is the distance Δ between A and B in the object plane. Δ′ and Δ follow a geometric relationship: ΔΔ′=l1l2,(2)where l1 and l2​ represent the distances from the scene to the mask and from the mask to the CMOS, respectively. To improve spatial resolution (i.e., reduce the distance Δ), several approaches can be adopted, such as decreasing the mask aperture size to reduce the PSF feature size Δ′, or decreasing the ratios of l1 to l2​. However, due to limitations in light throughput and diffraction effects, further reduction of the mask aperture in the LOEN system is not feasible. Increasing l2​ in the LOEN system with a fixed object distance l1 would result in an increase in Δ′, thereby reducing spatial resolution. Consequently, the LOEN system has reached its resolution limit, with an effective scene resolution of only 11×11 based on our experimental setup, making it inadequate for high-resolution scene tasks.

Our MEM-FR system addresses this limitation effectively, as shown in Fig. 1(d) (right). The light-gathering capability of MLA allows for the use of a mask with a larger aperture while still achieving a small PSF feature size Δ′ at the diffraction limit. Additionally, the distance from the mask to the CMOS, l2, is increased from 1 to 14.2 mm (the focal length of MLA) without increasing Δ′, resulting in improved light throughput and spatial resolution. In physical experiments, given the diffraction effects of MLA, the minimum PSF feature size Δ′ in the MEM-FR system is comparable to that of the LOEN system (approximately 30 µm). According to Eq. (1), with Δ′ and l1 constant and l2 increased by a factor of 14.2, the spatial resolution of the scene in the MEM-FR system theoretically improves by 14.2 times, achieving an effective resolution of 160×160. Figure 1(b) shows the process of optical convolution correction, where 33 Δ′ refers to the distance corresponding to 33 steps of the convolution window movement on the sensor (the size of a 5×5 dilated convolution with a dilation rate of 8 is 33×33). The spatial resolution of our MEM-FR system is equal to L/33, and then the effective resolution is equal to the scene size divided by the spatial resolution. Attributable to the increase in mask aperture size (from 30 to 300 µm), the light throughput of the MEM-FR system is also improved, enabling its application in natural scenes. Unlike standard convolution in the LOEN system, the PSF in the MEM-FR system is simulated as a dilated convolution in a neural network, where the increased receptive field of the dilated convolution, for the same number of parameters, enhances the effectiveness of privacy protection.

Figure 3 depicts our system’s end-to-end framework. The scene undergoes dilated convolution through the combined modulation of a mask and an MLA. The convolved features, which are also referred to as encrypted images, captured by the sensor, are fed into the backend electronic neural network for further feature extraction and identity recognition. Thus, our system comprises two parts: an optical dilated convolution layer and a backend electronic neural network, forming an opto-electronic hybrid network. We selected FaceNet^[37] as the backbone of our electronic neural network, a face recognition network based on Inception ResNet V1^[38]. Our system employs an end-to-end optimization approach, jointly training the opto-electronic hybrid network. To achieve optimal results, before the scene enters the entire face recognition network, we employ the MTCNN^[39] for face detection and cropping of the training dataset CASIA-WebFace^[40] and the test dataset LFW^[41,42]. Additionally, we introduce horizontal flipping, rotation, and other data augmentation techniques to enhance the network’s generalization capabilities.

Our optimization goal is to ensure that the optical component of the system extracts and records only a limited set of task-specific features while avoiding the formation of privacy-related features that do not contribute to the task. To strike a better balance between face recognition accuracy and privacy protection in end-to-end optimization, we devised a loss function comprising two components: a task loss function and a privacy loss function. The task loss function employs the commonly used cross-entropy loss, gauging the disparity between the predicted and true distributions. The privacy loss function consists of two parts: a dual cosine similarity loss function and a weight gradient loss function.

Dual Cosine Similarity Loss Function: To prevent the recognition of identity information in encrypted images captured by the sensor, we designed a cosine similarity loss function, defined as the cosine similarity between the encrypted image and the original image. Thus, minimizing the cosine similarity loss increases the distance between the encrypted images and the original, thereby achieving better privacy protection effectiveness. Since a complementary image—defined as one where the sum of corresponding pixel values with another image remains constant—retains complete information of the original image and can be easily reconstructed through simple calculations, relying solely on a single cosine similarity loss does not adequately protect privacy. The dual cosine similarity loss effectively mitigates this issue by separately calculating the cosine similarities of the encrypted image and its complementary image with the original. The greater of these two values is then selected as the optimization loss, thereby preventing the emergence of the complementary image of the original and enhancing privacy protection. This loss function is given as Lcos=max[Ioriginal⊗Icaptured‖Ioriginal‖·‖Icaptured‖,Ioriginal⊗(1−Icaptured)‖Ioriginal‖·‖1−Icaptured‖],(3)where Ioriginal, Icaptured, and ⊗ represent the intensity of scene, intensity of the captured image, and matrix multiplication, respectively.

Weight Gradient Loss Function: A more concentrated weight distribution in the optical dilated convolution layer diminishes the privacy protection effectiveness of the encrypted images. Therefore, we devised a gradient loss function to maximize the spatial gradient magnitude of the dilated convolution weights, promoting a more dispersed distribution of the convolution kernel weights and enhancing privacy protection. Additionally, to prevent weight parameter explosion, we incorporated L2 regularization of the weights in the optical dilated convolution layer. This loss function is given as Lgrad=−∑i,j|∇ki,j|+λ∑i,jki,j2,(4)where λ=0.6, and ki,j and ∇ represent the elements of convolution kernel and gradient operator, respectively.

The overall loss function is structured as follows, with different losses weighted in a ratio of λ1:λ2:λ3=1∶20∶0.02 in our experiments: L=λ1Ltask+λ2Lcos+λ3Lgrad.(5)

The experimental configuration of the MEM-FR system comprises four components: the scene (displayed on a screen), the mask, the MLA, and the CMOS image sensor, as illustrated in Fig. 1(a). The scene on the screen, measuring 14.2 cm in both length and width, corresponds to an effective resolution of 160 pixel × 160 pixel in the MEM-FR system. The CMOS sensor employed is the FLIR BFS-U3-51S5C-BD2, featuring a pixel size of 3.45 µm. The MLA, Thorlabs MLA300-14AR, possesses a feature size of 300 µm and a focal length of 14.2 mm. The mask, fabricated on a chrome-coated glass substrate via photolithography, also has a feature size of 300 µm. The distance from the scene to the mask is set at 42.6 cm; the MLA is positioned immediately behind the mask, with each aperture precisely aligned; the distance from the MLA to the sensor is established at 14.2 mm. For comparative purposes, we configured the LOEN system based on prior research^[24], employing the same scene and CMOS sensor as the MEM-FR system. The mask in the LOEN system, lacking an MLA behind it, features a 30 µm size; the distance from the scene to the mask is likewise 42.6 cm; the distance from the mask to the CMOS sensor is fixed at 1 mm.

We utilized the CASIA-WebFace dataset to train the face recognition model, which comprises 494,414 facial images of 10,575 identities sourced from the Internet. The LFW dataset served to evaluate the trained model’s performance. Adhering to established protocols^[41,42], we conducted tenfold cross-validation to compute average accuracy. The FaceNet model, sourced from prior studies^[39], underwent pre-training data processing for face detection using the MTCNN algorithm. Both the network training and evaluation were executed using PyTorch framework on a server equipped with an Intel Xeon Platinum 6164 CPU at 1.90 GHz and a GeForce RTX 3090 GPU. In the MEM-FR system during training, the scene was resized to 160×160, the optical convolution kernel was set to 5×5, and the dilation rate was established at 8. Theoretically, larger optical convolution kernel sizes and higher dilation rates enhance the privacy protection effectiveness of the captured images, as shown in Fig. 2(c). However, excessively large kernel sizes and high dilation rates also complicate the training of the electronic neural network, leading to suboptimal recognition performance. As illustrated in Fig. 2(a), the privacy protection effectiveness, represented by the PSNR and SSIM, shows a slow decline with increasing kernel sizes, while recognition accuracy sharply declines. This disparity highlights the trade-off between maintaining privacy protection and achieving accurate recognition. Consequently, the 5×5 kernel, expanded to 33×33 as dilated convolution, proves ideal, offering a balanced solution with high recognition accuracy and effective privacy protection.

We present the face recognition performance of both MEM-FR and LOEN systems. Each system was trained using the CASIA-WebFace dataset and subsequently evaluated using the LFW dataset. The convolution kernels, refined through simulation, were implemented on masks for physical experiments; after acquiring the encrypted images from the CASIA-WebFace dataset, the backend electronic neural network underwent fine-tuning, followed by the acquisition of encrypted images from the LFW dataset for evaluation. The simulation and experimental results are depicted in Table 1, illustrating that our proposed MEM-FR system achieves superior accuracy compared to the LOEN system, attributable to the mask-encoded MLA’s enhanced spatial resolution. The results indicate a recognition accuracy of 94.97% in simulation, improved by 4.0%, and 92.33% in physical experiment, improved by 5.0%. Given the presence of environmental noise and forward model discrepancies, a performance gap exists between the simulation and the physical experiment; the MEM-FR system exhibited a 2.6% decrease, and the LOEN system showed a 3.9% reduction, demonstrating that the MEM-FR system’s simulation is more precise and holds greater potential for real-world applications.

In addition to recognition performance, a comparative analysis of the MEM-FR and LOEN systems was conducted in terms of field of view (FOV), light throughput, and spatial resolution. To ensure a valid comparison, both systems utilized masks with identical distributions. The FOV was calculated by measuring the movement range of a point light source within the camera’s dynamic range (0–255); the light throughput was gauged by the light intensity at the center of the PSF under the identical point light source and exposure time settings; the spatial resolution was determined by the convolution step size in the object plane when the optical convolution condition was met. The experimental outcomes are showcased in Table 1. Relative to the LOEN system, the MEM-FR system’s FOV expanded by 6.5°, the light throughput increased by 4.4 times, and the spatial resolution improved by 12.2 times. These results highlight notable improvements across key system performance indicators.

Finally, we conducted real-world face recognition experiments under typical indoor lighting conditions provided by strip lights. We collected 60 facial images of 14 testers using the MEM-FR system, with each tester captured in different poses: center, left, right, up, and down, resulting in 210 pairs of facial images, including 105 pairs of the same identity and 105 pairs of different identities. Employing the fine-tuned MEM-FR model for recognition on the dataset composed of those image pairs, the accuracy was 88.57%, underscoring our system’s application potential in natural environments. In contrast, using the standard model—a baseline face recognition system trained on original images and utilizing the same electronic neural network (based on Inception ResNet V1’s FaceNet) without the optical dilated convolution layer and privacy protection mechanisms of our system—for recognition yielded an accuracy of only 69.05%, significantly lower than our model. Compared to the work using the LOEN system for facial recognition in natural scenes^[43], our system operates effectively under typical indoor lighting conditions without additional facial illumination and offers enhanced privacy protection. Figure 4 displays the actual captured facial images, demonstrating that privacy is also effectively safeguarded.

To further validate the effectiveness of our proposed method in privacy protection, we conducted three experiments including human visual evaluation, standard model evaluation, and image deconvolution evaluation. In Fig. 5(a), we present the original images alongside the simulated and experimental encrypted images captured by the MEM-FR system. The alignment between the simulated and experimental results underscores the effectiveness of our optical convolution model. We evaluated the similarity of these captured images using established metrics: PSNR, SSIM, and learned perceptual image patch similarity (LPIPS). To facilitate comparison with the other metrics, the LPIPS values have been normalized by dividing by their maximum value. As shown in Fig. 5(b), images of different identities under similar lighting conditions exhibited higher PSNR and SSIM values alongside lower LPIPS scores, indicating enhanced similarity. Conversely, images of the same identity depicted under varying lighting conditions displayed lower PSNR and SSIM values and higher LPIPS scores, indicating increased dissimilarity. This variation suggests that environmental factors, such as lighting and external elements like glasses, significantly alter the spatial information processed during optical convolution, leading to human vision’s inability to recognize identities in the captured images.

To further substantiate the efficacy of our privacy protection measures, we processed captured images using both the MEM-FR system and a standard system to obtain embedding features. We then calculated the cosine similarity of these embedding features to assess the identity distinction capabilities of each system. Table 2 displays the results: the MEM-FR system successfully differentiated between the same and different identities by effectively capturing and analyzing identity-specific features within the embeddings. Conversely, the standard system, which lacks specialized training, was unable to reliably discern identities based on the extracted features. Additionally, we directly evaluated the captured images of the LFW dataset using the standard model, achieving a recognition accuracy of 63.10% (significantly lower than the 92.33% accuracy of the MEM-FR system), thereby affirming the effectiveness of our privacy protection.

Finally, we assessed our proposed method through image deconvolution to verify that encrypted images are challenging to reconstruct with high quality. In this phase, we assumed that an attacker had access to a certain number of encrypted images and original image pairs. Given that our system provides hardware-level privacy protection and the system PSF is not readily accessible to attackers, we employed a blind deconvolution verification method. We utilized U-Net^[44] to conduct blinddeconvolution attacks, training the model with 200 and 600 pairs of captured and original images, respectively, using half of each set for training and the other half for evaluation. The reconstructed images achieved 19.26 dB PSNR and 0.55 SSIM with 200 training pairs and 21.84 dB PSNR and 0.65 SSIM with 600 training pairs. The deconvolution outcomes are displayed in Fig. 6, showing that the image reconstruction quality remains low, with key facial features appearing blurred and difficult to recognize. Notably, the recognition accuracy of the standard model on images reconstructed from U-Net trained by 600 pairs was only 69.85%, significantly lower than the 92.33% achieved by our system. This demonstrates that breaching privacy becomes more challenging with our system when limited information is available.

We have developed an innovative privacy-preserving face recognition system based on a mask-encoded MLA, referred to as the MEM-FR system. Through end-to-end optimization and a meticulously designed privacy loss function, the MEM-FR system achieves a balance between privacy protection and recognition performance, yielding favorable results in both simulation and physical experiments. Compared to the original LOEN system, our MEM-FR system exhibits superior spatial resolution, enhances light throughput, and improves recognition performance, while maintaining the advantages of a compact structure, passive operation, and incoherent light. These advantages enable our system to perform privacy-preserving face recognition in natural scenes, highlighting its practical applicability. Our findings indicate that the encrypted images captured by the MEM-FR system are hardly distinguishable from human vision or traditional face recognition models, and they exhibit robust resistance to blind deconvolution, thereby substantiating the system’s efficacy in privacy protection.

The MEM-FR system employs binary amplitude modulation, which limits the complexity of the computational functions it can perform. Future enhancements could involve refining devices and increasing the modulation dimension, such as using more intricate masks for grayscale modulation, allowing for more complex computations at the optical layer, thereby reducing system power consumption and latency. Currently, our optical convolution layer is specialized for specific tasks post-deployment; however, exploring reconfigurable optical devices may facilitate switching between multiple tasks. Furthermore, by customizing parameters of the commercially available MLA, we can further enhance the system’s integration level, spatial resolution, recognition performance, and privacy protection effectiveness. This privacy-preserving approach also opens avenues for other visual privacy protection applications.