Non-line-of-sight (NLOS) imaging has attracted great attention with its widespread potential applications in object detection, autonomous driving, and anti-terrorist reconnaissance^[1–3]. According to whether a controllable light source is used, NLOS imaging is classified as active NLOS imaging^[4,5] and passive NLOS imaging^[6,7].

Passive NLOS imaging shows promising application and research prospects due to its simple device and convenient data acquisition. However, the NLOS problem is known as an inverse problem in mathematics, and we need to perform blind deconvolution, which is time-consuming and computationally burdened. Consequently, light-cone transform theory using matrix inverse^[8], back projection algorithm based on photon time-of-flight information^[9], and wave-based phasor field approach^[10] are proposed successively. But few of these methods perform well in passive NLOS moving target reconstruction because the steady-state detection mode of passive NLOS suffers from serious degradation of the diffusion spot on the relay surface^[11] and the superposition effect of isotropic diffuse reflection by close pixels^[12]. Currently, speckle coherence restoration^[13] and intensity-based data-driven reconstruction methods^[14,15] are used to solve the ill-posed passive NLOS imaging dilemma. Since the movement of the target induces motion blur to the intensity distribution on the relay surface and superposes with the obscurity caused by diffusion^[16], the current end-to-end deep-learning approach^[11,17] performs well only on static NLOS targets^[18] but shows defects in reconstruction quality for moving targets. In contrast, to realize high quality and efficient reconstruction, we deduce the event form detection-forward model of passive NLOS and establish the event-based inverse problem, on the basis of which we first put forward the event cues for passive NLOS moving target reconstruction. In this way, the dynamic information of the intensity diffusion is precisely captured by the event detection paradigm.

In this section, we present the working principle of the event camera and then explain the inverse problem setup by derivation of the forward model in passive NLOS imaging.

The event camera^[19], known as a novel neuromorphic vision device, only responds to brightness changes per pixel asynchronously, while traditional frame-based cameras measure absolute brightness at a fixed rate. The record paradigm of event-based vision provides high temporal resolution, high dynamic range, and low power consumption^[20]. Therefore, it finds great potential in challenging scenarios for standard cameras, such as high speed, high dynamic range imaging^[21], or object detection^[22] in a slightly changing optical field.

The working principle of the event camera is illustrated in Fig. 1. When the logarithmic intensity of the brightness changes reaches the trigger threshold, the pixel will be triggered and recorded as an event. Each pixel records the logarithmic form of the intensity when an event is fired and continuously monitors for sufficient amplitude changes based on this stored value. The event is described by four characteristic parameters: $\mathit{t}$ describes the time stamp of the fired event, which records the moment when the logarithmic value of the intensity changes exceeds the threshold; $\mathit{x},\mathit{y}$ describes the spatial address of the pixel where the event is fired, respectively; $\mathit{p}$ describes the reason for the excitation of the event, and increase or decrease in brightness, respectively.

Data collected by an event camera are recorded in the form of four characteristic parameters, time stamp $\mathit{t}$, spatial address $\mathit{x},\mathit{y}$, and polarity flag $\mathit{p}$. The $i$th event is noted as ${\mathrm{ev}}_i$, $${\mathrm{ev}}_i={[t_i,x_i,y_i,p_i]}^T,i\in N.$$(1)

The forward model of NLOS is a modeling expression of the light transport in data collecting, which could be regarded as the inverse process of NLOS imaging. It makes up of the theoretical basis of NLOS reconstruction from the scope of steady-state imaging and derivation.

The intensity of the diffusion spot on the relay surface^[23] could be expressed as $$\mathit{I}(p_y)={\iint }_{p_f\in F}\mathit{A}(p_f,p_y)·\mathit{I}(p_f)\mathrm{d}{p}_f+N,$$(2)where $\mathit{I}(p_y)$ represents the intensity distribution in the field of view (FoV) on the relay surface, $\mathit{I}(p_f)$ represents the intensity of the self-luminous target, N represents the summation of the ambient light noise and random noise in detection, and $\mathit{A}(p_f,p_y)$ stands for the point-to-point transmission process^[23], which could be specialized as $$\mathit{A}(p_f,p_y)=\frac{\cos [\angle (\overrightarrow{p_y}-\overrightarrow{p_f},\overrightarrow{n_{pf}})]·\cos [\angle (\overrightarrow{p_f}-\overrightarrow{p_y},\overrightarrow{n_{py}})]}{{\Vert \overrightarrow{p_y}-\overrightarrow{p_f}\Vert }_2^2}·\mu (p_f,p_y),$$(3)where $\angle (\overrightarrow{p_y}-\overrightarrow{p_f},\overrightarrow{n_{pf}})$ and $\angle (\overrightarrow{p_f}-\overrightarrow{p_y},\overrightarrow{n_{py}})$ donate the angle between the vector formed by the monitor pixel and the light source with the normal vector of each of their located plane, correspondingly. The cosine function of these two angles characterizes the effect on optical transmission by geometric relations. The Euclidean norm is used to quantify the attenuation of range. We assume that the relay surface can be approximately modeled as isotropic diffuse reflection, and coefficient $\mu$, which describes the effect of the bidirectional reflectance distribution function (BRDF), is a constant. We substitute Eq. (3) into Eq. (2), and present the pixel-wise detection function on the relay surface^[11], $$\mathit{I}(p_y)={\iint }_{p_f\in F}\frac{\cos [\angle (\overrightarrow{p_y}-\overrightarrow{p_f},\overrightarrow{n_{pf}})]·\cos [\angle (\overrightarrow{p_f}-\overrightarrow{p_y},\overrightarrow{n_{py}})]}{{\Vert \overrightarrow{p_y}-\overrightarrow{p_f}\Vert }_2^2}·\mu ·\mathit{I}(p_f)\mathrm{d}{p}_f+N,$$(4)which could be simplified as $$y=\mathrm{A}f+N.$$(5)

The detection function of diffusion on the relay surface could be written in the form of the matrix, $$\left[\begin{array}{c}{I}_{y,1}\\ ⋮\\ I_{y,h\times w}\end{array}\right]=\left[\begin{array}{ccc}{A}_{1,1}& \cdots & A_{1,H\times W}\\ ⋮& & ⋮\\ A_{h\times w,1}& \cdots & A_{h\times w,H\times W}\end{array}\right]\left[\begin{array}{c}{I}_{f,1}\\ ⋮\\ I_{f,H\times W}\end{array}\right]+N,$$(6)where the intensity of pixel $i$ in the FoV is expressed by the elements $I_{y,i}$, while $I_{f,i}$ represents the intensity of pixel i of the self-luminous target. The transmission matrix A is discretized as ${\mathit{A}}_{i,j}$, and $h\times w$, $H\times W$ are the pixel range for FoV and target, respectively.

In our work, we adopt the event cues to extract the movement information and texture feature of the targets. The most intuitive representation of the dynamic information on the relay surface is the origin event data. However, as addressed in Eq. (1), an event is composed of the triggered time stamp, spatial address, and polarity flag, which belong to three different data formats, respectively. As a result, it is critical to convert the sparse 4D event points into a featured tensor that contains both temporal and spatial characteristics. Therefore, we adopt the time-surface^[24] method to represent the event-based data and extract the featured diffusion spot, which contains rich information on the target movement.

We visualize the event-based data to demonstrate their representations. The relay surface is selected as a mirror for clear visualization. The target “A” is moving from left to right in the FoV, as shown in Fig. 2.

In Fig. 2(b), the frame-based image is captured by a traditional camera, while Fig. 2(c) shows the asynchronous events captured by an event camera, consisting of 3D scatter points, which represent the parameters $x$, $y$, and $t$, in which red stands for positive polarity and blue stands for negative polarity. We select a short time interval as the temporal length of the voxel grid and plot the time-surface 3D map with the time-surface calculation^[24], which contains both temporal and spatial correlations of the selected events, as shown in Fig. 2(e). If we normalize the time-surface value of each spatial address, project it into $xoy$ plane, and display it in the form of gray-scale intensity, we could express the spatiotemporal correlation information in the form of an event address map^[25,26], as shown in Fig. 2(d).

Based on the detection function in Eq. (6), we represent the intensity of pixel $i$ in the FoV at the current moment by elements $I_{y,i}$ in the matrix, while $I_{y,i}^{\prime }$ represent the intensity of pixel $i$ at the center time stamp of the adjacent voxel grid. According to the working principle of the event camera, we establish the event-based detection function by performing a difference operation to $I_{y,i}^{\prime }$ and $I_{y,i}$, as shown, $$\left[\begin{array}{c}{I}_{y,1}^{\prime }-I_{y,1}\\ I_{y,2}^{\prime }-I_{y,2}\\ ⋮\\ I_{y,h\times w}^{\prime }-I_{y,h\times w}\end{array}\right]=A\left[\begin{array}{c}{I}_{f,1}^{\prime }-I_{f,1}\\ I_{f,2}^{\prime }-I_{f,2}\\ ⋮\\ I_{f,H\times W}^{\prime }-I_{f,H\times W}\end{array}\right]+N^{\prime }-N,$$(7)and the intensity of the brightness changes at pixel $i$ is expressed as $$I_{y,i}^{\prime }-I_{y,i}=[\begin{array}{cccc}{A}_{i,1}& A_{i,2}& \cdots & A_{i,H\times W}\end{array}]\underset{\mathrm{\Delta }t\to t_0}{\lim }·\mathrm{\Delta }f+N_r=[\begin{array}{cccc}{A}_{i,1}& A_{i,2}& \cdots & A_{i,H\times W}\end{array}]\left[\begin{array}{c}{I}_{f,1}^m\\ I_{f,2}^m\\ ⋮\\ I_{f,H\times W}^m\end{array}\right]+N_r,$$(8)where $t_0$ is the time interval between two adjacent voxel grids, and we use the limitation form to show the accumulation of the target intensity changes at pixel $i$ during $t_0$, which can be written as $I_{f,i}^m$, containing the movement information on the target. The noise caused by ambient light is also counteracted by the sampling principle of event-based detection, and only random noise remains, i.e., $N_r=N^{\prime }-N$.

Then, we utilize the step function $\text{Fire}(I_y^{\prime }-I_y)$ to discriminate whether an event is fired. As shown in Fig. 3, when the difference value of intensity exceeds the threshold and falls in the red or blue area, an event is recorded at that pixel and outputs $[{\mathit{t}}_{\mathit{i}},{\mathit{x}}_{\mathit{i}},{\mathit{y}}_{\mathit{i}},{\mathit{p}}_{\mathit{i}}]$.

After extracting the dynamic information on the diffusion spot movements, we put forward an event-embedded framework that fuses the extracted event features with a UNet structure to solve the inverse problem. As shown in Fig. 4, we display a video containing a parallel moving target with a smartphone, and leverage the event-based vision to record the dynamic diffusion spot on the relay surface.

We perform a time-surface calculation on the voxel grid^[21] to extract the featured diffusion spot and represent event data at different time intervals by a series of 2D intensity images. Time-surface^[24] is expressed by $$S_i(\rho ,p)=e^{-[t_i-T_i(\rho ,p)]/\tau },$$(9)where $S_i$ is the time-surface value of ${\mathrm{ev}}_i$, which is defined by applying an exponential decay kernel with time constant $\tau$ on the values of the context time stamp $T_i(\rho ,p)$, where $p$ is the polarity flag. $T_i$ is defined to represent the time-context information around an incoming event ${\mathrm{ev}}_i$ as the array of most recent events times at $t_i$ for the neighboring pixels in the adjacent area with the radius of $\rho$, $$T_i(\rho ,p)=\underset{j\le i}{\max }\{t_j|r_j=r_i+\rho ,p_i=p\}.$$(10)

According to the assumption that the fired events of the adjacent voxel grids share a similar spatial address, the spatial address of the fired event stream is highly correlated. As a result, when we perform a time-surface calculation on the acquired event data in a voxel grid, the context information on the movements is accumulated. Then, the temporal and spatial features of the moving target can be expressed by $$\text{Emap}=S[\sum _{y\in \mathrm{FoV},\mathrm{\Delta }t\to t_0}\mathrm{Fire}(I_y^{\prime }-I_y)]=B·f,$$(11)where Fire(·) is referred to in Fig. 3, $S$ stands for the time-surface representation, and $B$ is the counterpart of A.

The event-based inverse problem is established by $$\mathit{f}=B^{-1}·\text{Emap}.$$(12)

A typical solution of this reconstruction is using optimal methods to solve the matrix inverse of $B$, as shown, $$f=\underset{f}{\mathrm{argmin}}{\Vert Bf-\text{Emap}\Vert }_2^2+J,$$(13)where $J$ donates the a priori, which is obtained by the asynchronous sampling paradigm of the event camera.

However, the condition number of matrix $B$ is relatively large due to the pixel-wise mutual interference, resulting in the low rank of matrix $B$. Also, iteratively solving the calculation of $B^{-1}$ is time-consuming. For the last step of solving the inverse problem in NLOS reconstruction, we employ a UNet structure, use skip connection to perform multiscale feature fusion, and add a residual block to avoid gradient explosion when training.

For the experimental proof of the event-based approach, we constructed experimental setups, as shown in Fig. 5. We displayed a video that provided the self-luminous moving target in the NLOS region of the event camera, blocked by the obstacle. The moving diffusion spot on the relay surface is recorded by the event camera (CeleX-V) in Section 1 of the Supplementary Material and Visualization 1.

The targets used in the experiment contain characters of number digits selected from the MNIST training set, MNIST test set, and PRINT test set (Arial font numbers), with the size of $3\mathrm{cm}\times 3\mathrm{cm}$, which are placed 25 cm away from the frosted aluminum fender.

We select 14 different kinds of characters for each digit (0–9) in the MNIST training set and test set, and then acquire both event-based data and frame-based data with different modes of the CeleX-V camera. The self-luminous target displayed by the smartphone translates from left to right in the FoV at a preset speed of 2.5 cm/s. When recording the moving diffusion spot in full-picture (F) mode, we get series of screenshots in different positions with the frame rate of 100 frames per second, while in event-intensity (EI) mode, we get a stream of event-based data of the diffusion spot movement. Data collected by these two modes are calibrated to the ground truth by the time stamp and made into image format data sets event MNIST NLOS (EM-NLOS) and frame MNIST NLOS (FM-NLOS), correspondingly.

To the best of our knowledge, we first established the EM-NLOS data set, which contains 4080 images in the training set and validation set and 210 images in the test set. The training set is made up of 3950 featured events. The time-surface map has 130 targets (13 groups, 0–9) at different positions, while the validation set contains 130 images. The test set consists of 110 images with 10 digits (0–9) selected from the MNIST test set and 100 images with 10 digits (0–9) in Arial font. As the counterpart, FM-NLOS contains the corresponding frame-based intensity diffusion spot movement, with 4180 images in total. We compare the training results on EM-NLOS and FM-NLOS, which are noted as an event-based method (E method) and a frame-based method (F method), respectively. The reconstruction accuracies of the F method and the E method on the MNIST test set and the PRINT test set are shown and compared in Fig. 6.

We trained our residual-UNet (R-UNet) on the EM-NLOS training set with an adaptive moment (Adam) estimation optimizer with Nvidia RTX 3090 GPU for 800 epochs. To achieve fair comparisons, we trained the R-UNet on the FM-NLOS training set with the same configuration^[17]. The structure of our R-UNet and training parameters are given in Section 2 of the Supplementary Material.

The reconstruction quality of the moving target is assessed from two perspectives: the visual reconstruction quality and the position accuracy. For the former, we introduce the peak signal-to-noise ratio (PSNR) and learned perceptual image patch similarity (LPIPS)^[27] to evaluate the reconstructions. The reconstruction accuracies of the F method and the E method on the MNIST test set and the PRINT test set are shown and compared in Figs. 6(a) and 6(b), respectively. It is obvious that the proposed E method with event-based data shows much better reconstruction quality than the F method, especially in recognizing the digits.

As for the position accuracy, we define the index contour distance (Cd) to measure the position of the reconstructions. The Cd value is evaluated by the average distance between the left edge and the digit left contour (made up by first pixels with a gray scale of 255 in each row after image binarization). Digit 7 in the MNIST test set and digit 3 in the PRINT test set are demonstrated as examples in this Letter. As shown in Fig. 7, the reconstruction by the E method is closer to ground truth than that of the F method. The average Cd deviation of the E method is far smaller than that of the F method, as shown in Fig. 8.

Furthermore, the visual reconstruction accuracy of the E method is also intuitively higher than that of the F method. One can see from Fig. 9 that the E method evidently performs better on both of the two metrics, indicating that the event-based approach exceeds the frame-based method under the same data set size and network structure.

We statistically analyze the reconstruction accuracy indexes of 10 digits in both the MNIST test set and the PRINT test set at different positions. The average LPIPS and PSNR of each reconstructed frame for different test digits with the E method and the F method are shown in Table 1, respectively. One can see from Table 1 that the reconstruction LPIPS obtained by the E method is 38% and 11% lower than by the F method on the two test sets, respectively, which demonstrate the higher quality of human perception. As for the PSNR, the E method scores higher than the F method on every test target and performs about 10% better numerically.

As for the generalization, see Section 3 of the Supplementary Material for discussions about target movement and Visualization 2 for the possibility of applying event-based NLOS imaging in real-world circumstances.

In summary, we leverage the sampling specialty of an event camera and propose a new detection and reconstruction method for passive NLOS imaging. The E method extracts rich dynamic information from the diffusion spot movements and provides a physical foundation for passive NLOS imaging of moving targets. Compared with the deep-learning approach with a traditional camera, the event-based framework shows better performance when reconstructing NLOS moving targets. We carry out experiments on two types of targets with different distribution forms and verify that the reconstruction quality is significantly improved with the framework we addressed in both visual accuracy and position accuracy. The reconstruction quality on the PRINT test set indicates that our method has extracted more movement information on moving targets with the event detection paradigm compared with traditional frame-based detection. We believe that the event approach for the inverse problem, together with the EM-NLOS data set, is a big step and can inspire new ideas toward the development of feature-embedded passive NLOS imaging with multidetector information fusion^[12] and NLOS target tracking^[28]. The event cues we demonstrated fuse the event paradigm information of NLOS moving target with end-to-end data-driven methods for solving the event-based inverse problem. In future work, we will take target movements and environment disturbance into consideration, and continually put forward the applications of event-based cues in practice by providing enhancement for methods based on other dimensions of the light field. The event-based vision utilized in this work has great potential to facilitate further research on feature-embedded passive NLOS and its applications.