Real-time electroholography based on a computer-generated hologram (CGH) is considered to realize the ultimate three-dimensional (3D) television experience^[1–4]. However, the enormous amount of CGH computation associated with real-time electroholography prevents its practical realization.

The cost-effective graphics processing unit (GPU) provides high-perfomance computational power for facilitating the implementation of various numerical calculations. GPU-accelerated CGH calculations have been reported^[5–15], such as real-time electroholography using a multiple-GPU (multi-GPU) cluster system comprising many GPU boards^[16–18].

Amplitude-modulation-type CGH is applied to amplitude-modulation-type spatial light modulators (SLMs) such as the digital micromirror device (DMD). Although the light utilization efficiency of amplitude-modulation-type CGH is worse than that of the phase-only-type CGH, a DMD can display CGHs at higher frame rates than the phase-only-type SLM^[19]. A color DMD LED holographic display has been reported^[20]. A DMD holographic image quality enhancement method using localized random down-sampling and adaptive intensity accumulation^[21] and error diffusion^[22,23] has been reported. By utilizing the advantage, wide-viewing-angle holographic displays have been reported^[24–27]. The use of a DMD for gradation-expressible electroholography has also been reported^[28,29].

In our recent work, we proposed gradation-expressible electroholograpy using multiple bit planes consisting of binary-weighted CGHs (BW-CGHs)^[30–32]. The proposed BW-CGHs can freely control the light intensity of reconstructed object points without affecting the brightness of the reconstructed light. However, real-time electroholographic reconstruction of a 3D object with various gradation values had not been achieved.

In this Letter, we propose an efficient and high-speed BW-CGH calculation method for real-time gradation-expressible electroholography using an amplitude-modulation-type SLM. In generating the BW-CGHs as bit planes, we avoid duplicating the calculation of the light intensities of object points with various gradation values.

In this study, we used the conventional binary CGH to reconstruct a bright 3D electroholographic image using an amplitude-modulation-type SLM (i.e., the DMD). A CGH is obtained using the following equation based on the Fresnel approximation: $$I(x_h,y_h,0)=\sum _{j=1}^{N_{\mathrm{obj}}}{A}_j\cos \{\frac{\pi }{\lambda z_j}[{(x_h-x_j)}^2+{(y_h-y_j)}^2]\},$$(1)where $I(x_h,y_h,0)$ is the light intensity of the point ($x_h$, $y_h$, 0) on a CGH. ($x_j$, $y_j$, $z_j$) and $A_j$ are the coordinates and amplitude of the $j$th point of the 3D point cloud model, respectively. In this Letter, we set $A_j$ to one. $N_{\mathrm{obj}}$ is the total number of points in the 3D model. $\lambda$ is the wavelength of the reconstructing light. After calculating Eq. (1), a conventional binary CGH is generated by binarizing the ligh intensity $I$ on a CGH. The corresponding pixel on the binary CGH is drawn in white when the light intensity $I(x_h,y_h,0)$ is greater than zero. Otherwise, it is drawn in black.

As shown in Figure 1, light intensities from a DMD can be controlled using binary pulse-width modulation (PWM) when the digital image with 8 bit depth (256 gradations) is input to a DMD module, as in a projector product^[33]. A binary PWM sequence pattern is used to generate the light intensities from the DMD in proportion to the percentage of time the mirror is “on” during a single-frame-refresh period.

A BW-CGH is generated by changing the white of the conventional binary CGH to the gray with the constant gray-level value from 0 to 255. The BW-CGH can be displayed on a DMD using binary PWM. Figure 1 shows that the display time of the binary CGH can be controlled by the gray-level value of a BW-CGH. The intensity of the light diffracted by the BW-CGH is then weakened compared to that diffracted by the conventional binary CGH. That is, the BW-CGH acts as a neutral density filter for the conventional binary CGH. Figure 2 shows the reconstructions from BW-CGHs and the conventional binary CGH. The object points a, b, and c are reconstructed from two BW-CGHs (BW-CGH1, BW-CGH2) with different gray-level values and the conventional binary CGH, respectively. A higher gray-level value of a BW-CGH results in the reconstructed object point having a higher light intensity.

Object points with various gradation values can be reconstructed when multiple BW-CGHs are used as multiple bit planes. Figure 3 shows an example of the object points expressed in eight gradations. As shown in Fig. 3, object points with gradation values ranging from one to seven are assigned to bit planes $B^0$, $B^1$, and $B^2$, respectively. The bit planes $B^0$, $B^1$, and $B^2$ are obtained from the assigned object points using Eq. (1). The bit planes $B^0$, $B^1$, and $B^2$ are repeatedly displayed on an amplitude-modulation-type SLM, while the reconstructing light illuminates the SLM, thereby reconstructing a 3D-gradation image. Here, the reconstructed image from the $m$th bit plane $B^m$ has a gradation value of $2^m$. For the BW-CGH corresponding to the $m$th bit plane $B^m$, we previously obtain the gradation value $G^m$ of the gray area in an optical experiment. Therefore, the garadation values of the reconstructed 3D image from bit planes $B^0$, $B^1$, and $B^2$ become $2^0$, $2^1$, and $2^2$, respectively.

Figure 4 shows a simple method for calculating the bit planes $B^0$, $B^1$, and $B^2$. In this Letter, we call the method shown in Fig. 4 the duplicate calculation method. For example, the gradation values of the object points P1 to P7 are 1 to 7, respectively. As shown in Fig. 4, the object points P1 to P7 are assigned to the corresponding bit planes $B^0$, $B^1$, and $B^2$, respectively. Lists $B^0$, $B^1$, and $B^2$ include the coordinate data of object points assigned to bit planes $B^0$, $B^1$, and $B^2$, respectively. The bit planes $B^0$, $B^1$, and $B^2$ are calculated from Lists $B^0$, $B^1$, and $B^2$, respectively. The object point P7 is assigned to the bit planes $B^0$, $B^1$, and $B^2$. Therefore, using the method shown in Fig. 4, the light intensity of the object point P7 is calculated three times to generate bit planes $B^0$, $B^1$, and $B^2$. To improve efficiency, a way to avoid this duplicate calculation is desirable.

Figure 5 shows the proposed more efficient calculation method for avoiding duplicate calculation, whereby the object points of the 3D point cloud model are grouped by their gradation values. Object points with gradation values from 1 to 7 are assigned to Groups 1 to 7, respectively, with each object point in the 3D point cloud model belonging to only one group. Figure 6 shows the bit-plane flags of the seven groups, which indicate the bit planes to which the object points are assigned. Figure 5 shows how the program variables are prepared to store the light intensities $I_{B0}$, $I_{B1}$, and $I_{B2}$ of bit planes $B^0$, $B^1$, and $B^2$. In the respective groups, the light intensities are calculated for the appropriate object points using Eq. (1). Here, the light intensity of each object point in the 3D model is calculated only once. The calculated values are then added to the light intensities of the corresponding bit planes based on the bit-plane flags shown in Fig. 6.

For example, in Group 7, the appropriate object points are assigned to bit planes $B^0$, $B^1$, and $B^2$. As shown in Fig. 5, the light intensity of object point P7 is calculated using Eq. (1) just once. The value of the calculated light intensity is then added to light intensities $I_{B0}$, $I_{B1}$, and $I_{B2}$.

After calculating the light intensities of all the groups in the corresponding bit planes, these light intensities are binarized, and then BW-CGHs are generated that correspond to bit planes $B^0$, $B^1$, and $B^2$ from the binarized data.

The total number of floating-point arithmetic operations in Eq. (1) is $N_{\mathrm{fp}}{N}_{\mathrm{obj}}+(N_{\mathrm{obj}}-1)$, where the number of floating-point arithmetic operations per object point is $N_{\mathrm{fp}}$. We estimate the number of the respective floating-point arithmetic operations in the algorithms shown in Figs. 4 and 5. As shown in Fig. 6, the numbers of object points assigned to the bit planes $B^0$, $B^1$, and $B^2$ are $N_{\mathrm{GP}7}+N_{\mathrm{GP}5}+N_{\mathrm{GP}3}+N_{\mathrm{GP}1}$, $N_{\mathrm{GP}7}+N_{\mathrm{GP}6}+N_{\mathrm{GP}3}+N_{\mathrm{GP}2}$, and $N_{\mathrm{GP}7}+N_{\mathrm{GP}6}+N_{\mathrm{GP}5}+N_{\mathrm{GP}4}$, respectively, where the number of the object points belonging to Groups 1 to 7 is $N_{\mathrm{GP}1}$ to $N_{\mathrm{GP}7}$, respectively. In the algorithm of Fig. 4, the number of the floating-point arithmetic operations to generate bit planes is $B^0$, $B^1$, and $B^2$ becomes $N_{\mathrm{fp}}[3N_{\mathrm{GP}7}+2(N_{\mathrm{GP}6}+N_{\mathrm{GP}5}+N_{\mathrm{GP}3})+N_{\mathrm{GP}4}+N_{\mathrm{GP}2}+N_{\mathrm{GP}1}]$. While, in the proposed algorithm of Fig. 5, the number of the floating-point arithmetic operations becomes $N_{\mathrm{fp}}(N_{\mathrm{GP}7}+N_{\mathrm{GP}6}+N_{\mathrm{GP}5}+N_{\mathrm{GP}4}+N_{\mathrm{GP}3}+N_{\mathrm{GP}2}+N_{\mathrm{GP}1})$. Therefore, the speed-up $S$ of the proposed algorithm compared with the algorithm shown in Fig. 4 theoretically becomes $$S=\frac{3N_{\mathrm{GP}7}+2(N_{\mathrm{GP}6}+N_{\mathrm{GP}5}+N_{\mathrm{GP}3})+N_{\mathrm{GP}4}+N_{\mathrm{GP}2}+N_{\mathrm{GP}1}}{N_{\mathrm{GP}7}+N_{\mathrm{GP}6}+N_{\mathrm{GP}5}+N_{\mathrm{GP}4}+N_{\mathrm{GP}3}+N_{\mathrm{GP}2}+N_{\mathrm{GP}1}}.$$(2)

The computational complexity of Eq. (1) becomes $O(N_{\mathrm{obj}}{N}_{\mathrm{cgh}})$, where $N_{\mathrm{cgh}}$ is the total number of pixels on a CGH. The calculation time is enormous, as the BW-CGH is calculated as a bit plane by Eq. (1). To realize real-time gradation-expressible electroholography based on BW-CGHs, we accelerate the BW-CGH calculation using a multi-GPU cluster electroholography system, as shown in Fig. 7. This system consists of a CGH display node with a single SLM and CGH calculation nodes. These nodes are connected via a gigabit ethernet network. In the example shown in Fig. 7, each of the CGH calculation nodes in the system has three GPUs.

The CGH calculation nodes use pipeline processing to accleralate the BW-CGH calculations, as shown in Figure 8. Each of the GPUs from GPU 1 to GPU N generates three BW-CGHs, which become the three bit planes $B^0$, $B^1$, and $B^2$ at each frame of the eight-gradation 3D holographic video. In each frame, the respective symbols from Group 7 to 1 in Fig. 8 indicate the calculations of the light intensities of the object points assigned to the respective groups. In these groups, the calculated values are added to the corresponding light intensities $I_{B0}$, $I_{B1}$, and $I_{B2}$.

In each frame, the calculated light intensities $I_{B0}$, $I_{B1}$, and $I_{B2}$ are binarized and packed to reduce the transfer of data to 1 bit per pixel^[17]. The packed data are then sent to the CGH display node from each of the CGH calculation nodes. The CGH display node receives the packed data in the frame order of the holographic 3D video at the display time interval. GPU 0 on the CGH display node unpacks the packed data to generate three BW-CGHs, which become the bit planes $B^0$, $B^1$, and $B^2$. The eight-gradation 3D holographic video can be reconstructed by the persistence of vision if the high-speed playback of the bit planes $B^0$, $B^1$, and $B^2$ is achieved.

The proposed method requires three BW-CGHs to be displayed on an SLM within a single-frame refresh period. For this reason, we adopted a DMD as the amplitude-modulation-type SLM. Red, green, and blue images can be sequentially displayed on a DMD in a time division manner within a single-frame refresh period when a red–green–blue (RGB) color image is input to a DMD module.

To achieve the high-speed BW-CGH playback, “synthesized RGB BW-CGH” shown in Figure 9 is used as an RGB color image that is input to a DMD module. Figure 9 shows how to make “synthesized RGB BW-CGH.” The respective BW-CGHs corresponding to $B^2$, $B^1$, and $B^0$ were converted into red, green, and blue BW-CGHs. The red, green, and blue BW-CGHs are drawn in red and black, green and black, and blue and black, respectively. The red, green, and blue colors have the same gradation values as the gradation values $G^2$, $G^1$, and $G^0$, respectively. Here, $G^2$, $G^1$, and $G^0$ are the gradation values of the gray areas of the bit planes $B^2$, $B^1$, and $B^0$, respectively. “Synthesized RGB BW-CGH” is synthesized from the red, green, and blue BW-CGHs. “Synthesized RGB CGH” is then input to the DMD module. As shown in the lower time chart of Figure 9, the DMD module automatically separates “synthesized RGB CGH” into the red, green, and blue BW-CGHs, and the respective bit planes $B^2$, $B^1$, and $B^0$ are displayed in order on the DMD panel within a single-frame refresh period. Here, a one-colored light is used as the reconstructing light. The color of the reconstructed 3D image is the same color as the reconstructing light.

To evaluate the performance of the proposed method, we conducted an experiment using a multi-GPU cluster system consisting of a CGH display node with a single GPU board and four CGH calculation nodes. Each CGH calculation node has three GPU boards. Table 1 shows the specifications of the personal computers (PCs) that serve as the nodes in the multi-GPU cluster system.

Figure 10 shows the display-time interval against the number of object points assigned to the respective bit planes $B^0$, $B^1$, and $B^2$. Here, the resolution of the calculated CGH is $1920(\mathrm{pixels})\times 1024(\mathrm{pixels})$. The black and red lines show the single-frame display-time intervals of the conventional electroholography using a binary CGH^[17] and the eight-gradation-expressible electroholography using the proposed method, respectively. In the proposed method, the three bit planes $B^0$, $B^1$, and $B^2$ are transferred at each frame. In Fig. 10, the transfer time per frame of the proposed method is approximately 6.0 ms, which results in a bottleneck, as compared with the calculation time of the three bit planes in the range of 30,720 points or less. In the range of 30,720 points or more, the single-frame display-time interval when using the proposed method falls within about 1.1 times that of using the conventional method^[17] without the gradation expression, although the proposed method requires three BW-CGHs per frame. The proposed method realizes real-time eight-gradation-expressible holographic video reconstruction of a 3D object comprising approximately 180,000 points.

Figure 11 shows the optical setup, for which a 532 nm laser is used as a light source, with objective and collimator lenses used to generate parallel light beams from the light source. We used DLP LightCrafter 6500 EVM (Texas Instruments, micromirror pixel pitch: 7.6 µm, micromirror array size: $1920\times 1080$) as a DMD module. The parallel light is obliquely incident to the DMD panel with an angle of incidence $\theta$, which serves as the reconstructed light. In this experiment, we set $\theta$ to 24°. We placed the 3D point cloud models 1.0 m from the CGH and used a Canon EOS R6 digital camera.

We used “Jack-o’-lantern” and “Stanford bunny” as 3D videos. The 3D videos “Jack-o’-lantern” and “Stanford bunny” have 182,357 to 163,764 and 181,782 to 68,481 points, respectively. Figure 12 shows the gradation values of the 3D videos. The gradation values of the 3D models decrease in the $z$-axis direction from the front to the back. In every frame of the 3D videos, the respective areas with the respective gradation values remain the same, as shown in Fig. 12. The sizes of the reconstructed 3D videos “Jack-o’-lantern” and “Stanford bunny” are $4.5\mathrm{cm}\times 4.5\mathrm{cm}\times 5.0\mathrm{cm}$ and $3.0\mathrm{cm}\times 3.5\mathrm{cm}\times 4.0\mathrm{cm}$, respectively.

Figures 13 and 14 show snapshots of the reconstructed holographic 3D videos “Jack-o’-lantern” (Data File 1) and “Stanford bunny” (Data File 2) using the proposed method. Here, in Data File 1, we set the gradation values $G^2$, $G^1$, and $G^0$ to 170, 110, and 110, respectively. In Data File 2, we set the gradation values $G^2$, $G^1$, and $G^0$ to 170, 40, and 25, respectively.

Table 2 shows the number of object points of 3D models “Jack-o’-lantern” in Fig. 13 and “Stanford bunny” in Fig. 14. In the duplicate calculation method shown in Fig. 4, Lists $B^0$, $B^1$, and $B^2$ assigned to bit planes $B^0$, $B^1$, and $B^2$ are used. From Table 2, the total number of the assigned object points of 3D models “Jack-o’-lantern” and “Stanford bunny” is 295,712 and 128,917, respectively. In the proposed method, the object points assigned Groups 1 to 7 were used. From Table 2, the total number of the assigned object points of 3D models “Jack-o’-lantern” and “Stanford bunny” is 165,600 and 68,489, respectively. From Eq. (2), the respective theoretical speed-up of the 3D models “Jack-o’-lantern” and “Stanford bunny” is 1.79 and 1.88, respectively.

In a PC with a single GPU (Table 1), we compared the performance of the proposed method with that of the duplicate calculation method. Table 3 shows the respective display time intervals of the duplicate calculation method and the proposed method using a PC with a single GPU. In two 3D models, the speed-up of the proposed method exceeds 98% of the theoretical speed-up.

In the multi-GPU cluster system consisting of a CGH display node with a single GPU board and four CGH calculation nodes with three GPUs (Table 1), we compared the performances of the proposed method with that of the duplicate calculation method. Here, in the duplicate calculation method, each frame of the 3D video was assigned to each CGH calculation node. At each frame, the bit planes $B^0$, $B^1$, and $B^2$ were assigned and calculated by the respective GPUs of each CGH calculation node. Table 4 shows the respective display time intervals of the duplicate calculation and the proposed methods using the multi-GPU cluster system. In two 3D models, the speed-up of the proposed method exceeds the theoretical speed-up. In the duplicate calculation method, at each CGH calculation node, all GPUs do not calculate the next frame until they have finished calculating all bit planes $B^0$, $B^1$, and $B^2$ at the present frame. Therefore, the performance of the duplicate calculation method using a multi-GPU cluster system is degraded. In the proposed method, Tables 3 and 4 show that the multi-GPU cluster system with 13 GPUs is about 11 times faster than a PC with a single GPU.

Figure 15 shows the measured light intensities obtained from the snapshots of the reconstructed 3D videos (Data File 1 and Data File 2). Using our proposed method, we realized a reconstructed 3D video expressed in eight gradations.

In our previous work, we used BW-CGHs as bit planes to realize gradation-expressible amplitude-modulation-type electroholography without controlling the intensity of the reconstructed light. In this Letter, we proposed the efficient and high-speed BW-CGH calculation of real-time gradation-expressible electroholography with an amplitude-modulation-type SLM. By generating BW-CGHs as bit planes, the proposed method avoids duplicate calculation of the light intensities of object points with various gradation values.

We implemented the proposed method on a multi-GPU cluster system comprising 13 GPUs and a DMD. Although eight-gradation-expressible electroholography requires three BW-CGHs per frame, using the proposed method, we realized eight-gradation-expressible electroholography at approximately the same calculation speed as conventional electroholography without gradation expression using a binary CGH. Consequently, we were able to successfully reconstruct a real-time electroholographic 3D video comprising approximately 180,000 points expressed in eight gradations at 30 frames per second.