• Laser & Optoelectronics Progress
  • Vol. 60, Issue 2, 0200002 (2023)
Xuetao Li1, Yaoxiong Wang2, and Fang Gao1,*
Author Affiliations
  • 1School of Electrical Engineering, Guangxi University, Nanning 530004, Guangxi, China
  • 2Institute of Intelligent Machines, Chinese Academy of Sciences, Hefei 230031, Anhui, China
  • show less
    DOI: 10.3788/LOP212680 Cite this Article Set citation alerts
    Xuetao Li, Yaoxiong Wang, Fang Gao. Review of Image Inpainting Methods[J]. Laser & Optoelectronics Progress, 2023, 60(2): 0200002 Copy Citation Text show less
    RNN based pixel generation method
    Fig. 1. RNN based pixel generation method
    CNN based pixel generation method
    Fig. 2. CNN based pixel generation method
    Structure of traditional auto-encoder[7]
    Fig. 3. Structure of traditional auto-encoder[7]
    Context encoder[40]
    Fig. 4. Context encoder[40]
    Structure diagram of GAN
    Fig. 5. Structure diagram of GAN
    Place2 dataset[74]
    Fig. 6. Place2 dataset[74]
    Paris Street View dataset[75]
    Fig. 7. Paris Street View dataset[75]
    CelebA-HQ dataset[77]
    Fig. 8. CelebA-HQ dataset[77]
    Nvidia Mask dataset[51]
    Fig. 9. Nvidia Mask dataset[51]
    Quick Draw Irregular Mask dataset[80]
    Fig. 10. Quick Draw Irregular Mask dataset[80]
    CategoryMethodYear/SourceContribution
    Exemplar-based texture synthesisEfros et al81999/ICCVNonparametric texture synthesis
    Wei et al92000/ACMGaussian pyramid model based on Markov Random Field
    Efros et al102001/ACMSimple and efficient image quilting technology
    Ballester et al142001/IEEE TransMatching local features by calculating image gradients
    Drori et al112003/ACMComputing confidence for each patch
    Hays et al122007/ACMSearching patchs within external databases
    He et al152014/IEEE TransMatching local features by using the statistics of similar patches
    Exemplar-based structure synthesisBertalmio et al172000/ACMUsing Partial Differential Equation to generate linear structural patches
    Criminisi et al182004/IEEETexture and structure information can be transmitted simultaneously
    Chen et al202020/Laser & Optoelectronics ProgressImproved the priority calculation formula18with the method of refining data items
    Wang212020/Laser & Optoelectronics ProgressOptimized the priority calculation formula18by introducing the local color variance
    Chen et al222020/Laser & Optoelectronics ProgressOptimized the priority calculation formula18by introducing the information entropy of measuring the complexity of the pixel block
    Cheng et al192005/IEEEOptimized the priority function in Criminisi18
    Kwatra et al262005/ACM TOGUsing planar exemplar guidance
    Simakov et al232008/IEEEA mathematical model for local restoration of untextured images
    Barnes et al242009/ACMFast stochastic calculation based on NNF
    Ružić et al272014/IEEEGlobal repair algorithm combined with Markov Random Field
    Huang et al252014/ACM TOGUsing planar structure guidance
    Diffusion-based techniquesBertalmio et al172000/ACMDiffusion method based on isophote lines
    Shen et al282002/SIAMCombined total variation denoising model with Partial Differential Equation
    Telea et al292004/JGTFast Marching Method
    Table 1. Summary of traditional methods on image inpainting
    CategoryMethodYear/SourceContribution
    Pixel-generation-based techniquesRichard et al352001/VIIPFast image restoration method based on diffusion convolution kernel(Gaussian)
    Hadhoud et al362008/SIPThe position of zero weight value of diffusion convolution kernel35
    Jain et al372008/NIPSA neural network structure for denoising
    Auto-encoder-based techniquesXie et al392012/NIPSStacked sparse denoising Auto-encoders
    Pathak et al402016/CVPRContext encoder to capture more semantic information
    Iizuka et al412017/ACMGlobal and local context discriminators added to the auto-encoder
    Yu et al442018/CVPRA parallel encoder model based on attention mechanism
    Sagong et al452019/CVPRA shared encoding network with two parallel decoding tasks
    Shin et al462020/NNLSAdaptive dilated convolutional layers added to PEPSI45 model
    Yang et al472017/CVPRMulti-scale neural patch synthesis approach
    Yan et al492018/ECCVA special shift-connection layer Shift-Net
    Liu et al512018/ECCVA partial convolution structure based on U-Net structure
    Xie et al532019/ICCVA learnable bidirectional attention module which can automatically update the mask
    Liu et al542019/ICCVA network architecture based on coherent attention mechanism layer
    Nazeri et al562019/ArxivA two-stage adversarial model EdgeConnect
    Li et al572019/ICCVA progressive reconstruction of visual structure network
    Ren et al582019/ICCVA novel two-stage network which can generate texture structures consistent with context semantics
    Zeng et al622019/CVPRA pyramid context encoder network combining high-level semantics and texture information
    Yi et al632020/CVPRA context residual aggregation network for high resolution image inpainting
    Li et al642020/CVPRA cyclic feature inference network for recovering the large missing regions of damaged images
    GAN-based techniquesRadford et al652015/ArxivDCGANs combining Convolutional Neural Network(CNN)and unsupervised learning
    Isola et al552017/CVPRPatchGAN based on patch identification
    Yeh et al662017/CVPRDGMs to repair irregular regions and capture richer semantics
    Lou et al682018/PRRSRMSProp optimization algorithm is added to WGAN to maintain good performance on non-convex problems
    Yu et al692019/ICCVSN-Patch GAN network based on gated convolution to improve the details and semantic accuracy of repaired results
    Wang et al702021/IEEEThe validity transfer convolution and region compound normalization modules to realize the dynamic selection of valid information
    Zheng et al722019/CVPRPIC NET to generate a variety of repair results
    Zhao et al732020/CVPRAn unsupervised cross-space translation generative adversarial network
    Table 2. Summary of image inpainting methods based on deep learning
    DatasetMethodPSNRSSIMFIDMAEMSESize of imageMask type(image-to-mask ratio)
    CelebA-HQSagong et al4525.600.90256×256Square(25%)
    28.600.92256×256Irregular
    Shin et al4625.500.89256×256Square(25%)
    28.500.92256×256Irregular
    Liu et al5434.690.980.720.04256×256Irregular(10%-20%)
    32.580.980.940.07256×256Irregular(20%-30%)
    25.320.922.180.37256×256Irregular(30%-40%)
    24.140.882.850.44256×256Irregular(40%-50%)
    26.540.931.830.27256×256Square(25%)
    Li et al5733.560.980.007256×256Irregular(10%-20%)
    27.760.930.02256×256Irregular(30%-40%)
    22.880.810.047256×256Irregular(50%-60%)
    Zhao et al7326.380.881.51256×256Irregular(10%-20%)
    Place2Yu et al4418.918.602.10256×256Irregular(10%-20%)
    Liu et al5133.750.940.49256×256Irregular(1%-10%)
    27.710.861.18256×256Irregular(10%-20%)
    24.540.772.07256×256Irregular(20%-30%)
    22.010.683.19256×256Irregular(30%-40%)
    20.340.534.37256×256Irregular(40%-50%)
    18.210.466.45256×256Irregular(50%-60%)
    Xie et al3925.590.781.93256×256Irregular(20%-30%)
    Liu et al5427.750.930.01256×256Square(25%)
    Nazeri et al5621.750.828.163.86256×256Irregular(25%)
    24.920.864.912.59256×256Irregular(20%-30%)
    Li et al6427.750.930.014256×256Irregular(10%-20%)
    22.630.810.038256×256Irregular(30%-40%)
    18.920.590.076256×256Irregular(50%-60%)
    Ren et al5825.220.907.03256×256Irregular(20%-40%)
    Zeng et al620.7815.199.94256×256Square(25%)
    Yi et al634.895.43512×512Irregular
    4.895.431024×1024Irregular
    4.895.492048×2048Irregular
    4.895.504096×4096Irregular
    Paris Street ViewPathak et al4017.590.100.23128×128Square(25%)
    Yang et al4718.000.23128×128Square(25%)
    Yan et al4926.510.900.02256×256Irregular(10%-20%)
    Li et al5731.710.950.011256×256Irregular(10%-20%)
    26.440.860.027256×256Irregular(30%-40%)
    22.400.680.054256×256Irregular(50%-60%)
    Table 3. Quantitative evaluation results of algorithms on common datasets