Introduction
In infrared (IR) thermal imaging systems, the IR image obtained by an IR focal plane array (IRFPA) presents many disadvantages, such as low resolution (LR), blurred edges, and low contrast because of the limited number of IRFPA detector units. Increasing the detector unit density of IRFPA is currently difficult because of the limitations of the existing processing technology. Hence, obtaining an IR super-resolution (SR) image using an SR reconstruction (SRR) algorithm is an effective method [1–4].
Tsai and Huang [5] first proposed a frequency domain method for multi-frame SRR in 1984; this simple and intuitive method requires a small amount of calculation but is sensitive to model errors, thereby limiting its application. SR image reconstruction algorithms based on L1-norm, L2-norm, and L1 and L2 mixed norm have also been proposed. Zhan and Deng [6] used total variation regularization based on L1-norm to reconstruct SR images; the estimation operator based on L1-norm can improve not only the anti-noise performance but also the operation speed of the algorithm. However, the estimated error of L1-norm is higher than that of L2-norm; thus, the image reconstruction results are not as suitable as those of the estimation operator based on L2-norm in the case of Gaussian noise. Yi et al. [7] reconstructed SR images using the L2-norm estimation operator; the edges and image details are properly maintained, but the fitting degree is excessively high. The value of the outliers that appeared is larger than the normal value, and the visual effect of the reconstruction results is poor. L1-norm exhibits high robustness to the gray singular value, but the model estimation error is large. Although L2-norm decreases the estimation error, the estimation operator is still sensitive to the singular value and obtains poor noise immunity. The traditional L1 and L2 mixed norm not only decreases the estimation error but also improves the estimation operator anti-noise performance; however, the traditional algorithm cannot preserve IR image edge information properly when considering the IR image characteristics, such as blurred edges, low contrast, and lack of details [7–10].
The cost function for reconstruction based on the maximum a posteriori (MAP) framework was studied and an algorithm based on L1- and L2-norms that preserves IR image edge information was proposed to remedy the deficiency in the traditional estimate operator. Several aspects of the proposed algorithm are described in this paper as follows. Section 2 discusses the principle and model analysis of L1- and L2-norms. Section 3 explains the proposed algorithm based on L1- and L2-norms. Finally, Section 4 elaborates the SRR based on the proposed method.
Principle and model analysis of L1- and L2-norms
whererepresents the HR image with the size ; represents the nth LR image with the size ; ,, and are sub-sampling, fuzzy, and geometric motion matrixes, respectively; represents all noise types in the sampling process.
SRR was utilized to estimate an HR image with the following known conditions: , , , , and . The ill-posed property of SR image reconstruction can be solved by converting it to a least squares problem. Given the down-sampling model, the cost function was obtained through the following equation:
which represents the L1-norm estimate operator when p equals 1, whereas it represents the L2-norm estimate operator when p equals 2. This equation shows the proximity between the original and estimated HR images; this proximity is known as the fidelity term. High proximity degree means the differences between the original and estimated HR images are small.
The L1-norm estimate operator’s fitting degree between the original and reconstructed images is low and the estimation error is large; however, the L1-norm estimate operator is insensitive to outliers (i.e., its point gray value is irregular). The L2-norm estimate operator’s fitting degree is exceedingly relative, but the values of the outliers that appear are larger than the normal value.
The traditional algorithm combines L1- and L2-norms to decrease the estimation error and improve the anti-noise performance. The cost function based on L1 and L2 mixed norm is shown in Eq. (3).
where the first item in Eq. (3) is the fidelity term, and the second term is the prior term. This study adopts the regularization operator based on the bilateral total variation to correctly preserve the edge information of the image. λ is the regularization parameter that balances the fidelity and prior terms. Ting Li’s algorithm was adopted to select the λ value, which can select the regularization parameter based on the image gradient adaptively [12]. The threshold value µ in Eq. (3) is also manually selected. Thus, the traditional algorithm based on L1 and L2 mixed norm still exhibits defects, such as fuzzy edges and poor anti-noise performance. An improved method based on the above description is proposed in Section 3.
Proposed method
The proposed method uses high frequency layers generated from the given LR IR image to reconstruct the HR IR image. Most of the techniques reported in literature have been applied directly onto the LR IR image [13,14]. Our method was applied instead onto the high frequency layer for constructing the HR high frequency layer. Adopting such an approach adequately preserved the local edge structures in the HR IR image as validated through the experiments provided in Section 4. The constructed HR high frequency layer was then added with the HR IR image reconstructed from the LR image to obtain the final HR IR image. The details of the proposed method are discussed in the following subsections.
Separating high frequency and low frequency information
Equation (1) shows that, for an LR IR image , the corresponding HR IR image is ; their relationship can be expressed as follows:
where is the down-scaled factor, represents the blur matrix (such as a bicubic low-pass filter and Gaussian low-pass filter), is the warp matrix, and is an additive noise in the acquisition process. The additive noise can be ignored in extracting high frequency information. Given that the bicubic interpolation can maintain the low-frequency information of the reconstructed HR image, the first step is to extract high frequency information as follows:
where is the image with a size the same as that of the final HR image, and is up-scaled using bicubic interpolation by the up-scaled factor to obtain . is then down-scaled using linear interpolation by the down-scaled factor ; this factor imitates the degenerative process of the image and further up-scaled two times using bicubic interpolation by the up-scaled factor to obtain (i.e., an image with missing high frequency information). is an image with only high frequency information, which is produced by subtracting from .
Construction of HR high frequency layer and original HR IR image
The proposed cost function based on L1- and L2-norms can be demonstrated in Eqs. (8) and (9):
Equation (8) is the cost function for the LR high frequency layer for obtaining the HR high frequency layer, and Eq. (9)s the cost function for the LR IR image for obtaining the original HR IR image. Separating noise from the original image is difficult because of the frequency distribution of noise in different frequency bands in the actual image sequences. However, dividing the image into high frequency and low-frequency parts shows that the high frequency part contains most of the IR image edge information. Thus, we adopted the L2-norm for the high frequency part to preserve edge information and the L1-norm for the original LR IR image for the purpose of ensuring the overall anti-noise performance of the algorithm.
Construction of final HR IR image
We combined the image (which was processed by the estimation operator based on L2-norm) with the image (which was processed by estimation operator based on L1-norm) to obtain the final HR IR image. The process can be represented as follows:
Experimental results and analysis
The SRR was performed using image sequences, and the effectiveness of the proposed algorithm was verified by comparing with the algorithm based on L1-norm in Ref. [9], L2-norm in Ref. [7], and traditional mixed norm in Ref. [10]. All experiments were performed using MATLAB software on a computer unit with Intel(R) Core(TM)2 E7500 CPU and 4 GB memory. The input LR sequential images were generated from the original HR image through the degradation process for down-sampling, blurring, and movement. Down-sampling by interlaced sampling, blurring by Gaussian fuzzy operator, and the standard deviation of Gaussian fuzzy operator were set to 1.4 in Fig. 1 and 1.6 in Fig. 2. The warping parameters were set to 0.5 (1 represents one pixel spacing) in Fig. 1 and 0.7 in Fig. 2. Reconstructed results are shown in Figs. 1 and 2.
Given that the IR image characteristics, such as LR, blurred edges, and low contrast, we utilized a manual zoom for the local of Figs. 1 and 2 to clearly observe the visual effect after reconstruction. The results are shown in Figs. 3 and 4.
Fig.3 (a) Local magnification of Fig. 1(b) (107×175); (b) local magnification of Fig. 1(c) (107×175); (c) local magnification of Fig. 1(d) (107×175); (d) local magnification of Fig. 1(e) (107×175); (e) local magnification of Fig. 1(f) (107×175); (f) local magnification of Fig. 1(g) (107×175) |
Fig.4 (a) Local magnification of Fig. 2(b) (105×91); (b) local magnification of Fig. 2(c) (105×91); (c) local magnification of Fig. 2(d) (105×91); (d) local magnification of Fig. 2(e) (105×91); (e) local magnification of Fig. 2(f) (105×91); (f) local magnification of Fig. 2(g) (105×91) |
The reconstructed image results are presented in Figs. 1 to 4. The proposed method preserves the edge information and enhances the visual quality of the IR image unlike other methods. The visual effect of the IR images reconstructed by the proposed method was clearly improved with clear edges and rich details.
The proposed method was evaluated to objectively evaluate the reconstructed images using three quantitative quality evaluations: peak signal-to-noise ratio (PSNR) [15], mean gradient (MG) [16], and structural similarity index (SSIM) [15]. The results of the three quantitative evaluations are listed in Tables 1 and 2.
Tab.1 Evaluation results of Fig. 1 |
| images items | Fig. 1(c) | Fig. 1(d) | Fig. 1(e) | Fig. 1(f) | Fig. 1(g) |
|---|---|---|---|---|---|
| PSNR | 22.662 | 23.563 | 24.274 | 24.866 | 25.364 |
| MG | 0.009 | 0.010 | 0.012 | 0.015 | 0.017 |
| SSIM | 0.762 | 0.821 | 0.874 | 0.892 | 0.922 |
Tab.2 Evaluation results of Fig. 2. |
| images items | Fig. 2(c) | Fig. 2(d) | Fig. 2(e) | Fig. 2(f) | Fig. 2(g) |
|---|---|---|---|---|---|
| PSNR | 22.013 | 22.721 | 23.260 | 23.701 | 24.792 |
| MG | 0.023 | 0.027 | 0.030 | 0.034 | 0.040 |
| SSIM | 0.725 | 0.797 | 0.842 | 0.873 | 0.902 |
Table 1 shows that, compared with the traditional algorithm based on bicubic interpolation, L1-norm, L2-norm, and traditional mixed norm, the MG values of the image reconstructed by the proposed algorithm were improved by 1.89, 1.70, 1.42, and 1.13 times, respectively. Table 2 shows that the MG values of the image reconstructed by the proposed algorithm were improved by 1.74, 1.48, 1.33, and 1.18 times. The PSNR and SSIM were also improved to a small degree. The reconstructed results evaluated by a visual subjective evaluation and three quantitative quality evaluations show the effectiveness of the proposed algorithm.
Conclusion
This study presents an effective multi-frame IR image SR algorithm based on high frequency layer features to address the weakness of traditional algorithms based on L1- and L2-norms in SR IR image reconstruction. The proposed algorithm preserves the edge structure information and improves the visual quality of the IR image in the SR process. The experimental results show that the definition and visual effect of the image reconstructed by the proposed algorithm improved and the edges and image details were maintained effectively. Therefore, the proposed algorithm can be used in SR IR image reconstruction.