Reconstruction algorithm of super-resolution infrared image based on human vision processing mechanism

Abstract

Aiming at solving the problem of low resolution and visual blur in infrared imaging, a super-resolution infrared image reconstruction method using human vision processing mechanism (HVPM) was proposed. This method combined a mechanism of vision lateral inhibition with an algorithm projection onto convex sets (POCS) reconstruction, the improved vision lateral inhibition network was utilized to enhance the contrast between object and background of low-resolution image sequences, then POCS algorithm was adopted to reconstruct super-resolution image. Experimental results showed that the proposed method can significantly improve the visual effect of image, whose contrast and information entropy of reconstructed infrared images were improved by approximately 5 times and 1.6 times compared with traditional POCS reconstruction algorithm, respectively.

Key words： human vision processing mechanism (HVPM); projection onto convex sets (POCS); super-resolution; infrared image; reconstruction algorithm

Introduction

Due to the limitations of unit number of infrared detector array and unit size of detection, spatial sampling frequency cannot meet the requirement of sampling theorem, resulting in low spatial resolution of infrared images, and frequency aliasing phenomenon is serious [ 1]. Therefore, it is always a hot issue on how to improve the spatial resolution of image in digital image processing field. Digital signal processing technology has important application in super-resolution acquisition, because of its economy, efficiency and not involving hardware. Generally speaking, the super-resolution reconstructions are based on either frequency domain or spatial domain. And the most common ones are respectively based on interpolation, wavelet transform, probability theory, set theory, and the most promising include projection onto convex sets (POCS), maximum a posterior probability (MAP) and fusion algorithm of POCS and MAP [ 2]. However, these algorithms do not integrate the human visual characteristics. And currently, there are few researches aiming at super-resolution using human visual characteristics. Thus, this paper proposes a reconstruction algorithm of super-resolution infrared image based on human visual characteristics.

The proposed algorithm first uses visual lateral inhibition mechanism to adaptively filter noise and enhance target edge of low-resolution infrared image sequences, then uses POCS algorithm to reconstruct the super-resolution infrared image. The proposed algorithm has stronger anti-noise property and better visual effect.

Visual information acquisition and processing

POCS image reconstruction algorithm

Vision system is the most important sensory organ, acquiring more than 70% of the human information. The optical signal of external scene reaches retina though cornea, crystalline and vitreous body. Information would be converted from optic signal to electrical signal, and preliminarily processed on the retina, then outputted to lateral geniculate nucleus via optic nerve. Geniculate body itself is a transit structure, which is connected with visual cortex via optic radiation. At last, identification and understanding of objects are completed on visual cortex. The entire process is shown below:

Retina is the first information processing component of visual system, which has five layers of cells (see Fig. 1): photoreceptor cell layer, horizontal cell layer, bipolar cell layer, amacrine cell layer and ganglion cell layer. On the basis of the information processing procedure, the multi-layer network can be divided into two separate stages of information processing, namely the outer plexiform layer and the inner plexiform layer [ 3].

The photoreceptor cells, horizontal cells and bipolar cells constitute the outer plexiform layer through the interconnection of chemical synapses. Similarly, inner plexiform layer is composed of bipolar cells, amacrine cells and ganglion cells.

During the course of two plexiform layers' information processing, first, the horizontal cells receive information from the photoreceptors and transmit it to a number of horizontal cells surrounded by bipolar neurons. Due to the negative feedback effect of the horizontal cells, bipolar cells would form a center-around antagonism which is a general model of concentric antagonistic receptive field; secondly, horizontal cells and amacrine cells play a role of connecting two plexiform layers, which is the physiologic basis of lateral inhibition mechanism.

Due to above reasons, the significance of retinal information processing is: the similar information would be suppressed and weakened and the different information would be enhanced, as the XOR operation which removes redundant information and strengthens difference information. So the retina does not only implement the information acquisition, but also the preliminary information processing.

Fig.1 Retina cell layer structure

Full size|PPT slide

Super-resolution method based on simulating retina processing mechanism

Seen from the information processing in the retina, it is obvious that the inhibitory field and lateral inhibition are two important concepts of retinal information process. To gain efficient information enhancement and feature extraction, simulating retinal information processing mechanisms has became a hot spot in the image processing. Inspired by this mechanism, this paper proposes a new method to acquire super-resolution infrared images. The diagram is shown in Fig. 2.

Fig.2 Super-resolution method simulating retina visual processes

Full size|PPT slide

Lateral inhibition mechanism and POCS reconstruction algorithm

Lateral inhibition mechanism and improved lateral inhibition network

Found by Hartline and his colleagues [ 4, 5] for 40 years' research on limulus vision, lateral inhibition is a common phenomenon in sensory systems. The function of lateral inhibition mechanism lies in inhibitory interactions between nerve cells, which make the spatial similar image information to suppress and weaken. Generally, lateral inhibition network is divided into cyclic model and acyclic model. Chen and Ouyang [ 6] pointed out acyclic model is stronger in the ability of synaptic border. So, 2-D subtraction acyclic lateral inhibition model would be used in this paper, the specific expression is described as below [ 7]:

(1) r_{i,j}=e_{i,j}\mathrm{-}{\displaystyle \sum _{p=-N}^N{\displaystyle \sum _{q=-N}^N{k}_{ij,pq}}}\cdot e_{p,q},

where e_{i,j} , r_{i,j} represent sensor's input and output respectively, k_{ij,pq} is sensor unit (p,q) 's inhibition coefficient to sensor unit (i,j) , whose value decreases with increases of the distance between the sensor units, N is the radius of inhibition field which decides the scope of active inhibition, e_{p,q} is the same meaning with e_{i,j} in another style. For example, when the size of lateral inhibition coefficient matrix is 3 × 3, N equals 1. And the lateral inhibition coefficient matrix is shown as below:

(2) \mathit{H}=\left[\begin{array}{lll}\hfill k\left(-\mathrm{1},-\mathrm{1}\right)\hspace{1em}\hfill & \hfill k\left(-\mathrm{1}\mathrm{,}\mathrm{0}\right)\hspace{1em}\hfill & \hfill k\left(-\mathrm{1}\mathrm{,}\mathrm{1}\right)\hfill \\ \hfill k\left(\mathrm{0},-\mathrm{1}\right)\hspace{1em}\hfill & \hfill k\left(\mathrm{0}\mathrm{,}\mathrm{0}\right)\hspace{1em}\hfill & \hfill k\left(\mathrm{0}\mathrm{,}\mathrm{1}\right)\hfill \\ \hfill k\left(\mathrm{1},-\mathrm{1}\right)\hspace{1em}\hfill & \hfill k\left(\mathrm{1}\mathrm{,}\mathrm{0}\right)\hspace{1em}\hfill & \hfill k\left(\mathrm{1}\mathrm{,}\mathrm{1}\right)\hfill \end{array}\right].

Traditional lateral inhibition network enhances the image detail while the image noise is also enhanced; in order to eliminate the image noise, mean filter is used to reduce network’s sensitivity to noise, and its expression is shown as follows:

(3) r_{i,j}={\overline{e}}_{i,j}\mathrm{-}{\displaystyle \sum _{p=-N}^N{\displaystyle \sum _{q=-N}^N{k}_{ij,pq}}}\cdot e_{p,q},

where {\overline{e}}_{i,j} represents the mean filtering of the neighborhood averaging method, lateral inhibition coefficient decides the adjacent sensors' inhibition intensity, and Gauss distribution is used in this paper as lateral inhibition coefficient distribution. And its expression is shown as follows:

(4) k_{ij,pq}=A\cdot {\displaystyle \frac{\mathrm{1}}{\sqrt{\mathrm{2}\pi }\sigma }}\exp \left(-{\displaystyle \frac{d_{ij,pq}^{\mathrm{2}}}{{\sigma }^{\mathrm{2}}}}\right),

(5) d_{ij,pq}=\sqrt{{(i-p)}^{\mathrm{2}}+{(j-q)}^{\mathrm{2}}},

where d_{ij,pq} represents the distance between sensor (i,j) and (p,q) , \sigma decides the decay rate of the lateral inhibition coefficient distribution.

Due to parameters of traditional lateral inhibition coefficient are approximately selected and the template of lateral inhibition coefficient is fixed, thus it is unable to achieve the adaptive adjustment of lateral inhibition ability of different receptor cell under different neurons' stimulus in lateral inhibition network, therefore this paper utilizes the independent strength transmission mode (IDS) [ 8] based on an assumption of lateral human retinal stimulus characteristics proposed to determine the lateral inhibition coefficient adaptively.

(6) \sigma ={\displaystyle \frac{\mathrm{1}}{\sqrt{I(m,n)}}},

(7) k_{ij,pq}=A\cdot {\displaystyle \frac{\mathrm{1}}{\sqrt{\mathrm{2}\pi I(m,n)}}}\exp \left(-{\displaystyle \frac{d_{ij,pq}^{\mathrm{2}}}{I(m,n)}}\right),

where I(m,n) represents the gray value of pixel (m,n) , the selection of \sigma makes the lateral inhibition coefficient change adaptively while the image information changes, A is a constant, in order to guarantee the stability of the lateral inhibition network, and the value of A must meet the sum of the lateral inhibition coefficient matrix less than 1. According to the experience, A is usually 0.38 in image enhancement processing.

Because of gray values of image inhibited by lateral inhibition network, resulting in the average gray value of image decreasing, so it is necessary to compensate the gray values, and the compensation formula is approximated as

(8) G(x,y)={\displaystyle \frac{\mathrm{sum}(f(x,y))}{\mathrm{sum}(g(x,y))}}\cdot g(x,y),

where f(x,y) is the original image, g(x,y) is the image after lateral inhibition network processing, G(x,y) is the compensated image, \mathrm{sum}(\cdot ) is the sum of image pixels' gray value. Equation (8) expands the contrast of image.

POCS image reconstruction algorithm

POCS algorithm [ 9, 10] refers to an alternative iterative method to incorporating prior knowledge about the solution into the reconstruction process based on set theory. With the estimates of registration parameters, this algorithm simultaneously solves the restoration and interpolation problems to reconstruct the super-resolution(SR) image.

In the process of POCS image reconstruction, the first step is to formulate an observation model to obtain low-resolution (LR) images. As is known to all, the observed LR images result from warping, blurring, and sub-sampling operators performed on the high-resolution (HR) image f . Assuming that each LR image is corrupted by additive noise, the common model of acquiring LR image is shown as follows:

(9) g_l(m,n)={\displaystyle \sum _{(x,y)}f(x,y)h_l(m,n;x,y)}+\eta (m,n),

where g_l refers to lth frame LR image, and f(x,y) represents original HR image, then h_l(m,n;x,y) refers to space point spread function (PSF) and \eta (m,n) is the additive noise.

According to the method of POCS, adding a prior knowledge (like limited energy, positive definiteness and bounded support, etc) into the solution can be interpreted as restricting the solution to be a member of a closed convex set C_i that are defined as a set of vectors which satisfy a particular property. If the constraint sets have a nonempty intersection, a solution that belongs to the intersection set C_s={\cap }_{i=\mathrm{1}}^m{C}_i which is also a convex set, can be found by alternating projections onto these convex sets after a limited number of iterations. The method of POCS can be applied to find a vector, which belongs to the intersection by the recursion.

(10) x^{n+\mathrm{1}}=p_m{p}_{m-\mathrm{1}}\cdots p_{\mathrm{2}}{p}_{\mathrm{1}}{x}^n,

where x^{\mathrm{0}} is for an arbitrary starting point, and p_i is projection operator which projects discretionary x onto convex sets C_i .

And the consistency constraint set based on observation model (Eq. (9)) can be expressed for any pixel point of LR image g_l(m,n) as follow:

(11) C_{m,n}=\{x(i_{\mathrm{1}},i_{\mathrm{2}}):r^y(m,n)\le {\delta }_{\mathrm{0}}\},\hspace{1em}\mathrm{0}\le m\le M-\mathrm{1},\mathrm{0}\le n\le N-\mathrm{1},

(12) r^{(y)}(m,n)=g(m,n)-{\displaystyle \sum _{i_{\mathrm{1}}=\mathrm{0}}^{M-\mathrm{1}}{\displaystyle \sum _{i_{\mathrm{2}}}^{N-\mathrm{1}}x(i_{\mathrm{1}},i_{\mathrm{2}})}}h(m,n;i_{\mathrm{1}},i_{\mathrm{2}}),

where r^{(y)}(m,n) refers to projection residuals and x(i_{\mathrm{1}},i_{\mathrm{2}}) represents the HR reference image estimated, {\delta }_{\mathrm{0}} represents the confidence of observed results. If the additive noise is Gaussian distribution with {\sigma }_v as its variance, its expression is shown as follows:

(13) \delta =_{\mathrm{0}}c{\sigma }_v(c\ge \mathrm{0}).

In each iteration process, the absolute value of difference between the pixel point at (m,n) in observed image is obtained and the corresponding pixel point value in simulative LR imaging process is limited in a boundary set in advance. The projection of any point x(i_{\mathrm{1}},i_{\mathrm{2}}) onto C_{m,n} can be defined as follows:

(14)

If the projection operator is given, the estimated HR image \widehat{f}(x,y) can be obtained by all LR images g(m,n) using finite iterations.

(15) {\widehat{f}}^{i+\mathrm{1}}(x,y)=\tilde{T}{\widehat{f}}^i(x,y),

where, \tilde{T} refers to a collection of all relaxation projection operators related to set C_{m,n} , initial estimation f^{\mathrm{0}}(x,y) is acquired by bilinear interpolation from the reference frame referring to HR grid.

Implementation of the proposed algorithm

As prior knowledge can be added in the reconstruction model conveniently, POCS reconstruction algorithm is very promising in this field with its fast convergent speed. But POCS algorithm is not excellent because of its unsatisfactory ability of highlighting the edges of image and poor anti-noise property [ 11] when it aims at infrared image processing. Thus, this paper proposes an innovative method combining lateral inhibition with POCS. The new scheme not only enhances reconstructed image edges and contrasts, but also improves the anti-noise property significantly. Block diagram and implementation process are as follows.

Algorithmic implementation steps are as follows:

Step 1: Select an image g_k(m,n) from LR image sequence ( g_k(m,n) refers to a LR image numbered k ). Process it with adaptive lateral inhibition and bilinear interpolation. Then use it as reference frame to recover SR image and initial estimation of HR image;

Step 2: Rule iterative time and set initial values as zero;

Step 3: Calculate every frame of LR image;

Step3.1: Process remainder LR images with adaptive lateral inhibition. Then calculate motion estimation between image sequence and reference frame to get corresponding motion tracks of pixel points and movement compensation value of each LR image;

Step 3.2: Use iterative projection to process every pixel in each image;

Step 3.3: Calculate the residual between simulative LR image and the current LR image;

Step 3.4: Correct the pixel value in HR image according to the residual in the previous step;

Step 3.5: Examine if the iterative time equals to the number set;

Step 4: Examine whether all LR images are corrected;

Step 5: End of program.

Experimental results

All the experiments use real infrared images, utilize 2 × 2 micro-scanning mode to obtain four low-resolution images, all the values of gray scale of LR images are the average of 4 pixels value corresponding to the original HR image. This paper selects the mean filter template size of adaptive lateral inhibition for 3 × 3 and the inhibition field template size for 5 × 5. PSF’s support domain size is 3 with the threshold error of 3 and iterative times are 10.

Fig.3 Reconstruction renderings of low contrast infrared image. (a) is the original HR image; (b) is the LR image sequences, whose size is 154 × 114; (c) is the local image of one LR image, which is magnified to the size of original image; (d) is the processed image by bilinear interpolation; (e) is the processed image by traditional POCS reconstruction algorithm; (f) represents processed image by the proposed algorithm

Full size|PPT slide

The reconstruction renderings of low contrast infrared image, whose size is 308 × 228, is shown in Fig. 3, where Fig. 3 shows that bilinear interpolation makes the edge of the image blur. POCS enhances edges better than bilinear interpolation after reconstruction. But its ability to improve the infrared image contrast is not obvious. Owing to joining adaptive lateral inhibition network, the proposed algorithm enhances contrast and edge significantly and the visual effect is the best.

Contrast and information entropy are common indexes of objective evaluation. Contrast represents the difference of gray value between the brightest and the darkest pixels. Usually images are divided into several blocks of 3 × 3, the contrast of each block can be calculated by Eq. (16), the contrast of entire image is the average of contrast of all the blocks, and their expressions are shown as follows:

(16) C_o={\displaystyle \frac{g_{\max }-g_{\min }}{g_{\max }+g_{\min }}},

(17) C={\displaystyle \sum _{o=\mathrm{1}}^L{C}_o}/L,

where g_{\max } and g_{\min } represent the maximum and minimum gray values in block respectively, C_o is the contrast of each block, C is the contrast of entire image, and L refers to the number of blocks in image. When the contrast is improved, the image will be clearer with higher resolution.

Information entropy refers to the amount of information of image. And the bigger the value is, the clearer the image detail is. The formula is shown as follows:

(18) H=-{\displaystyle \sum _{k=\mathrm{1}}^{M}P(k){\log }_{\mathrm{2}}P(k)},

(19) P(k)={\displaystyle \frac{A_k}{N\times N}}\hspace{1em}k=\mathrm{0},\mathrm{1},\mathrm{2},\mathrm{...},N

In this equation, P(k) represents the probability of grayscale k appeared, N\times N refers to the whole number of pixels, and M represents the maximum grayscale of the image.

(20) \mathrm{PSNR}=\mathrm{10}\log \left[{\displaystyle \frac{{\mathrm{255}}^{\mathrm{2}}}{{\displaystyle \frac{\mathrm{1}}{M\mathrm{\times }N}}{\displaystyle \sum _{m=\mathrm{0}}^{M-\mathrm{1}}{\displaystyle \sum _{n=\mathrm{0}}^{N-\mathrm{1}}{\left(x_{mn}-{\widehat{x}}_{mn}\right)}^{\mathrm{2}}}}}}\right]

In the peak signal to noise ratio (PSNR) (Eq. (20)), x_{mn} represents gray value after processing, {\widehat{x}}_{mn} represents original gray value, M\times N represents image resolution.

Table 1 gives the contrast and information entropy of low contrast infrared images in Fig. 3. Seeing from the Table 1, we found that the contrast and the information entropy of reconstructed infrared image increase by approximately 5 times and 1.6 times compared with traditional POCS reconstruction algorithm respectively. The experimental data are consistent with the subjective effect.

Tab.1 Evaluation indexes of the low contrast image after processing

algorithm	image evaluation index
contrast	information entropy
original image bilinear interpolation POCS proposed algorithm	0.0222 0.0265 0.0271 0.1254	3.6401 3.6037 3.6578 5.74

The reconstruction renderings of high noise infrared image whose size of 300 × 232 is shown in Fig. 4, and the size of LR image sequences is 150 × 116. The POCS algorithm enhances the edge of roofs and windows vividly. However, the non-uniformity noise of the infrared image is serious and the vertical stripes of the night sky can be seen in the image. But the proposed algorithm has the ability to suppress noise, at the same time it does well in edge enhancing and contrast improving, because of adding mean filter to the lateral inhibition network as anti-noise factor, the proposed algorithm can obtain better visual effect.

Fig.4 Reconstruction renderings of noise infrared image. (a) is the original HR image; (b) is the LR image sequences, whose size is 150 × 116; (c) is the local image of one LR image; (d) is the processed image by bilinear interpolation; (e) is the processed image by traditional POCS reconstruction algorithm; (f) represents processed image by the proposed algorithm

Full size|PPT slide

As Table 2 illustrates, the contrast, information entropy and PSNR of reconstructed infrared images are respectively improved significantly compared with other reconstruction algorithms. The data are consistent with the subjective effects.

Tab.2 Evaluation index of the high noise image after processsing

algorithm	image evaluation index
contrast	informationentropy	PSNR
original image bilinear interpolation POCS proposed algorithm	0.0550 0.0560 0.0602 0.2290	5.3755 5.3506 5.3913 6.9316	– 33.22 36.45 40.35

Conclusions

In this paper, a super-resolution infrared image reconstruction algorithm based on HVPM was proposed. This new method has the ability to adaptively suppress same background information, enhance edges, improve contrast and information entropy and reduce noise interference in the reconstruction process by combining adaptive lateral inhibition network with POCS reconstruction algorithms. And experimental results showed that the visual effect of the proposed method is better, especially aiming at low contrast and low-SNR infrared image.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 61275099 and 61102131), the Project of Key Laboratory of Signal and Information Processing of Chongqing (No. CSTC2009CA2003), the Natural Science Foundation of Chongqing Science and Technology Commission (Nos. CSTC2010BB2411, CSTC2010BB2398 and CSTC2006BB2373), the Natural Science Foundation of Chongqing Municipal Education Commission (Nos. KJ060509 and KJ080517), the Science and Technique Foundation of Chongqing (CSTC) (No. 2011AB2008), and the Natural Science Foundation of Chongqing University of Posts and Telecommunications (CQUPT) (Nos. A2006-04 and A2006-86).

References

1 Park S C, Park M K, Kang M G. Super-resolution image reconstruction: a technical overview. Signal Processing Magazine, IEEE, 2003, 20(3): 21–36

2 Tang L. Research on multi-frame image super-resolution reconstruction. Dissertation for the Doctoral Degree. Zhejiang: Zhejiang University, 2011

3 Pei Z J. The creation of retinal information processing model. Dissertation for the Doctoral Degree. Tianjin: Tianjin Medical University, 2011

4 Ratliff F, Hartline H K. The responses of Limulus optic nerve fibers to patterns of illumination on the receptor mosaic. The Journal of General Physiology, 1959, 42(6): 1241–1255

PMID

5 Hartline H K. The receptive fields of optic nerve fibers. American Journal of Physiology–Legacy Content, 1940, 130(4): 690–699

6 Hui Chen, Kai Ouyang. Simulation Comparison of Lateral Inhibition Models in Image Enhancement. Journal of System Simulation, 2003, 15(1): 100–103

7 Wang Q, He Y Q, Zhou Y C. The infrared image enhancement based on wavelet transformation and lateral inhibition network. Infrared Technology, 2011, 33(9): 541–544 (in Chinese)

8 Deng G, Cahill L W. An adaptive Gaussian filter for noise reduction and edge detection. In: Proceedings of IEEE Nuclear Science Symposium and Medical Imaging Conference. 1993, 1615–1619

9 Xiao J X. Super-resolution image reconstruction based on POCS algorithm. Dissertation for the Doctoral Degree. Shanghai: Shanghai Jiao Tong University, 2009

10 Li X L, Tao D C. A multi-frame image super-resolution method. Signal Processing, 2010, 90(2): 405–414 (in Chinese)

11 Sutour C, Aujol J F, Deledalle C A. Adaptive regularization of the NL-means: Application to image and video denoising. Hal-00854830, 2013, 2(10): 1–15