Silicon-based optoelectronics: progress towards large scale optoelectronic integration and applications

Introduction

Infrared focal plane arrays (IRFPAs) have been widely used in infrared detection systems to obtain infrared image. However, because of technical limitations and material defects in the manufacturing process, IRFPAs always usually consist of substantial defective pixels which significantly affect the quality of infrared image in IRFPAs [1]. In order to obtain high-quality infrared images, two major methods have been used, namely improving calibration level in factories or in laboratories, and identifying and compensating defective pixels. The first method requires advanced calibration technology, which will increase the cost considerably and lead to add technical difficulty. On the other hand, the second method is simple and effective, so it is commonly employed.

There are basically two steps to realize defective pixel correction. First of all, identifying the defective pixels in infrared image and generating a defective pixel map are required [2]. Then the replacement process [3,4] is implemented using the defective pixel map with proper algorithms such as the nearest neighbor algorithm [5].

In this paper, we present a novel method based on Matlab to identify the defective pixels both automatically and manually, and then we correct the defective pixels by using proportion-spatial defective pixel replacement (PSDPR) algorithm. The proposed method assumes the values of ratios of neighborhood pixels are nearly equivalent and defective pixels can be replaced with the most approximate pixels. The rest of paper is organized as follows: in Section 2, the definition of defective pixels in infrared image and our proposed techniques of defective pixels detection are discussed; in Section 3, the methodology for PSDPR algorithm is introduced. Our approach utilizes information obtained from several neighboring pixels and closely approximates the original data after the replacement of defective pixels; in Section 4, the result of verification experiment is presented and it shows that our proposed method is able to successfully identify and compensate the defective pixels.

Defective identification

Defective pixels are pretty common in IRFPA resulting from material either material defect in manufacturing or electrical optical effects during operation. Such pixels may result in a high reading, a low reading or a reading varying significantly from normal pixels. Generally speaking, there are two kinds of defective pixels namely dead pixels and hot pixels. The dead pixels generate a minimum gray scale value irrespective of the exposure intensity since the responsivity of such a pixel is<10% of the average responsivity [6]. On the other hand, the hot pixels are dependent on exposure intensity and are always brighter than any other pixel. In the phase of identification, we use the IRFPAs to detect the uniform blackbody at temperature of 80°C, and show the fetched gray value on Matlab [7]. Because the responsivity of defective pixels is obviously different from other normal pixels, there are many pulses in the matched positions [8]. Figure 1 shows the infrared image with the defective pixels simulated on Matlab. After detecting uniform blackbody, we took a photo of the electric light on the ceiling in our laboratory. We can see that there are many defective pixels in the infrared image, as it is showed in Fig. 2.

Fig.1 Infrared image at temperature of 80°C before replacement simulated on Matlab

Full size|PPT slide

Fig.2 Infrared image with defective pixels from IRFPA

Full size|PPT slide

To compensate the defective pixels in an infrared image, the defective pixels must be recognized effectively from a 2D image array. To achieve this goal, we propose a novel identification method which combines the median filtering algorithm [9] with the improved standard deviation algorithm. The proposed method is significantly different from traditional methods.

Improved median filtering algorithm.

Median filtering is a nonlinear filter which preserves sharp changes in signal and works effectively in removing impulsive noise. Therefore, median filtering is usually used to identify defective pixels in IRFPA system. Median filtering can be described as follows:

(1)$$\stackrel{︵}{f}(i,j)=median\{f(x,y)\},$$

where $f(x,y)$ represent gray value of all pixels in the $n\times n$ median filter windows.

For the pixel $f(i,j)$ in the infrared image, if

(2)$$\left|f(i,j)-\stackrel{︵}{f}(i,j)\right|>T.$$

The pixel in the position (i, j) is judged as a defective pixel, where T is threshold playing a decisive role in identifying defective pixels. Therefore, selecting the value of threshold is significantly important in an IRFPA system.

Unlike traditional statistical methods, the value of threshold is determined manually by observation based on Matlab in our median filtering algorithm and assist in more appropriate threshold. We calculate the gray value in original infrared image and process the original image by median filtering, then subtract original image with the filtered image. A final image including defective pixels information will be obtained. Then, the threshold can be determined from the final image by observation.

Improved standard deviation algorithm

The output of normal pixels complies with random distribution around the expectation and approximately suits Gaussian distribution. It can be described as follows:

(3)$$\varphi (v)={\displaystyle \frac{\mathrm{1}}{\mathrm{2}\pi \sigma }}\exp \left(-{\displaystyle \frac{{(v-u)}^{\mathrm{2}}}{\mathrm{2}{\sigma }^{\mathrm{2}}}}\right),$$

where u represents the average gray value at uniform temperature, and $\sigma$ represents the standard deviation of all pixels in the infrared image.

Due to mathematical statistics theory and Gaussian distribution theory, the pixels whose responsivity deviated from expectation within 3σ are considered as normal pixels. Oppositely, the pixels without 3σ are defective pixels. This specific algorithm is showed as follows:

(4)$$\{\begin{array}{l}u={\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^{N}f\left(i,j\right)}}\hfill \\ \sigma =\sqrt{{\displaystyle \frac{1}{M\times N}}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{\left[f\left(i,j\right)-u\right]}^2}}}\hfill \end{array}.$$

Therefore, if $\left|f(i,j)-u\right|>\mathrm{3}\sigma$, the pixel in position (i, j) is judged as a defective pixel. Standard deviation algorithm can easily identify defective pixel. However, the identification result is always affected by temporal noise and non-uniform noise. Therefore, an improved standard deviation algorithm is proposed in this paper.

Because of the information of defective pixels is included during the calculation of $$u$$, the result of standard deviation will not closely meet the threshold. We propose that the evident defective pixels should be kicked out before calculating the standard deviation. The median filtering can be used to achieve this goal. It is described as follows:

(5)$$u={\displaystyle \sum _{i=\mathrm{1}}^M{\displaystyle \sum _{j=\mathrm{1}}^{N}median\{f(i,j)\}.}}$$

Then, the infrared image is processed according to standard deviation theory. Compared with standard deviation algorithm, our proposed method obviously avoids the effect originating from hot pixels.

Combination of our proposed methods

Improved median filtering algorithm and improved standard deviation algorithm have their character respectively. In our IRFPA systems, the defective pixel map 1 is generated from infrared image processed by improved median filtering algorithm and map 2 is created by improved standard deviation algorithm. Then, a new defective pixel map (Fig. 3) is updated combining map 1 with map 2, which guarantees all defective pixels are identified. Furthermore, if there are some defective pixels missed, we can add the pixels to defective map manually by observation based on Matlab assisting in obtaining the final perfect defective pixel map.

Fig.3 Defective pixels map based on Matlab

Full size|PPT slide

Proportion-spatial defective pixel replacement (PSDPR)

The compensation of defective pixel is required after the process of defective pixel identification to improve the quality of infrared image [10]. The replacement algorithm should be designed to replace the defective pixel with the most approximate one in space [11]. The nearest neighbor and linear interpolation are the most common algorithms in practical situations. However, these algorithms have problems for clustered defective pixels. In this paper, a novel method called PSDPR algorithm is proposed to replace the defective pixel. The proposed algorithm utilizes information from many different spatial neighboring pixels. Therefore it is capable of providing better approximate replacement compared with other traditional replacement algorithms [12-14], especially for the clustered defective pixels.

In the PSDPR algorithm, it is assumed that the output values from neighboring pixels are nearly identical. To realize the algorithm, a rectangle sub-window must be defined with the defective pixels being replaced in the center.

According to the previous assumption, an equation can be described as follows:

(6)$${\displaystyle \frac{f_k(i,j)}{f(i,j-k)}}={\displaystyle \frac{f(i-k,j)}{f(i-k,j-k)}},$$

where $f(i,j)$ represents the gray value in the position (i, j) of infrared image. Then the defective pixel can be replaced by the median value of $f_k(i,j)$. The gray value of replacement pixel can be described as follows:

(7)$$f_{replacement}(i,j)=median\{f_k(i,j)\},$$

where $f_{replacement}(i,j)$ is the replacement value of $f(i,j)$.

Equation (6) reflects the related proportion of spatial neighborhood pixels. The median function is designed to filter out the outliers in spatial ratios. Compared with other choices, this proposed method has a great advantage when clustered defective pixels are processed.

Verification experiment and conclusions

The proposed algorithms of defective pixel identification and compensation described in previous sections have been simulated based on Matlab and investigated experimentally. In the simulation and experiments, the size of infrared image is 320×240 pixels. Original images are collected using the uniform blackbody at a temperature of 80°C and 0°C, respectively. Defective pixel maps are obtained by our proposed identification method based on Matlab and the final defective map can be obtained by the combination of defective maps.

After defective pixels identification, the PSDPR algorithm is realized on Matlab and FPGA and the results are showed in Figs. 4 and 5.

Fig.4 Infrared image at temperature of 80°C after replacement simulated on Matlab

Full size|PPT slide

Fig.5 Infrared image after replacement from IRFPA

Full size|PPT slide

From the figures, we can see that the defective pixels around the electric light have disappeared and cluster pixels have been identified and replaced without any error. The simulation on Matlab also shows that the final infrared image is clear without pulse pixel.

In summary, in this paper we proposed a novel method to identify and replace the defective pixels in IRFPA systems. Compared with previous methods, our proposed methods provide defective pixel identification without any error and the best approximate replacement for the defective pixels. We have carried out both simulation and verification experiments to verify our methods and the results show that the defective pixels have been replaced perfectly.

References

[1] Liu,S.P., Feng,J.B., Tian,Y., Zhao, H., Jin,L., Ouyang,B.L., Zhu,J.G., Guo,J.: Thermo-optic phase shifters based on silicon-on-insulator platform: state-of-the-art and a review. Front. Optoelectron. 15(1), 9 (2022)

[2] Seiler,P.M., Georgieva, G., Winzer,G., Peczek,A., Voigt,K., Lischke,S., Fatemi, A., Zimmermann,L.: Toward coherent O-band data center interconnects. Front. Optoelectron. 14(4), 414–425 (2021)

[3] Qiu,C.Y., Xiao,H.F., Wang,L.H., Tian, Y.H.: Recent advances in integrated optical directed logic operations for high performance optical computing: a review. Front. Optoelectron. 15(1), 1 (2022)

[4] Adcock,J.C., Ding,Y.H.: Quantum prospects for hybrid thin-film lithium niobate on silicon photonics. Front. Optoelectron. 15(1), 7 (2022)

[5] Tan,M., Wang,Y.H., Wang,X.Z., Yu, Y., Zhang,X.L.: Circuit-level convergence of electronics and photonics: basic concepts and recent advances. Front. Optoelectron. 15(1), 16 (2022)

[6] Cheng,J.W., Zhao,Y.H., Zhang,W.K., Zhou, H.L., Huang,D.M., Dong,J.J., Zhang,X.L.: A small microring array that performs large complex-valued matrix-vector multiplication. Front. Optoelectron. 15(1), 15 (2022)

[7] Georgieva,G., Mai,C., Seiler,P.M., Peczek, A., Zimmermann,L.: Dual-polarization multiplexing amorphous Si:H grating couplers for silicon photonic transmitters in the photonic BiCMOS backend of line. Front. Optoelectron. 15(1), 13 (2022)

[8] Tao,Z.H., Wang,B., Bai,B.W., Chen, R.X., Shu,H.W., Zhang,X.G., Wang,X.J.: An ultra-compact polarization-insensitive slot-strip mode converter. Front. Optoelectron. 15(1), 5 (2022)

[9] Ma,C., Hou,J., Yang,C.Y., Shi, M., Chen,S.P.: Complete photonic bandgap in a silicon nitride slab by utilizing the effective index difference between polarizations. Front. Optoelectron. 15(1), 20 (2022)

RIGHTS & PERMISSIONS

2022 The Author(s) 2022