A new automatic exposure algorithm for video cameras using luminance histogram

Abstract

Automatic exposure (AE) is one of the indispensable functions of modern video cameras. According to the attention mechanism of human visual systems, peak regions in luminance histogram correspond to the region of no interest in an image. Based on this assumption, a new AE algorithm using the luminance histogram of an image is proposed in this paper. The algorithm finds the first two largest peak regions in the histogram and calculates the mean weighted luminance (MWL) of the entire image by weighting the luminance of pixels inside the two peak regions. The MWL is then used to control the exposure of video cameras. The weight of pixel luminance is decided by a set of quadratic curves, and the parameters of the quadratic curves are affected by the brightness of the image background. Fuzzy logic is also applied to optimize the practical AE systems. Results show that the proposed algorithm gives efficient exposure control over various scene tests.

Key words： automatic exposure; luminance histogram; peak regions; weighting curves; fuzzy logic

1 Introduction

Automatic exposure (AE) is an autocontrol process to optimize image luminance according to the characteristics of a human visual system (HVS) by adjusting camera settings. With the popularization of consumer video cameras, AE has become an indispensable function. In some existing AE systems, the average luminance of an image is compared with a reference value to control the exposure. A shooting scene is usually composed of several objects, and all the objects can be reproduced with an appropriate luminance on the screen with this method when the contrast of the objects is low. However, when the contrast of the objects is high, a saturated bright area or a masked dark area will be produced because of the narrow dynamic range of signal on the CCD or CMOS sensor in commonly used video cameras. Specifically, it is difficult to obtain an appropriate image luminance under backlighted or excessive frontlighted condition where considerable luminance difference between the main objects and background exists. In such conditions, the average luminance is dominated by the luminance of background, and improper exposure of the main objects would be produced if the average luminance is employed for the exposure control.

Different kinds of AE algorithms have been developed to perform efficient exposure control especially under backlighted and frontlighted conditions. The method proposed by Haruki and Kikuchi divides a screen into 6 regions and weighs the luminance data of each region to put emphasis on the center of the screen 1. Shimizu et al. have proposed a method in which 3 thresholds of luminance are set up, and discriminations between the backlighted, frontlighted and excessive frontlighted conditions are made based on the ratio of the number of pixels whose luminance are over each threshold 2. Murakami and Honda proposed an exposure control system using color information of an image to have well balanced illumination compensation for both the main object and the background 3.

These AE algorithms have dramatically improved the exposure performance of video cameras, but they are still not suited for any shooting scene. For example, the method giving more weight on the appropriate areas of an image 1 is inevitably influenced by the position of main objects. Without any forms of image partition, the method proposed by Shimizu is not influenced by the position of objects 2. The algorithm can detect the degree of backlighting and frontlighting of an image and perform exposure control accordingly. However, it cannot discriminate the high contrast image from a backlighted image because the proposed HIST parameters mainly represent the degree of contrast in an image. These parameters cannot establish whether the high contrast is caused by backlighting or the objects themselves in the image. As a result, the algorithm cannot perform efficient exposure in such cases.

In some of today's cameras, the image is divided into several regions uniformly, and exposure control is performed according to relations between statistical luminance data of each region and representative shooting scenes 4^,5. However, AE algorithms of this kind must be supported by an image database. Various contrast enhancement techniques are also used to perform more efficient exposure control, and these techniques are employed in image rendering processes 6–9789. The AE algorithm proposed in this paper estimates exposure status by using mean weighted luminance (MWL) based on a luminance histogram and performs exposure control by adjusting image sensor settings of video cameras.

The remainder of this paper is organized as follows. The principle of the proposed algorithm is illustrated in Sect. 2. The proposed AE algorithm and detailed description are presented in Sect. 3. The experimental results are then shown in Sect. 4. Finally we present our conclusion in Sect. 5.

2 Principle

The essence of the algorithm is to decrease the significance of the region of no interests (RONI) on the exposure control. Therefore, the emphasis of exposure is put on the region of interests (ROI) in an image, and the image luminance is optimized. The information about the ROI and the RONI of an image can be obtained from the corresponding luminance histogram.

A representative example of underexposure and the corresponding luminance histogram are illustrated in

Fig0 Underexposure image and its corresponding luminance histogram.(a) Image with improper exposure; (b) its luminance histogram

Full size|PPT slide

Fig. 1. By observation and analysis, the largest and sharpest peak region in Fig. 1(b) corresponds to the sky area in Fig. 1(a). Through the analysis of a number of experimental results, a simple corresponding relation between an image luminance and its histogram is found.

Compared with flat regions in the histogram, peak regions usually contain less gray-levels, and the entropy of signals in the region is relatively small. In other words, peak regions in the histogram correspond to image regions with approximately the same luminance value, which have low local complexity and contain less lines and edges. In terms of frequency, such regions have relatively low frequency components. On the other hand, flat regions in the histogram may contain more gray-levels, i.e., lines and edges, which have high local complexity and high entropy, and correspond to high frequency components in an image. HVS tends to be attentive to image regions with high local complexity 10, and to perceive as much information from an image as possible. As a result, regions with high local complexity become the focus of attention (FOA). Based on the attention mechanism of HVS stated above, here ROI is defined as regions with high local complexity and RONI as regions with low local complexity. Obviously, RONI in an image corresponds to the peak regions in the histogram.

3 Proposed algorithm

3.1 Overview

Fig0 Block diagram of the proposed AE algorithm

Full size|PPT slide

Figure 2 shows the block diagram of the proposed AE algorithm for video cameras. The exposure control loop is based on the MWL of each of the temporal successive images. The histogram calculator finds the first two largest peak regions in the luminance histogram and calculates their size respectively. The background brightness of the shooting scene is determined in the brightness calculator. The fuzzy controller calculates MWL for the current image according to the background brightness and the size of two peak regions. Finally, the exposure control signal is obtained by comparing the MWL with a predetermined reference signal. A detailed description about each module is presented in the following subsections.

3.2 Peak regions

For a convenient description of the proposed algorithm, the peak region in a histogram is defined as an interval with certain width, and the number of pixels in the interval is the local maximum compared with neighboring intervals with the same width. An example of peak region is illustrated in

Fig0 Find peak regions in histogram.(a) Convolving process; (b) convolving results

Full size|PPT slide

Fig. 3(a), as indicated by the hatching area, in which x indicates luminance level.

As shown in Fig. 3(b), peak regions can be easily found by convolving the normalized histogram with a unit gate sequence.

(1)P_{\text{s}}(x)={\displaystyle \sum _{i=x}^{x+P_{\text{w}}-1}h(i)},

where h(i) is the luminance histogram; P_s is the size of a peak region, which is defined as the ratio of the number of pixels in the peak region to that of the entire image; and P_w is the width of the gate sequence, which is a predefined value. Each vertex in Fig. 3(b) corresponds to a peak region in Fig. 3(a). For example, peak 2 in Fig. 3(b) corresponds to the hatching region in Fig. 3(a). In case P_w is given, it is apparent that the P_s of a steeper peak region is larger. To find the RONI in an image with more precision, the appropriate P_w is needed. An excessively large P_w makes it difficult to find RONI with more precision, whereas an extremely small P_w will impair the performance of the proposed AE system. Tests show that when P_w is set to 16 for an image with 256 gray-levels, efficient exposure can be performed for various shooting scenes.

There is usually more than one peak region in the luminance histogram of an image. In case only the largest peak region is weighted, discontinuous change of MWL may occur when the order of the size of the first two largest peak regions alternate, thus the luminance flicker in temporal successive images would be produced. To eliminate the inconvenient effect, both the first two largest peak regions need to be weighted simultaneously.

The left peak region of the first two largest peak regions is denoted as P_bk, which corresponds to the dark area in an image, and the right one is denoted as P_br, which corresponds to a relatively bright area. In Sect. 3.4, the weighting method for the first two largest peak regions, i.e., P_bk and P_br, is investigated.

3.3 Background brightness

The brightness of the shooting scene 11 can be calculated by following equation, and the range of the normalized brightness value is [0, 1]:

(2)B_{\text{r}}=k\Sigma /(GS),

where B_r is brightness of the shooting scene; k is a scaling factor; Σ is the integrated luminance value of the imaging signal; G is gain of the AGC circuit; and S is shutter time of the CCD/CMOS sensor.

Keeping the lighting condition unchanged, the luminance change of temporal successive images is mainly caused by moving objects. In term of human visual perception, the brightness of the shooting scene in the previous image can be considered as background brightness of the shooting scene in the current image. Therefore, the brightness B_{\text{r}}^{i-1} for the (i - 1)^th image calculated by Eq. (2) can be taken as background brightness of the i^th image.

3.4 Weighting curves

Based on the statistical analysis of experimental results, a set of quadratic curves for weighting pixel luminance in peak regions is illustrated in

Fig0 A set of weighting curves(1-B_r = 0 for P_br, B_r = 1 for P_bk; 2-B_r = 0.1 for P_br, B_r = 0.9 for P_bk; 3-B_r = 1 for P_br, B_r = 0 for P_bk)

Full size|PPT slide

Fig. 4. Since large RONI in an image will cause improper exposure of ROI, the weight W is decreased as RONI increases to reduce the significance of RONI during the exposure control. When the image is almost occupied by RONI, assigning a small W for these regions will lead to improper exposure of the entire image. In such a case, W should be increased. Therefore, the weighting curves in Fig. 4 can ensure that the luminance of an image falls into the range of proper exposure when the image is captured under either over or under exposure shooting conditions. The weight W changes with the size of the peak region P_s as

(3)W=a_i+b_i{\left(P_{\text{s}}-c_i\right)}^2.

The parameter c_i is a predetermined value, ranging exclusively from 0.5 to 1.0. Therefore, the weight of the peak region is set to 1 when P_s is small, as the image region corresponding to the peak region in the luminance histogram is not usually a RONI when P_s is small. Even if the peak region with small P_s corresponds to a RONI in the image, the influence on the exposure control is negligible since this RONI comprises a small part of the entire image. The parameter a_i is determined by the background brightness of the shooting scene. In case the background is bright, the image region corresponding to P_br can be considered part of the image background. A small a_i is selected when B_r is relatively large as shown in Fig. 4 to decrease the influence of the RONI on the exposure control, placing the emphasis on the ROI. For the same reason, a small a_i is chosen for P_bk in a dark background. When a_i and c_i are determined, b_i can be solved according to Eq. (3).

Given the weights of the first two largest peak regions as described above, the MWL M_W can be calculated by Eq. (4). The range of M_W is [0, 1]. When both weights take the value of 1, MWL is the ordinary mean luminance value. When both weights take the value of 0, MWL is the mean luminance value of the image excluding the RONI.

(4)M_{\text{W}}=\frac{P_{\text{acc}}-{\displaystyle \sum _{i=1}^2{P}_{\text{acc}i}\left(1-W_i\right)}}{N\left[1-{\displaystyle \sum _{i=1}^2{P}_{\text{s}i}\left(1-W_i\right)}\right]},

where W_i is the weight for each of the two largest peak regions in the histogram; P_si is the size of each peak region in the histogram; N is the size of an image, represented by the number of total pixels; P_acc is the accumulative luminance signal of all pixels in the image; and P_acci is the accumulative luminance signal of pixels in each peak region in the histogram.

(5)P_{\text{acc}}={\displaystyle \sum _{j=0}^{255}jh(j)},

(6)P_{\text{acc}i}\left(x_i\right)={\displaystyle \sum _{j=x_i}^{x_i+P_{\text{w}}-1}jh(j)(i=1,2)},

where x_i denotes the starting position of the peak region, i.e., the left end point of the corresponding interval along the abscissa axis of the luminance histogram.

3.5 Application of fuzzy logic

A set of fuzzy logic rules can be applied to the proposed AE system instead of the weighting curves in Fig. 4 to generate weights for the first two largest peak regions in the luminance histogram. Therefore, continuous and convenient exposure control for temporal successive images can be performed for practical AE systems. Considering the quadratic weighting curves illustrated in Fig. 4, three quadratic membership functions U(1), U(2) and U(3) for P_s, and another three triangle membership functions V(1), V(2) and V(3) for B_r are created as shown in

Fig0 Membership functions forP_s and B_r

Full size|PPT slide

Fig. 5, where D_m is degree of membership.

A set of fuzzy logic rules are also introduced:

(7)\text{rule}\left(1,1\right):\text{if}{P}_{\text{s}}\text{is}S\text{and}{B}_{\text{r}}\text{is}S\text{then}W\text{is}W\left(1,1\right)

(8)\text{rule}\left(1,2\right):\text{if}{P}_{\text{s}}\text{is}M\text{and}{B}_{\text{r}}\text{is}S\text{then}W\text{is}W\left(1,2\right)

(9)\begin{array}{l}\dots \\ \text{rule}\left(3,3\right):\text{if}{P}_{\text{s}}\text{is}B\text{and}{B}_{\text{r}}\text{is}B\text{then}W\text{is}W\left(3,3\right)\end{array}

The weights corresponding to all rules for P_bk and P_br are decided empirically and shown in

Fig0 Weighting values.(a) P_bk; (b) P_br

Full size|PPT slide

Fig. 6. The number of symbol “*” indicates weighting value, for example, “*****” denotes the largest weighting value.

In fuzzy logic reasoning, the degree of membership u(i,j) of rule(i, j) is determined by

(10)u(i,j)=\min \left(P_s\times U(i),B_r\times V(j)\right),1\le i,j\le 3.

The defuzzified weight is obtained by

(11)\begin{array}{l}W={\displaystyle \sum _i{\displaystyle \sum _j(u(i,j)W(i,j))}}/{\displaystyle \sum _i{\displaystyle \sum _{j}u(i,j)}},\\ 1\le i,j\le 3.\end{array}

More continuous and natural weights can be generated for P_bk and P_br by the fuzzy reasoning process described above. Therefore, the proposed AE system can be optimized by the application of fuzzy logic. It should be noted that the basic principle for calculating the weight is the same whether fuzzy logic rules or weighting curves are used.

4 Experimental results

The experiment is carried out by using a video camera with the proposed AE algorithm.

Fig0 Experimental results.(a) Backlighted scene; (b) AE result for (a); (c) frontlighted scene; (d) AE result for (c); (e) ordinary scene; (f) AE result for (e)

Full size|PPT slide

Figure 7 illustrates the experimental results in three representative shooting scenes. Figure 7(a) shows a square under the backlighted condition, and Fig. 7(b) is the exposure control result of the proposed algorithm. It can be observed that the details of buildings and people are clear-cut, and the subjective visual quality is obviously improved. Figure 7(c) is a view outside a supermarket in the dark, and it is under a special kind of frontlighted condition. Applying the proposed AE algorithm, the interior of the supermarket is properly exposed as shown in Fig. 7(d). Figure 7(e) is an image with excellent exposure, and the proposed AE algorithm will not impair the exposure effect, as illustrated in Fig. 7(f).

5 Conclusion

A novel AE algorithm for video cameras is presented in this paper, which uses the MWL of an image to perform exposure control. The corresponding relation between RONI and the largest peak regions in the luminance histogram is revealed in our study, based on which a method for calculating MWL is proposed to assign relatively small weight to the RONI. Therefore, the significance of the RONI can be reduced, and the emphasis is put on the ROI during exposure control. The relation between the weight for and the size of the peak region is described by a set of weighting curves, and curve selection is determined by the brightness of image background. In a practical AE system, fuzzy logic can be used instead of weighting curves to determine the weight and to optimize the system.

Because a histogram was used, the advantage of the proposed algorithm is that the exposure control is not influenced by the position of objects in the captured image. In addition, the algorithm does not need the support of a database and is characterized by easy realization, low complexity and adaptability for various shooting fields. This AE algorithm has also been applied to video phone terminals.

Acknowledgements

The authors would like to express their sincere thanks to Zhao Guangyao and Wei Xiaoxia, who have extended generous help to this development project. This work was supported by the National Natural Science Foundation of China (Grant No. 60772134), the 111 Project of China (Grant No. B08038), and the Special Fund for the United Laboratory of Xidian University and Huawei Technologies Co, Ltd on Multimedia Communication.

References

1 HarukiT, KikuchiK. Video camera system using fuzzy logic. IEEE Transactions on Consumer Electronics, 1992, 38(3): 624–634

2 ShimizuS, KondoT, KohashiT, . A new algorithm for exposure control based on fuzzy logic for video cameras. IEEE Transactions on Consumer Electronics, 1992, 38(3): 617–623

3 MurakamiM, HondaN. An exposure control system of video cameras based on fuzzy logic using color information. In: Proceedings of the Fifth IEEE International Conference on Fuzzy System. New York: IEEE, 1996, 3: 2181–2187

4 SampatN, VenkataramanS, YehT, . System implications of implementing auto-exposure on consumer digital cameras. Proceedings of SPIE. Bellingham: SPIE, 1999, 3650: 100–107

5 XuP, LiZ, SunJ. Study on auto-exposure algorithm based on images analysis. Optical Instruments, 2005, 27(2): 59–61 (in Chinese)

6 StarkJ A. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on Image Processing, 2000, 9(5): 889–896

7 SuM C, GuoJ H, LinD T, . New compensation algorithm for color backlight images. In: Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN'02). New York: IEEE, 2002, 2: 1396–1400

8 NeumannL, MatkovicK, PurgathoferW. Automatic exposure in computer graphics based on the minimum information loss principle. In: Proceedings of Computer Graphics International. New York: IEEE, 1998, 666–677

9 MatkovicK, NeumannL. Interactive calibration of the mapping of global illumination values to display devices. In: Proceedings of the Twelfth Spring Conference on Computer Graphics, Bratslavia, Slovakia. 1996, 155–161

10 KadirT, BradyM. Saliency, scale and image description. International Journal of Computer Vision, 2001, 45(2): 83–105

11 KunoT, SugiuraH, MatobaN. A new automatic exposure system for digital still cameras. IEEE Transactions on Consumer Electronics, 1998, 44(1): 192–199