A primary-secondary background model with sliding window PCA algorithm

Abstract

Rain and snow seriously degrade outdoor video quality. In this work, a primary-secondary background model for removal of rain and snow is built. First, we analyze video noise and use a sliding window sequence principal component analysis de-nosing algorithm to reduce white noise in the video. Next, we apply the Gaussian mixture model (GMM) to model the video and segment all foreground objects primarily. After that, we calculate von Mises distribution of the velocity vectors and ratio of the overlapped region with referring to the result of the primary segmentation and extract the interesting object. Finally, rain and snow streaks are inpainted using the background to improve the quality of the video data. Experiments show that the proposed method can effectively suppress noise and extract interesting targets.

Keywords

sliding window sequence principal component analysis / primary-secondary background model / removal of rain and snow / Gaussian mixture model (GMM)

Introduction

Complicated climate conditions, such as rain, snow, fog, and illumination variation, cause great interference to video analysis, especially for small object detection, recognition, and tracking. Noises produced by camera equipment make it worse. Therefore, many studies involving noise suppression, eliminating weather influence to improve visual effect, and extracting interesting objects have been developed for outdoor video. For example, the methods of object detection for illumination changing [1-3] and fog processing in a space-time field on the pixel level [4] were proposed recently. Also, video processing now focuses on how to erase the scintillation effect caused by physical particles, such as raindrops and snowflakes. Generally, there are two methods to do this: one is that non-interesting objects, such as raindrops and snowflakes described above, are directly viewed as background [5,6]; the other one is that raindrops and snowflakes in the near range are viewed as special foreground objects and filtered. Finally, the corresponding areas are inpainted using the background [7-11].

Stauffer et al. [5] built a Gaussian mixture model (GMM) for whole video, which can train and update its parameters online. The GMM processes raindrops and snowflakes as background, so it does not need any other work. This is because those raindrops and snowflakes dropping quickly close to the camera lens will form rain and snow streaks in an image since the exposure time of a camera is too long compared to the dropping speed of raindrops and snowflakes. GMM cannot distinguish rain and snow streaks from common moving objects, which is the so-called “myopia effect”. Greenspan et al. [6] proposed a piecewise Gaussian mixture model (PGMM) method which clusters active targets in the space-time field of the video feature space. PGMM can detect video events and edit images. It performs well when using PGMM to detect big objects moving slowly, but not so good with rain and snow streaks with high speed. Because it cannot split video to slices perfectly in such a scene, PGMM cannot build an expected model.

Garg et al. [7-9] studied the physical characteristics of raindrops and proposed a rain removal algorithm that has the ability of detecting rain streaks and rendering them as well. Barnum et al. [10,11] transformed an image to the frequency domain through Fourier transformation and simulated rain and snow steaks employing blurred Gaussian streaks in a spatio-temporal streak model. After that, the authors found the frequency range of rain and snow streaks and filtered them finally. Barnum’s model works well despite the sensitivity to the size and orientation of rain and snow streaks. In Ref. [11] the raindrop’s visual features were analyzed thoroughly and a rain removal method by changing the working parameters of a camera was proposed, which can avoid thinking of a complex rain and snow removal algorithm. A slight drawback of that model is that it is difficult to master all the model’s adjusting parameters. Starik et al. [12] tackled the rain and snow streaks as a kind of particular foreground object and removed them using the median filter algorithm whose aliasing effect makes activity targets blurred. Zhang et al. [13] solved the problem in the way of k-means algorithm after an exact image registration and histogram distribution analysis.

The above methods merely focus on rain and snow removal and neglect the fact that video may be polluted by noises. In this paper, a primary-secondary background model is proposed to extract interesting objects from video with noise. In Fig. 1, a sliding window principal component analysis (SWPCA) algorithm is derived, the result of which is a quantity of principal components of image sequences. The current image is projected to the subspace comprised of those principal components to get the feature image. The outputs of SWPCA are input to GMM to perform the primary foreground segmentation and the binary image sequence is obtained containing all moving objects. Finally, the secondary foreground segmentation algorithm is performed according to different motion characteristics of objects to classify them. Thus, we know the space information of rain and snow streaks and use it to remove the streaks to get the purpose of video enhancement.

Principal component analysis (PCA) method for noise suppression

Noises in outdoor video

Cameras working outdoor mostly are made of CCD/CMOS; some noises interfuse with the image sequence while imaging and transmitting, such as photon noise, readout noise, and fixed pattern noise. In addition, the environmental factors, higher or lower temperature, lack of illumination intensity, etc., lead to the image signal-to-noise ratio (SNR) decline. The power spectrum density of noises follows overall a Gaussian distribution, thus they can be reduced as white noise [14].

Image reconstruction by PCA

Let W\times H denote image size, {\mathrm{CH}}_1{,CH}_2,\cdots ,{\mathrm{CH}}_{\mathit{c}} denote c channels. Pixel values at the same position and the same channel in N frames compose the input vector {(x^i)}^T={(x_{\mathrm{1}}^i,x_{\mathrm{2}}^i,\cdots ,x_N^i)}^T, i=\mathrm{1},\mathrm{2},\cdots ,W\times H. For convenience, we omit the superscript i and denote x=x^i-\overline{x}={(x_{\mathrm{1}},x_{\mathrm{2}},\cdots ,x_N)}^T as zero mean vector and its mathematical expectation \overline{x}=E\{x^i\}. The covariance matrix of BoldItalic is

(1){\mathit{C}}_{\mathit{x}}=E\{\mathit{x}{\mathit{x}}^T\}=V\mathrm{\Lambda }{V}^T,

where \mathbf{\Lambda }=\mathrm{diag}({\lambda }_{\mathrm{1}},{\lambda }_{\mathrm{2}},\cdots ,{\lambda }_N)\in {ℝ}^{N\times N}. From Eq. (1) N eigenvalues {\lambda }_{\mathrm{1}}\ge {\lambda }_{\mathrm{2}}\ge \cdots \ge {\lambda }_N and corresponding orthonormal eigenvector matrix \mathit{V}=({\mathit{V}}_{\mathrm{1}},{\mathit{V}}_{\mathrm{2}},\cdots ,{\mathit{V}}_N) of the real symmetric matrix {\mathit{C}}_{\mathit{x}} are calculated, where {\mathit{V}}_j={(v_{j\mathrm{1}},v_{j\mathrm{2}},\cdots ,v_{jN})}^T, j=\mathrm{1},\mathrm{2},\cdots ,N. Let w_T denote the weight of image information. If {\displaystyle {\sum }_{j=\mathrm{1}}^l{\lambda }_j}\ge w_T(l\le N) that lets the first l eigenvectors {\mathit{V}}_{\mathrm{1}},{\mathit{V}}_{\mathrm{2}},\cdots ,{\mathit{V}}_l as the basis of the feature subspace {\mathit{S}}_l. The data of the same channel of the current image are projected to {\mathit{S}}_l through the Hotelling transformation result in the point BoldItalic in subspace is

(2)\mathit{y}={\mathit{S}}_l^T\mathit{x},

and the current image is reconstructed as

(3)\widehat{\mathit{x}}={\mathit{S}}_l^T\mathit{y}.

Using Eq. (3) to all c channels, the whole feature image will then be reconstructed. Because {\mathit{V}}_{l+\mathrm{1}},{\mathit{V}}_{l+\mathrm{2}},\cdots ,{\mathit{V}}_N contain mostly noise and are cast away, it is obviously an offline image reconstruction method.

Images reconstruction by SWPCA

It is known that the calculating cost of offline method is not endured owing to mass data of surveillance video. In this paper, an SWPCA de-noising algorithm is proposed for image reconstruction, which is an online one and based on time-space correlation of video. In SWPCA, outputs of the algorithm are direct inputted to GMM to build a background model. Starting from the first element of the input vector, getting m elements x_{\mathrm{1}},x_{\mathrm{2}},\cdots ,x_m successively to construct a window whose width is m, the window’s principal components are estimated in the window, and the feature image is reconstructed subsequently; when the (m+1)th frame comes, the window slides one frame forward and the window’s width stays invariant, x_{\mathrm{2}},x_{\mathrm{3}},\cdots ,x_{m+\mathrm{1}} is a new window with width m. The above steps are repeated and the principal components are updated while the window slides, so that the principal components always follow the dynamic scene.

Now we formulate the SWPCA as follows.

From Eqs. (1)-(3) current image is reconstructed using the first principal component

(4)\widehat{\mathit{x}}(k)={\mathit{V}}_{\mathrm{1}}{\mathit{y}}_{\mathrm{1}}(k)\Rightarrow {\mathit{y}}_{\mathrm{1}}(k)={\mathit{V}}_{\mathrm{1}}^T\mathit{x}(k)={\displaystyle \sum _{p=\mathrm{1}}^m{v}_{\mathrm{1}p}{x}_p(k)},

where k is the iteration times, m is the window width. Its cost function is

(5)J_{\mathrm{1}}({\mathit{V}}_{\mathrm{1}})=E\left[{\Vert \mathit{x}(k)-{\mathit{V}}_{\mathrm{1}}{\mathit{V}}_{\mathrm{1}}^{\mathrm{T}}\mathit{x}(k)\Vert }_{\mathrm{2}}^{\mathrm{2}}\right],

and the equivalent form is

\{\begin{array}{l}{J}_{\mathrm{1}}({\mathit{V}}_{\mathrm{1}})={\displaystyle \frac{\mathrm{1}}{\mathrm{2}}}{\Vert {\mathit{e}}_{\mathrm{1}}(t)\Vert }_{\mathrm{2}}^{\mathrm{2}}={\displaystyle \frac{\mathrm{1}}{\mathrm{2}}}{\displaystyle \sum _{p=\mathrm{1}}^m{e}_{\mathrm{1}p}^{\mathrm{2}}(t),}\hfill \\ {\mathit{e}}_{\mathrm{1}}(t)=x(t)-{\mathit{V}}_{\mathrm{1}}{\mathit{y}}_{\mathrm{1}}(t),\hfill \\ {\mathit{y}}_{\mathrm{1}}(t)=V_{\mathrm{1}}^T\mathit{x}(t),\hfill \end{array}

where {\mathit{e}}_{\mathrm{1}}(t) is the residual error between origin image and the reconstruction one at time t using the first principal component. Calculating the derivation for both sides of Eq. (5), we can get the result with an iteration step parameter \alpha:

\begin{array}{l}{\displaystyle \frac{d{v}_{1p}}{dt}}=-{\alpha }_1{\displaystyle \frac{\partial J_1\left({\mathit{V}}_1\right)}{\partial v_{1p}}}=-{\alpha }_1\left[{\mathit{y}}_1{e}_{1p}\left(t\right)\right],\hfill \\ p=1,2,\cdots ,m,\hfill \end{array}

and then,

{\displaystyle \frac{d{\mathit{V}}_{\mathrm{1}}}{dt}}={\alpha }_{\mathrm{1}}{\mathit{y}}_{\mathrm{1}}{\mathit{e}}_{\mathrm{1}}={\alpha }_{\mathrm{1}}{\mathit{y}}_{\mathrm{1}}(\mathit{x}-{\mathit{V}}_{\mathrm{1}}{\mathit{y}}_{\mathrm{1}})\mathrm{.}

We have the iterative equation for SWPCA by the first principal component:

(6){\tilde{\mathit{V}}}_{\mathrm{1}}(k+\mathrm{1})={\mathit{V}}_{\mathrm{1}}(k)+{\beta }_{\mathrm{1}}(k){\mathit{y}}_{\mathrm{1}}(k)[\mathit{x}(k)-{\mathit{V}}_{\mathrm{1}}(k){\mathit{y}}_{\mathrm{1}}(k)],

where {\beta }_{\mathrm{1}}(k) denotes the adjustment coefficient. The normalization of Eq. (6) is

(7){\mathit{V}}_{\mathrm{1}}(k+\mathrm{1})={\displaystyle \frac{{\tilde{\mathit{V}}}_{\mathrm{1}}(k+\mathrm{1})}{{\Vert {\tilde{\mathit{V}}}_{\mathrm{1}}(k+\mathrm{1})\Vert }_{\mathrm{2}}}}.

Now, we compute the second principal component using the residual {\mathit{e}}_{\mathrm{1}}(k) as input data:

(8)\{\begin{array}{l}{\mathit{e}}_1\left(k\right)={\mathit{x}}_2\left(k\right)={\mathit{x}}_1\left(k\right)-\widehat{\mathit{x}}\left(k\right)\hfill \\ ={\mathit{x}}_1\left(k\right)-{\mathit{V}}_1\left(k\right){\mathit{y}}_1\left(k\right),\hfill \\ {\mathit{y}}_2\left(k\right)={\mathit{V}}_2^{\mathrm{T}}{\mathit{e}}_1\left(k\right).\hfill \end{array}

Substituting Eq. (8) to Eq. (4), we can get

\widehat{\mathit{x}}(k)={\displaystyle \sum _{p=\mathrm{1}}^m{v}_{\mathrm{2}p}{x}_p(k)\mathrm{.}}

Then, recalculating Eqs. (5)-(8), we have the iteration equation for the second principal component of SWPCA:

(9)$$\begin{gathered} {\tilde{\mathit{V}}}_{\mathrm{2}}(k+\mathrm{1})={\mathit{V}}_{\mathrm{2}}(k)+{\beta }_{\mathrm{2}}(k){\mathit{y}}_{\mathrm{2}}(k)[\mathit{x}(k)-{\mathit{V}}_{\mathrm{2}}(k){\mathit{y}}_{\mathrm{2}}(k)] \\ ={\mathit{V}}_{\mathrm{2}}(k)+{\beta }_{\mathrm{2}}(k){\mathit{V}}_2^T(k){\mathit{e}}_{\mathrm{1}}(k)[\mathit{x}(k)-{\mathit{V}}_{\mathrm{2}}(k){\mathit{V}}_2^T(k){\mathit{e}}_{\mathrm{1}}(k)] \\ ={\mathit{V}}_{\mathrm{2}}(k)+{\beta }_{\mathrm{2}}(k){\mathit{V}}_2^T(k)({\mathit{x}}_{\mathrm{1}}(k)-{\mathit{V}}_{\mathrm{1}}\left(k\right){\mathit{y}}_{\mathrm{1}}(k))\cdot [\mathit{x}(k)-{\mathit{V}}_{\mathrm{2}}(k){\mathit{V}}_{\mathrm{2}}^T(k)({\mathit{x}}_{\mathrm{1}}(k)-{\mathit{V}}_{\mathrm{1}}(k){\mathit{y}}_{\mathrm{1}}(k))]\mathrm{.} \end{gathered}$$

The normalized one is

(10){\mathit{V}}_{\mathrm{2}}(k+\mathrm{1})={\displaystyle \frac{{\tilde{\mathit{V}}}_{\mathrm{2}}(k+\mathrm{1})}{{\Vert {\tilde{\mathit{V}}}_{\mathrm{2}}(k+\mathrm{1})\Vert }_{\mathrm{2}}}}\mathrm{.}

In Eq. (10), as input data, the first principal component of {\mathit{e}}_{\mathrm{1}}(k) is used to calculate the second principal component and we have the second residual error {\mathit{e}}_{\mathrm{2}}(k):

(11)\{\begin{array}{l}{\mathit{e}}_{\mathrm{2}}(k)={\mathit{x}}_{\mathrm{3}}(k)={\mathit{x}}_{\mathrm{2}}(k)-{\mathit{V}}_{\mathrm{2}}(k){\mathit{y}}_{\mathrm{2}}(k),\hfill \\ {\mathit{y}}_{\mathrm{3}}(k)={\mathit{V}}_3^T{\mathit{e}}_{\mathrm{2}}(k).\hfill \end{array}

Repeating the above steps, the first principal component of {\mathit{e}}_{\mathrm{2}}(k) as input data is used to compute the third principal component and so on and so forth, and we have the whole iteration equations of SWPCA for all principal components from Eqs. (4)-(11):

(12)\{\begin{array}{l}{\mathit{V}}_i(k+\mathrm{1})={\displaystyle \frac{{\tilde{\mathit{V}}}_i(k+\mathrm{1})}{{\Vert {\tilde{\mathit{V}}}_i(k+\mathrm{1})\Vert }_{\mathrm{2}}}},\hfill \\ {\tilde{\mathit{V}}}_i(k+\mathrm{1})={\mathit{V}}_i(k)+{\beta }_i(k){\mathit{y}}_i(k)[{\mathit{x}}_i(k)-{\mathit{V}}_i(k){\mathit{y}}_i(k)],\hfill \\ {\mathit{e}}_i(k)={\mathit{x}}_{i+\mathrm{1}}(k)={\mathit{x}}_i(k)-{\mathit{V}}_i(k){\mathit{y}}_i(k),\hfill \\ {\mathit{y}}_i(k)={\mathit{V}}_i^T(k){\mathit{x}}_i(k),\hfill \\ {\mathit{x}}_{\mathrm{1}}(k)=\mathit{x}(k),\hfill \\ i=\mathrm{1},\mathrm{2},\cdots ,m;k=\mathrm{1},\mathrm{2},\cdots .\hfill \end{array}

The traditional PCA applies K-L algorithm to compute the basis of subspace {\mathit{S}}_l^T, in which eigenvalue {\lambda }_i and eigenvector {\mathit{V}}_i of the covariance matrix need a large amount of calculations. The proposed SWPCA gets the first principal component and residual error by the gradient descent algorithm, and then uses the residual error as input data to get the second principal component and so on, avoiding heavy calculation. Furthermore, traditional PCA needs to store all images and SWPCA only stores current m images in the sliding window.

Foreground segmentation algorithms

In this section, we use the feature image sequence obtained by SWPCA to perform two-phase foreground segmentation.

Primary segmentation using GMM

After getting feature subspace {\mathit{S}}_l={({\mathit{V}}_{\mathrm{1}},{\mathit{V}}_{\mathrm{2}},\mathrm{...},{\mathit{V}}_l)}^T from Eq. (12), we can estimate the value of BoldItalic:

(13)\widehat{\mathit{x}}=\overline{\mathit{x}}+{\mathit{S}}_l{\mathit{S}}_l^{\mathrm{T}}\mathit{x}=\overline{\mathit{x}}+{\mathit{S}}_l\phi ,

where \phi ={\mathit{S}}_l^T\mathit{x} is the projection weight of BoldItalic, then P(\widehat{\mathit{x}})=P(\phi ). According to GMM from Ref. [5],

P({\mathit{x}}_t)={\displaystyle \sum _{i=\mathrm{1}}^l{\omega }_{i,t}}\eta ({\mathit{x}}_t,{\mu }_{i,t},{\mathrm{\Sigma }}_{i,t}),

we have

(14)P(\mathit{x})\propto {\displaystyle \sum \exp \left\{-{\displaystyle \frac{\mathrm{1}}{\mathrm{2}}}{\mathit{x}}^T{\mathbf{C}}^{-\mathrm{1}}\mathit{x}\right\}}\mathrm{.}

From Eqs. (13) and (14),

(15)P(\widehat{\mathit{x}})=P(\phi )={\displaystyle \sum {\displaystyle \frac{\exp \left\{-{\displaystyle \frac{\mathrm{1}}{\mathrm{2}}}{\displaystyle \sum _{k=\mathrm{1}}^l{\displaystyle \frac{{\phi }_k^{\mathrm{2}}}{{\lambda }_k}}}\right\}}{{(\mathrm{2}\pi )}^{l/\mathrm{2}}{\displaystyle \prod _{k=\mathrm{1}}^l{\lambda }^{\mathrm{1}/\mathrm{2}}}}}}\mathrm{.}

Thus, we have the background model [5]

(16)B=\arg \underset{l}{\min }\left({\displaystyle \sum _{k=\mathrm{1}}^l{\phi }_k>T}\right),

and the binary image at time t:

(17){\mathit{I}}_t=\{\begin{array}{ll}255,\hfill & \mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d},\hfill \\ 0,\hfill & \mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{g}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}.\hfill \end{array}

As described previously, the image {\mathit{I}}_t contains each moving object both interesting and non-interesting.

Secondary foreground segmentation

To remove the rain and snow streaks, namely non-interesting targets in the video, it is necessary to perform a secondary segmentation for binary image {\mathit{I}}_t. Based on the observation, the streaks move at a higher speed than other objects (e.g., the pedestrian). Furthermore, raindrops and snowflakes show strong statistics property of the direction to ground horizons. We use the von Mises distribution probability [15] density function to represent it:

(18)p(\theta ;{\mu }_{\mathrm{von}},d_{\mathrm{von}})={\displaystyle \frac{\mathrm{1}}{\mathrm{2}\pi B(d_{\mathrm{von}})}}\exp \{d_{\mathrm{von}}\cos (\theta -{\mu }_{\mathrm{von}})\},

where B(d_{\mathrm{von}})={\displaystyle {\sum }_{r=\mathrm{0}}^{\infty }{\left({\displaystyle \frac{\mathrm{1}}{r!}}\right)}^{\mathrm{2}}{\left({\displaystyle \frac{\mathrm{1}}{\mathrm{2}}}{d}_{\mathrm{von}}\right)}^{\mathrm{2}r}} is the Bessel function of the first kind and order 0, \theta is the direction angle of the velocity vector, {\mu }_{\mathrm{von}} is the velocity magnitude, d_{\mathrm{von}}=\mathrm{1}/{\sigma }^{\mathrm{2}} is the dispersion parameter, and here we use the mean velocity of all pixels in an object.

After training the samples via k-means algorithm, we get c cluster centers of the object direction. Current observation is classified into rain and snow streaks or interesting objects using the Euclidean distance. However, it cannot classify all objects completely only by the direction angle feature, it must be used to form a joint classifier along with the magnitude of the mean velocity vector.

Optical measure of rain and snow streaks is determined by camera parameters [11,16] (focal length, exposure time, etc.). Speed of a raindrop with the diameter a is [17]

(19)s(a)=-\mathrm{0.2}+\mathrm{5.0}a-\mathrm{0.9}{a}^{\mathrm{2}}+\mathrm{0.1}{a}^{\mathrm{3}},

where ; the speed of a snowflake is half that of a raindrop [7,9], about s(a)/\mathrm{2}. The interesting objects for us, pedestrians, move at a lower speed. Therefore, the motion property of an object will be indirectly represented through the overlapped area of the same object between two consecutive frames, denoted as P(O_{n,t},O_{n,t+\mathrm{1}}) (see Fig. 2). Garg and Nayar [8] use the photometric constraint to reduce pixels belonging to rain through intensity changing over space and time. The intensity d_t of the pixel with rain in the tth frame is replaced by (d_{t-\mathrm{1}}+d_{t+\mathrm{1}})/\mathrm{2}. It is based on pixels and most effective in a relatively textureless scenario. However, most surveillance videos have dynamic and complex background. We develop a streaks removal algorithm based on region to overcome that.

Suppose that there are M objects.

Step 1　Locate the centroid of the nth object in binary image {\mathit{I}}_t, \mathrm{1}\le n\le M;

Step 2　Search object pixel from the centroid until we find a pixel whose pixel value is 255 and then use it as seed to perform region growth algorithm to segment the whole object. Thus, we get this object’s pixel value set;

Step 3　Calculate the mean velocity {\overline{\mathit{v}}}_n and its direction angle {\theta }_n of an object O_n; compute the Euclidean distance D({\theta }_n,c) between {\theta }_n and each of the c cluster centers; using the nearest neighbor algorithm to classify O_n into raindrop/snowflake candidate or interesting objects;

Step 4　Calculate P(O_{n,t},O_{n,t+\mathrm{1}}): if (the threshold value) then O_n is a raindrop/snowflake and can be filled by background; else, an interesting object;

Step 5　Repeat Step 1 to Step 4 M times.

Experimental results

Next, we show a set of experiments in which we extract interesting objects within the proposed primary-secondary background model after SWPCA de-nosing and Nayar’s method [9]. We start with an experiment for SWPCA and traditional PCA, choose the number of principal components depending on scene complexity, and topically is 5-50.

As described in Fig. 3, the error of SWPCA at the beginning is higher than traditional PCA and declines sharply while iteration goes on. All the principal components must be initialized with arbitrary real numbers, and we use 1.0. Finally, SWPCA and traditional PCA show the same performance. In addition, SWPCA delay m frames to form a sliding window.

The second example is presented in Fig. 4. In this scene, some people are walking in a snowy scene. Figure 4(a) is the video and a moving area of an interesting object, the velocity field of the moving block and the corresponding orientation by the way of von Mises distribution; Fig. 4(b) shows the result of primary segmentation; Fig. 4(c) describes that snow streaks and pedestrian have different motion property; Fig. 4(d) shows the snow streaks removal effect using the proposed method; and Fig. 4(e) gives the comparison with Nayar’s method described in Ref. [7]. Compared to the proposed method, the person contour is blurred showing that Nayar’s method cannot classify objects with different speeds.

Conclusions

It is hard to build a background model in a dynamic weather scene. We proposed a primary-secondary model to extract interesting objects. In the first phase, we combine SWPCA to GMM to segment all active objects in the video with noises; in the second phase we combine von Mises distribution with the overlapped region probability algorithm to extract interesting targets, and remove the rain and snow streaks successfully. Experiments show that the method can suppress noise and extract interesting targets effectively.

References

[1] Drew M S, Wei J, Li Z N. Illumination — invariant image retrieval and video segmentation. Pattern Recognition, 1999, 32(8): 1369–1388

[2] Bianco S, Cusano C. Color target localization under varying illumination conditions. Computational Color Imaging. Lecture Notes in Computer Science, 2011, 6626: 245–255

[3] Freedman D, Turek M W. Illumination — Invariant tracking via graph cuts. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE Computer Society, 2005, 2: 10–17

[4] Narasimhan S G, Nayar S K. Vision and the atmosphere. International Journal of Computer Vision, 2002, 48(3): 233–254

[5] Stauffer C, Grimson W E L. Adaptive background mixture models for real-time tracking. In: Proceedings of the 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE Computer Society, 1999, 2: 2246–2252

[6] Greenspan H, Goldberger J, Mayer A. Probabilistic space-time video modeling via piecewise GMM. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(3): 384–396

[7] Garg K, Nayar S K. Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE Computer Society, 2004, 1: 528–535

[8] Garg K, Nayar S K. Photorealistic rendering of rain streaks. ACM Transactions on Graphics, 2006, 25(3): 996–1002

[9] Garg K, Nayar S K. Vision and rain. International Journal of Computer Vision, 2007, 75(1): 3–27

[10] Barnum P C, Narasimhan S, Kanade T. Analysis of rain and snow in frequency space. International Journal of Computer Vision, 2009, 86(2-3): 256–274

[11] Barnum P, Kanade T, Narasimhan S G. Spatio-temporal frequency analysis for removing rain and snow from videos. In: Proceedings of the First International Workshop on Photometric Analysis for Computer Vision, in conjunction with International Conference of Computer Vision. Rio de Janeiro: INRIA, 2007, 1–8

[12] Starik S, Werman M. Simulation of rain in videos. In: Proceedings of the 3rd International Workshop on Texture Analysis and Synthesis. Edinburgh: IEEE Press, 2003, 95–100

[13] Zhang X P, Li H, Qi Y Y, Leow W K, Ng T K. Rain removal in video by combining temporal and chromatic properties. In: Proceedings of the 2006 IEEE International Conference on Multimedia and Expo. Washington DC: IEEE Computer Society, 2006, 461–464

[14] Faraji H, MacLean W J. CCD noise removal in digital images. IEEE Transactions on Image Processing, 2006, 15(9): 2676–2685

[15] Khatri C G, Mardia K V. The von Mises-Fisher matrix distribution in orientation statistics. Journal of the Royal Statistical Society. Series B (Methodological), 1977, 39(1): 95–106

[16] Rousseau P, Jolivet V, Ghazanfarpour D. Realistic real-time rain rendering. Computers & Graphics, 2006, 30(4): 507–518

[17] Foote G B, Du Toit P S. Terminal velocity of raindrops aloft. Journal of Applied Meteorology, 1969, 8(2): 249–253

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg