Variational quality control of non-Gaussian innovations and its parametric optimizations for the GRAPES m3DVAR system

Abstract

The magnitude and distribution of observation innovations, which have an important impact on the analyzed accuracy, are critical variables in data assimilation. Variational quality control (VarQC) based on the contaminated Gaussian distribution (CGD) of observation innovations is now widely used in data assimilation, owing to the more reasonable representation of the probability density function of innovations that can sufficiently absorb observations by assigning different weights iteratively. However, the inaccurate parameters prevent VarQC from showing the advantages it should have in the GRAPES (Global/Regional Assimilation and PrEdiction System) m3DVAR system. Consequently, the parameter optimization methods are considerable critical studies to improve VarQC. In this paper, we describe two probable CGDs to include the non-Gaussian distribution of actual observation errors, Gaussian plus flat distribution and Huber norm distribution. The potential optimization methods of the parameters are introduced in detail for different VarQCs. With different parameter configurations, the optimization analysis shows that the Gaussian plus flat distribution and the Huber norm distribution are more consistent with the long-tail distribution of actual innovations compared to the Gaussian distribution. The VarQC’s cost and gradient functions with Huber norm distribution are more reasonable, while the VarQC’s cost function with Gaussian plus flat distribution may converge on different minimums due to its non-concave properties. The weight functions of two VarQCs gradually decrease with the increase of innovation but show different shapes, and the VarQC with Huber norm distribution shows more elasticity to assimilate the observations with a high contamination rate. Moreover, we reveal a general derivation relationship between the CGDs and VarQCs. A novel schematic interpretation that classifies the assimilated data into three categories in VarQC is presented. They are conducive to the development of a new VarQC method in the future.

Graphical abstract

Keywords

data assimilation / variational quality control / contaminated Gaussian distribution / non-Gaussian distribution / innovation

Jie He received his Ph.D. Degree in meteorology from Nanjing University of Information Science and Technology (NUIST), Nanjing, China in 2021, mainly focusing on the research of variational quality control and ensemble reanalysis. He continues as a postdoc to study data assimilation and ensemble forecast at the Guangzhou Institute of Tropical and Marine Meteorology.E-mail: [e-mail].

Yang Shi received her M.S. Degree in meteorology from NUIST, Nanjing, China in 2015. She is an Engineer at the Guangdong Meteorological Observatory and her current research includes ensemble forecasts and postprocessing of NWP.E-mail: [e-mail]

Boyang Zhou received her M.S. Degree in meteorology from NUIST, Nanjing, China in 2017. ​She is an engineer at the Qingdao Air Traffic Management Station of Civil Aviation of China and her current research includes data assimilation and ensemble perturbation schemes.E-mail: [e-mail].

Qiuping Wang is a graduate student in meteorology from the School of Atmospheric Sciences at NUIST, Nanjing, China. Her research interest focuses on ensemble forecast and data assimilation. E-mail: [e-mail].

Xulin Ma received his Ph.D. Degree in meteorology from NUIST, China in 2008. His major field is data assimilation, ensemble forecast, and meso- and micro-scale numerical simulation. He works as a Professor at the School of Atmospheric Sciences, NUIST, Nanjing, China since 2008. He is devoted to studying ensemble forecast theory, hybrid data assimilation, and adaptive or targeting observation.E-mail: [e-mail]

References

[1] Anderson E, Jarvinen H (1999). Variational quality control.Q J R Meteorol Soc, 125(554): 697–722

[2] Courtier P, Andersson E, Heckley W, Vasiljevic D, Hamrud M, Hollingsworth A, Rabier F, Fisher M, Pailleux J (1998). The ECMWF implementation of three-dimensional variational assimilation (3D-Var).Part I: formulation. Q J R Meteorol Soc, 124(550): 1783–1807

[3] Dharssi I, Lorenc A C, Ingleby N B (1992). Treatment of gross errors using maximum probability theory.Q J R Meteorol Soc, 118(507): 1017–1036

[4] Duan B H, Zhang W M, Yang X F, Dai H J, Yu Y (2017). Assimilation of typhoon wind field retrieved from scatterometer and SAR based on the Huber norm quality control.Remote Sens (Basel), 9(10): 987

[5] Fernández C, Steel M F (1998). On Bayesian modeling of fat tails and skewness.J Am Stat Assoc, 93(441): 359–371

[6] Fowler A, van Leeuwen P J (2013). Observation impact in data assimilation: the effect of non-Gaussian observation error.Tellus, 65(1): 20035

[7] Gauthier P, Chouinard C, Brasnett B (2003). Quality control: methodology and applications. In: Swinbank R, Shutyaev V, Lahoz W A, eds. Data Assimilation for the Earth System. Dordrecht: Springer-Verlag, 177–187

[8] Guitton A, Symes W W (2003). Robust inversion of seismic data using the Huber norm.Geophysics, 68(4): 1310–1319

[9] Hampel F R (1977). Rejection rules and robust estimates of location: an analysis of some Monte Carlo results. In: Transactions of the Seventh Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes. Dordrecht, Hingham, MA, 187–194

[10] Hampel F R (2001). Robust statistics: a brief introduction and overview. In: Research report/Seminar für Statistik, Eidgenössische Technische Hochschule. Zurich, Switzerland, 1–5

[11] He J, Ma X L, Ge X Y, Liu J J, Cheng W, Chan M Y, Xiao Z N (2021). Variational quality control of non-Gaussian innovations in the GRAPES m3DVAR system: mass field evaluation of assimilation experiments.Adv Atmos Sci, 38(9): 1510–1524

[12] Hollingsworth A (1989). The role of real-time four-dimensional data assimilation in the quality control, interpretation, and synthesis of climate data. In: Anderson D L T, and Willebrand J, eds. Oceanic Circulation Models: Combining Data and Dynamics. Dordrecht: Springer-Verlag, 304–339

[13] Houtekamer P L, Mitchell H L (1998). Data assimilation using an ensemble Kalman filter technique.Mon Weather Rev, 126(3): 796–811

[14] Huber P J (1972). The 1972 wald lecture robust statistics: a review.Ann Math Stat, 43(4): 1041–1067

[15] Huber P J (2011). Robust statistics. In: Lovric M, ed. International Encyclopedia of Statistical Science. Heidelberg: Springer-Verlag, 1248–1251

[16] Ingleby N B, Lorenc A C (1993). Bayesian quality control using multivariate normal distributions.Q J R Meteorol Soc, 119(513): 1195–1225

[17] Jarvinen H, Unden P (1997). Observation screening and first guess quality control in the ECMWF 3D-Var data assimilation system.In: ECMWF Technical Memoranda, (236): 1–33

[18] Kalnay E (2003). Atmospheric Modeling, Data Assimilation and Predictability.New York: Cambridge University Press, 198–204

[19] Lahoz W A, Schneider P (2014). Data assimilation: making sense of earth observation.Front Env Sci—Switz, 2: 16

[20] Legrand R, Michel Y, Montmerle T (2016). Diagnosing non-Gaussianity of forecast and analysis errors in a convective scale model.Nonlinear Process Geophys, 23: 1–12

[21] Lorenc A C (1988). Optimal nonlinear objective analysis.Q J R Meteorol Soc, 114(479): 205–240

[22] Lorenc A C, Hammon O (1988). Objective quality control of observations using Bayesian methods: theory, and a practical implementation.Q J R Meteorol Soc, 114(480): 515–543

[23] Ma X L, Zhuang Z R, Xue J S, Lu W S (2009). Development of 3-D variational data assimilation system for the nonhydrostatic numerical weather prediction model-GRAPES.Acta Meteorol Sin, 67(1): 50–60

[24] Ma X L, He J, Zhou B Y, Li L, Ji Y, Guo H (2017). Effect of variational quality control of Non-Gaussian distribution observation error on heavy rainfall prediction. Trans Atmos Sci, 40(2): 170−180 (in Chinese)

[25] Parrish D F, Derber J C (1992). The national meteorological center spectral statistical interpolation analysis system.Mon Weather Rev, 120(8): 1747–1763

[26] Pires C A, Talagrand O, Bocquet M (2010). Diagnosis and impacts of non-Gaussianity of innovations in data assimilation.Physica D, 239(17): 1701–1717

[27] Rabier F, Järvinen H, Klinker E, Mahfouf J F, Simmons A (2000). The ECMWF operational implementation of four-dimensional variational assimilation.I: experimental results with simplified physics. Q J R Meteorol Soc, 126(564): 1143–1170

[28] Sondergaard T, Lermusiaux P F (2013). Data assimilation with Gaussian mixture models using the dynamically orthogonal field equations.Part I: theory and scheme. Mon Weather Rev, 141(6): 1737–1760

[29] Storto A (2016). Variational quality control of hydrographic profile data with non-Gaussian errors for global ocean variational data assimilation systems.Ocean Model, 104: 226–241

[30] Su X J, Pursera R J (2013). A new observation error probability model for nonlinear variational quality control and applications within the NCEP gridpoint statistical interpolation. In: Sixth WMO Symposium on Data Assimilation. Sixth WMO Symposium on Data Assimilation, 33–34

[31] Swinbank R, Shutyaev V, Lahoz W A (2003). Data assimilation for the earth system. Maratea: Springer, 31–35

[32] Tavolato C, Isaksen L (2015). On the use of a Huber norm for observation quality control in the ECMWF 4D–Var.Q J R Meteorol Soc, 141(690): 1514–1527

[33] Tukey J W (1960). A survey of sampling from contaminated distributions. In: Olkin I, ed. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling. Stanford, CA: Stanford University Press, 448–485

[34] Wang X G, Lei T (2014). GSI-based four-dimensional ensemble-variational (4DEnsVar) data assimilation: formulation and single-resolution experiments with real data for NCEP global forecast system.Mon Weather Rev, 142(9): 3303–3325

[35] Yang Y X (1991). Robust bayesian estimation.J Geod, 65(3): 145–150

[36] Zhao H, Zou X L, Qin Z K (2015). Quality control of specific humidity from surface stations based on EOF and FFT—case study.Front Earth Sci, 9(3): 381–393

[37] Zhu J J (1996). Robustness and the robust estimate.J Geod, 70(9): 586–590

[38] Zhu J J, Zeng Z Q (1999). The theory of surveying adjustment under contaminated error model. Acta Geodaetica et Cartographica Sinica, 28(3): 91–91(in Chinese)

Acknowledgments

The authors thank Yinghui Lu for his helpful advice and grammar correction. Jie He is supported by the China Scholarship Council. This work is primarily sponsored by the National Key R&D Program of China (Nos. 2018YFC1506702 and 2017YFC1502000). We acknowledge the High Performance Computing Center of Nanjing University of Information Science & Technology (NUIST) for their support of this work. The data sets in this paper are archived and accessible on the super computer of NUIST.

RIGHTS & PERMISSIONS

2022 Higher Education Press