Information Divergence and the Generalized Normal Distribution: A Study on Symmetricity

Abstract

This paper investigates and discusses the use of information divergence, through the widely used Kullback–Leibler (KL) divergence, under the multivariate (generalized) $\gamma $-order normal distribution ($\gamma $-GND). The behavior of the KL divergence, as far as its symmetricity is concerned, is studied by calculating the divergence of $\gamma $-GND over the Student’s multivariate t-distribution and vice versa. Certain special cases are also given and discussed. Furthermore, three symmetrized forms of the KL divergence, i.e., the Jeffreys distance, the geometric-KL as well as the harmonic-KL distances, are computed between two members of the $\gamma $-GND family, while the corresponding differences between those information distances are also discussed.

Keywords

Kullback–Leibler divergence / Jeffreys distance / Resistor-average distance / $\gamma $-order normal distribution')">Multivariate $\gamma $-order normal distribution / Multivariate Student’s t-distribution / Multivariate Laplace distribution

References

[1] Amari S, Nagaoka H. Methods of Information Geometry. 2000 A. M. Society Oxford: Oxford University Press

[2] Ali SM, Silvey SD. A general class of coefficients of divergence of one distribution from another. JRSS B. 1966, 28 1 131-142

[3] Cover TM, Thomas JA. Elements of Information Theory. 2006 2 New York: Wiley

[4] Gradshteyn IS, Ryzhik IM. Table of Integrals, Series, and Products. 2007 Amsterdam: Elsevier

[5] Harremoes P. Interpretation of Rényi entropies and divergences. Phys. A. 2006, 365 1 57-62

[6] Jeffreys H. An invariant form for the prior probability in estimation problems. Proc. R. Soc. A. 1946, 186 453-461

[7] Johnson, D., Sinanovic, S.: Symmetrizing the Kullback-Leibler distance. IEEE Trans. Inf, Theory (2001). https://scholarship.rice.edu/handle/1911/19969

[8] Kitsos CP, Tavoularis NK. Logarithmic Sobolev inequalities for information measures. IEEE Trans. Inf. Theory. 2009, 55 6 2554-2561

[9] Kitsos CP, Toulias TL. Rassias MT. Inequalities for the Fisher’s information measures. Handbook of Functional Equations: Functional Inequalities. 2014 Berlin: Springer. 281-313

[10] Kitsos CP, Toulias TL. New information measures for the generalized normal distribution. Information. 2010, 1 13-27

[11] Kitsos CP, Toulias TL, Trandafir PC. On the multivariate $\gamma $-ordered normal distribution. Far East J. Theor. Stat.. 2012, 38 1 49-73

[12] Kitsos CP, Vassiliadis VG, Toulias TL. MLE for the $\gamma $-order generalized normal distribution. Discuss. Math. Probab. Stat.. 2014, 34 143-158

[13] Kullback S, Leibler RA. On information and sufficiency. Ann. Math. Stat.. 1951, 22 79-86

[14] Prudnikov AP, Brychkov YA, Marichev OI. Integrals and Series, Vol. I: Elementary Functions. 1998 London: Gordon and Breach Science Publishers

[15] Toulias TL, Kitsos CP. Kullback–Leibler divergence of the $\gamma $-ordered normal over $t$-distribution. Br. J. Math. Comput. Sci.. 2012, 2 4 198-212