Signal-Adapted Kernels and Lazy-Evaluation Gaussian Process Regression

Abstract

Constant-bandwidth radial basis function (RBF) regression models are popular due to their simplicity, but they exhibit oscillatory patterns that resemble the Runge and Gibbs phenomenon on many real-world datasets. The Gaussian process regression (GPR) can be considered as an extension of RBF regression in that some RBFs can be posed as stationary kernels, but the GPR can also use non-stationary kernels. The GPR scales poorly for large datasets, and existing GPR approximations are either complicated to implement, still require large computational resources, or suffer from discontinuous border issues if an ensemble of local GPRs are used. In this article, we developed a non-stationary kernel that adapts its shape from the regression data, under the existing dimensional expansion kernel framework. We also proposed an observation model for local training sets that continuously diminish the effect of training entries as they move further from the query input, and developed a corresponding lazy-evaluation GPR model that has continuity guarantees for the predictive posterior mean and variance functions. We compared the presence of oscillations and contour-aware smoothing behavior ability of our methods on real-world rainfall and elevation datasets.

Keywords

Graph signal processing / Gaussian process regression / Scattered data regression / Image processing / 62G08 / 68U10 / 68T45

References

[1] Behjat H, Richter U, Van De Ville D, Sörnmo L. Signal-adapted tight frames on graphs. IEEE Trans. Signal Process., 2016, 642260176029.

[2] Bentley JL, Stanat DF, Williams EH. The complexity of finding fixed-radius near neighbors. Inf. Process. Lett., 1977, 66209212.

[3] Bobach T, Farin G, Hansford D, Umlauf G. Natural neighbor extrapolation using ghost points. Comput. Aided Des., 2009, 415350365.

[4] Bornn L, Shaddick G, Zidek JV. Modeling nonstationary processes through dimension expansion. J. Am. Stat. Assoc., 2012, 107497281289.

[5] Boyd JP. Six strategies for defeating the Runge phenomenon in gaussian radial basis functions on a finite interval. Comput. Math. Appl., 2010, 601231083122.

[6] Boyd JP, Ong JR. Exponentially-convergent strategies for defeating the Runge phenomenon for the approximation of non-periodic functions, part i: single-interval schemes. Comput. Phys., 2009, 52–4484497.

[7] Boyd JP, Ong JR. Exponentially-convergent strategies for defeating the Runge phenomenon for the approximation of non-periodic functions, part two: Multi-interval polynomial schemes and multidomain chebyshev interpolation. Appl. Numer. Math., 2011, 614460472.

[8] Bozzini M, Lenarduzzi L, Rossini M, Schaback R. Interpolation with variably scaled kernels. IMA J. Numer. Anal., 2015, 351199219.

[9] Campi C, Marchetti F, Perracchione E. Learning via variably scaled kernels. Adv. Comput. Math., 2021, 47451.

[10] Chudy, F., Woźny, P.: Linear-time algorithm for computing the bernstein–bézier coefficients of b-spline basis functions. Comput. Aided Des. 154, 103434 (2023). https://doi.org/10.1016/j.cad.2022.103434

[11] De Marchi S, Lot F, Marchetti F, Poggiali D. Variably scaled persistence kernels (vspks) for persistent homology applications. J. Comput. Math. Data Sci., 2022, 4. 100050

[12] Durre I, Menne MJ, Vose RS. Strategies for evaluating quality assurance procedures. J. Appl. Meteorol. Climatol., 2008, 47617851791.

[13] Durre I, Menne MJ, Gleason BE, Houston TG, Vose RS. Comprehensive automated quality assurance of daily surface observations. J. Appl. Meteorol. Climatol., 2010, 49816151633.

[14] Egilmez HE, Chao Y-H, Ortega A. Graph-based transforms for video coding. IEEE Trans. Image Process., 2020, 29: 9330-9344.

[15] Fornberg B, Flyer N. A Primer on Radial Basis Functions with Applications to the Geosciences, 2015, Philadelphia: Society for Industrial and Applied Mathematics

[16] Fornberg B, Zuev J. The Runge phenomenon and spatially variable shape parameters in RBF interpolation. Comput. Math. Appl., 2007, 543379398.

[17] Fracastoro, G., Magli, E.: Steerable discrete cosine transform. In: 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6 (2015). https://doi.org/10.1109/MMSP.2015.7340827

[18] Franzen, R.: Kodak Image Suite. https://r0k.us/graphics/kodak

[19] Global Historical Climatology Network Daily. https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily

[20] Gramacy RB, Apley DW. Local gaussian process approximation for large computer experiments. J. Comput. Graph. Stat., 2015, 242561578.

[21] Hammond DK, Vandergheynst P, Gribonval R. Wavelets on graphs via spectral graph theory. Appl. Comput. Harmon. Anal., 2011, 302129150.

[22] Hinton, G.E., Salakhutdinov, R.R.: Using deep belief nets to learn covariance kernels for gaussian processes. In: Advances in Neural Information Processing Systems, vol. 20 (2007)

[23] Joharinad, P., Jost, J.: The Laplace operator and the geometry of graphs. In: Mathematical Principles of Topological and Geometric Data Analysis, pp. 83–113. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-33440-5_4

[24] Kittisopikul, M., Holy, T.E., Aschan, T.: JuliaMath/Interpolations.jl: v0.15.1. https://github.com/JuliaMath/Interpolations.jl (2023)

[25] Leonardi N, Van De Ville D. Tight wavelet frames on multislice graphs. IEEE Trans. Signal Process., 2013, 611333573367.

[26] Ljungskog, E.: eljungsk/ScatteredInterpolation.jl: v0.3.6. https://github.com/eljungsk/ScatteredInterpolation.jl (2021)

[27] Ma, Y.Z., Ma, Y.: Geostatistical estimation methods: Kriging. Quantitative Geosciences: Data Analytics, Geostatistics, Reservoir Characterization and Modeling, pp. 373–401 (2019)

[28] Menne MJ, Durre I, Vose RS, Gleason BE, Houston TG. An overview of the global historical climatology network-daily database. J. Atmos. Oceanic Tech., 2012, 297897910.

[29] Micchelli CA, Pontil M. On learning vector-valued functions. Neural Comput., 2005, 171177204.

[30] Missisquoi Watershed LiDAR. https://doi.org/10.5069/G9ST7MR9

[31] Ober, S.W., Rasmussen, C.E., Wilk, M.: The promises and pitfalls of deep kernel learning. In: Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, pp. 1206–1216 (2021)

[32] Ortega A. Introduction to Graph Signal Processing, 2022, Cambridge: Cambridge University Press

[33] Park C, Apley D. Patchwork kriging for large-scale gaussian process regression. J. Mach. Learn. Res., 2018, 197143

[34] Quiñonero-Candela J, Rasmussen CE. A unifying view of sparse approximate gaussian process regression. J. Mach. Learn. Res., 2005, 66519391959

[35] Rasmussen CE, Williams CKI. Gaussian Processes for Machine Learning, 2006, Cambridge: MIT Press

[36] Ryu J-S, Kim M-S, Cha K-J, Lee TH, Choi D-H. Kriging interpolation methods in geostatistics and dace model. KSME Int. J., 2002, 16: 619-632.

[37] Sampson, P.D., Damian, D., Guttorp, P.: Advances in modeling and inference for environmental processes with nonstationary spatial covariance. In: Monestiez, P., Allard, D., Froidevaux, R. (eds.) geoENV III—Geostatistics for Environmental Applications, pp. 17–32. Springer, Dordrecht (2001). https://doi.org/10.1007/978-94-010-0810-5_2

[38] Sandryhaila A, Moura JMF. Discrete signal processing on graphs: Frequency analysis. IEEE Trans. Signal Process., 2014, 621230423054.

[39] Scholköpf B, Smola AJ. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, 2002, Cambridge: MIT

[40] Shepard, D.: A two-dimensional interpolation function for irregularly-spaced data. In: Proceedings of the 1968 23rd ACM National Conference, pp. 517–524 (1968)

[41] Sibson, R.: A brief description of natural neighbour interpolation. Interpreting Multivariate Data, 21–36 (1981)

[42] Soulard R, Carré P. Elliptical monogenic wavelets for the analysis and processing of color images. IEEE Trans. Signal Process., 2016, 64615351549.

[43] Soulard R, Carré P. Characterization of color images with multiscale monogenic maxima. IEEE Trans. Pattern Anal. Mach. Intell., 2018, 401022892302.

[44] Soulard R, Carre P, Fernandez-Maloigne C. Vector extension of monogenic wavelets for geometric representation of color images. IEEE Trans. Image Process., 2013, 22310701083.

[45] Stein ML, Chi Z, Welty LJ. Approximating likelihoods for large spatial data sets. J. R. Stat. Soc. Ser. B Stat Methodol., 2004, 662275296.

[46] Strang G. The discrete cosine transform. SIAM Rev., 1999, 411135147.

[47] Thiébaut, É.: emmt/LocalFilters.jl: v2.0.0. https://github.com/emmt/LocalFilters.jl (2023)

[48] Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Sixth International Conference on Computer Vision, pp. 839–846 (1998). https://doi.org/10.1109/ICCV.1998.710815

[49] Unser M. A representer theorem for deep neural networks. J. Mach. Learn. Res., 2019, 20110130

[50] Unser M, Chenouard N, Van De Ville D. Steerable pyramids and tight wavelet frames in $L_{2}\left(\mathbb{R} ^{d}\right)$. IEEE Trans. Image Process., 2011, 201027052721.

[51] VandenHeuvel, D.: DanielVandH/NaturalNeighbours.jl: v1.3.6. https://github.com/DanielVandH/NaturalNeighbours.jl (2024)

[52] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 2004, 134600612.

[53] Wang T, Zhang L, Hu W. Bridging deep and multiple kernel learning: a review. Inf. Fus., 2021, 67: 3-13.

[54] Wendland H. Scattered Data Approximation, 2004, Cambridge: Cambridge University Press17

[55] Zhang C, Florencio D. Analyzing the optimality of predictive transform coding using graph-based models. IEEE Signal Process. Lett., 2013, 201106109.

[56] Zhang W, Sun Z, Liu J, Chen S. A more efficient and practical modified nyström method. Mathematics, 2023, 11112433.

Rights & permissions

The Author(s)