Batch Active Learning for Multispectral and Hyperspectral Image Segmentation Using Similarity Graphs

Abstract

Graph learning, when used as a semi-supervised learning (SSL) method, performs well for classification tasks with a low label rate. We provide a graph-based batch active learning pipeline for pixel/patch neighborhood multi- or hyperspectral image segmentation. Our batch active learning approach selects a collection of unlabeled pixels that satisfy a graph local maximum constraint for the active learning acquisition function that determines the relative importance of each pixel to the classification. This work builds on recent advances in the design of novel active learning acquisition functions (e.g., the Model Change approach in arXiv:2110.07739) while adding important further developments including patch-neighborhood image analysis and batch active learning methods to further increase the accuracy and greatly increase the computational efficiency of these methods. In addition to improvements in the accuracy, our approach can greatly reduce the number of labeled pixels needed to achieve the same level of the accuracy based on randomly selected labeled pixels.

Keywords

Image segmentation / Graph learning / Batch active learning / Hyperspectral image

References

[1.] Arya S, Mount DM, Netanyahu NS, Silverman R, Wu AY. An optimal algorithm for approximate nearest neighbor searching in fixed dimensions. J. ACM (JACM), 1998, 45(6): 891-923,

[2.] Belkin M, Niyogi P, Sindhwani V. Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res., 2006, 7: 2399-2434

[3.] Bertozzi AL, Flenner A. Diffuse interface models on graphs for classification of high dimensional data. Multiscale Model. Simul., 2012, 10(3): 1090-1118,

[4.] Bertozzi AL, Hosseini B, Li H, Miller K, Stuart AM. Posterior consistency of semi-supervised regression on graphs. Inverse Problems, 2021, 37(10): 105011,

[5.] Bertozzi AL, Luo X, Stuart AM, Zygalakis KC. Uncertainty quantification in graph-based classification of high dimensional data. SIAM/ASA J. Uncertain. Quantif., 2018, 6(2): 568-595,

[6.] Bertozzi, A.L., Merkurjev, E.: Graph-based optimization approaches for machine learning, uncertainty quantification and networks. In: Processing, Analyzing and Learning of Images, Shapes, and Forms. Part 2, 503-531, Handb. Numer. Anal., 20, Elsevier/North-Holland, Amsterdam (2019)

[7.] Boyd ZM, Bae E, Tai X-C, Bertozzi AL. Simplified energy landscape for modularity using total variation. SIAM J. Appl. Math., 2018, 78(5): 2439-2464,

[8.] Boyd ZM, Porter MA, Bertozzi AL. Stochastic block models are a discrete surface tension. J. Nonlinear Sci., 2020, 30(5): 2429-2462,

[9.] Bresson X, Esedoglu S, Vandergheynst P, Thiran J-P, Osher S. Fast global minimization of the active contour/snake model. J. Math. Imaging Vision, 2007, 28(2): 151-167,

[10.] Buades, A., Coll, B., Morel, J.-M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 60–65. IEEE (2005)

[11.] Cai, W., Zhang, Y., Zhou, J.: Maximizing expected model change for active learning in regression. In: 2013 IEEE 13th International Conference on Data Mining, pp. 51–60. IEEE (2013)

[12.] Chan TF, Vese LA. Active contours without edges. IEEE Trans. Image Process., 2001, 10(2): 266-277,

[13.] Chapman, J., Chen, B., Tan, Z., Calder, J., Miller, K., Bertozzi, A.L.: Novel batch active learning approach and its application on the synthetic aperture radar datasets. In: SPIE Defense and Commercial Sensing: Algorithms for Synthetic Aperture Radar Imagery XXX (2023)

[14.] Ciurte A, Bresson X, Cuisenaire O, Houhou N, Nedevschi S, Thiran J-P, Cuadra MB. Semi-supervised segmentation of ultrasound images based on patch representation and continuous min cut. PLoS ONE, 2014, 9(7),

[15.] Dasgupta S. Two faces of active learning. Theoret. Comput. Sci., 2011, 412(19): 1767-1781,

[16.] Gal, Y., Islam, R., Ghahramani, Z.: Deep Bayesian active learning with image data. In: International Conference on Machine Learning, pp. 1183–1192. PMLR (2017)

[17.] Garcia-Cardona C, Merkurjev E, Bertozzi AL, Flenner A, Percus AG. Multiclass data segmentation using diffuse interface methods on graphs. IEEE Trans. Pattern Anal. Mach. Intell., 2014, 36(8): 1600-1613,

[18.] Gilboa G, Osher S. Nonlocal operators with applications to image processing. Multiscale Model. Simul., 2009, 7(3): 1005-1028,

[19.] Hu, H., Sunu, J., Bertozzi, A.L.: Multi-class graph Mumford-Shah model for plume detection using the MBO scheme. In: Proceedings of the EMMCVPR Conference in Hong Kong. 8932, 209–222. Tai, X.-C. et al. (Eds), Springer Lecture Notes in Computer Science (2015)

[20.] Hu H, Laurent T, Porter MA, Bertozzi AL. A method based on total variation for network modularity optimization using the MBO scheme. SIAM J. Appl. Math., 2013, 73(6): 2224-2246,

[21.] Iyer G, Chanussot J, Bertozzi AL. A graph-based approach for data fusion and segmentation of multimodal images. IEEE Trans. Geosci. Remote Sensing, 2021, 59(5): 4419-4429,

[22.] Ji M, Han J. A variance minimization criterion to active learning on graphs. Artificial Intelligence and Statistics, 2012 PMLR 556-564

[23.] Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comp. Vision 1, 321–331 (2004)

[24.] Kushnir, D., Venturi, L.: Diffusion-based deep active learning. arXiv:2003.10339 (2020)

[25.] Ma, Y., Garnett, R., Schneider, J.G.: Sigma-optimality for active learning on Gaussian random fields. In: NIPS, pp. 2751–2759 (2013)

[26.] Ma, Y., Huang, T.-K., Schneider, J.G.: Active search and bandits on graphs using sigma-optimality. In: UAI, vol. 542, pp. 551 (2015)

[27.] Meng Z, Merkurjev E, Koniges A, Bertozzi AL. Hyperspectral image classification using graph clustering methods. IPOL J. Image Process. Online, 2017, 7: 218-245,

[28.] Merkurjev E, Garcia-Cardona C, Bertozzi AL, Flenner A, Percus AG. Diffuse interface methods for multiclass segmentation of high-dimensional data. Appl. Math. Lett., 2014, 33: 29-34,

[29.] Merkurjev E, Kostić T, Bertozzi AL. An MBO scheme on graphs for classification and image processing. SIAM J. Imaging Sci., 2013, 6(4): 1903-1930,

[30.] Merkurjev, E., Sunu, J., Bertozzi, A.L.: Graph MBO method for multiclass segmentation of hyperspectral stand-off detection video. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 689–693. IEEE (2014)

[31.] Miller, K., Bertozzi, A.L.: Model-change active learning in graph-based semi-supervised learning. https://doi.org/10.48550/arXiv.2110.07739 (2021)

[32.] Miller, K., Li, H., Bertozzi, A.L.: Efficient graph-based active learning with probit likelihood via Gaussian approximations. arXiv:2007.11126 (2020).

[33.] Miller, K., Mauro, J., Setiadi, J., Baca, X., Shi, Z., Calder, J., Bertozzi, A.L.: Graph-based active learning for semi-supervised classification of SAR data. arXiv:2204.00005 (2022)

[34.] Mumford D, Shah J. Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math., 1989, 42(5): 577-685,

[35.] O’Shea, K., Nash, R.: An introduction to convolutional neural networks. arXiv:1511.08458 (2015)

[36.] Qiao Y, Shi C, Wang C, Li H, Haberland M, Luo X, Stuart AM, Bertozzi AL. Uncertainty quantification for semi-supervised multi-class classification in image processing and ego-motion analysis of body-worn videos. Electron. Imaging, 2019, 31(11): 1-264,

[37.] Qin J, Lee H, Chi JT, Drumetz L, Chanussot J, Lou Y, Bertozzi AL. Blind hyperspectral unmixing based on graph total variation regularization. IEEE Trans. Geosci. Remote Sensing, 2021, 59(4): 3338-3351,

[38.] Schwenk, J., Rowland, J.: RiverPIXELS: paired Landsat images and expert-labeled sediment and water pixels for a selection of rivers v1.0. United States. https://data.ess-dive.lbl.gov/view/, https://doi.org/10.15485/1865732

[39.] Settles, B.: Active Learning vol. 6, pp. 1–114. Morgan & Claypool Publishers LLC, Carnegie Mellon University, USA (2012). https://doi.org/10.2200/s00429ed1v01y201207aim018

[40.] Shewchuk, J.R.: An introduction to the conjugate gradient method without the agonizing pain. Carnegie-Mellon University, Pittsburgh, PA (1994)

[41.] Thorpe, M., Nguyen, T.M., Xia, H., Strohmer, T., Bertozzi, A., Osher, S., Wang, B.: Grand++: graph neural diffusion with a source term. In: International Conference on Learning Representations (2021)

[42.] Von Luxburg U. A tutorial on spectral clustering. Stat. Comput., 2007, 17(4): 395-416,

[43.] Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: International Conference on Machine Learning, pp. 6861–6871. PMLR (2019)

[44.] Zhu F, Wang Y, Xiang S, Fan B, Pan C. Structured sparse method for hyperspectral unmixing. ISPRS-J. Photogramm. Remote Sens., 2014, 88: 101-118,

[45.] Zhu, X., Ghahramani, Z., Lafferty, J.D.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 912–919 (2003)

Funding

University of California, Los Angeles(L21GF3606); National Defense Science and Engineering Graduate; Los Alamos National Laboratory(20210213ER 20170668PRD1); National Geospatial-Intelligence Agency(HM04762110003)