Interactive image segmentation with a regression based ensemble learning paradigm

Abstract

To achieve fine segmentation of complex natural images, people often resort to an interactive segmentation paradigm, since fully automatic methods often fail to obtain a result consistent with the ground truth. However, when the foreground and background share some similar areas in color, the fine segmentation result of conventional interactive methods usually relies on the increase of manual labels. This paper presents a novel interactive image segmentation method via a regression-based ensemble model with semi-supervised learning. The task is formulated as a non-linear problem integrating two complementary spline regressors and strengthening the robustness of each regressor via semi-supervised learning. First, two spline regressors with a complementary nature are constructed based on multivariate adaptive regression splines (MARS) and smooth thin plate spline regression (TPSR). Then, a regressor boosting method based on a clustering hypothesis and semi-supervised learning is proposed to assist the training of MARS and TPSR by using the region segmentation information contained in unlabeled pixels. Next, a support vector regression (SVR) based decision fusion model is adopted to integrate the results of MARS and TPSR. Finally, the GraphCut is introduced and combined with the SVR ensemble results to achieve image segmentation. Extensive experimental results on benchmark datasets of BSDS500 and Pascal VOC have demonstrated the effectiveness of our method, and the com-parison with experiment results has validated that the proposed method is comparable with the state-of-the-art methods for in-teractive natural image segmentation.

Keywords

Interactive image segmentation / Multivariate adaptive regression splines (MARS) / Ensemble learning / Thin-plate spline regression (TPSR) / Semi-supervised learning / Support vector regression (SVR)

References

[1] Adamowski, J., Chan, H.F., Prasher, S.O. , , 2012. Com-parison of multivariate adaptive regression splines with coupled wavelet transform artificial neural networks for runoff forecasting in Himalayan micro-watersheds with limited data. J. Hydroinform., 14(3):731–744. https://doi.org/10.2166/hydro.2011.044

[2] Balcan, M.F., Blum, A., Yang, K., 2004. Co-training and expansion: towards bridging theory and practice.17th Int. Conf. on Neural Information Processing Systems, p.89–96.

[3] Blum, A., Mitchell , T., 1998. Combining labeled and unla-beled data with co-training.11th Annual Conf. on Com-putational Learning Theory, p.92–100. https://doi.org/10.1145/279943.279962

[4] Boykov, Y.Y., Jolly, M.P., 2001. Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images.8th IEEE Int. Conf. on Computer Vision, p.105–112. https://doi.org/10.1109/ICCV.2001.937505

[5] Boykov, Y.Y., Veksler , O., Zabih, R. , 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Patt. Anal. Mach. Intell., 23(11):1222–1239. https://doi.org/10.1109/ICCV.1999.791245

[6] Ding, J.J., Lin, C.J., Lu, I.F. , , 2015. Real-time interactive image segmentation using improved superpixels.IEEE Int. Conf. on Digital Signal Processing, p.740–744. https://doi.org/10.1109/ICDSP.2015.7251974

[7] Everingham, M., van Gool , L., Williams, C.K. , , 2010. The Pascal Visual Object Classes (VOC) challenge. Int. J. Comput. Vis., 88(2):303–338.https://doi.org/10.1007/s11263-009-0275-4

[8] Friedman, J.H., 1991. Multivariate adaptive regression splines. Ann. Statist., 19(1):1–67.

[9] Fu, Z., Wang, L., Zhang, D., 2014. An improved multi-label classification ensemble learning algorithm.In: Li, S., Liu, C., Wang, Y. (Eds.), Pattern Recognition. Springer Berlin Heidelberg, p.243–252. https://doi.org/10.1007/978-3-662-45646-0_25

[10] Galar, M., Fernandez , A., Barrenechea, E. , , 2012. A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans. Syst. Man Cybern. C, 42(4):463–484. https://doi.org/10.1109/TSMCC.2011.2161285

[11] Ge, L., Ju, R., Ren, T., , 2015. Interactive RGB-D image segmentation using hierarchical graph cut and geodesic distance. In: Ho, Y.S., Sang, J., Ro, Y.M., et al. (Eds.), Advances in Multimedia Information Processing. Springer International Publishing, p.114–124. https://doi.org/10.1007/978-3-319-24075-6_12

[12] Gulshan, V., Rother, C., Criminisi, A. , , 2010. Geodesic star convexity for interactive image segmentation.IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, p.3129–3136. https://doi.org/10.1109/CVPR.2010.5540073

[13] Jian, M., Jung, C., 2016. Interactive image segmentation using adaptive constraint propagation. IEEE Trans. Image Process., 25(3):1301–1311. https://doi.org/10.1109/TIP.2016.2518480

[14] Jobst, A.M., Kingston , D.G., Cullen, N.J. , , 2016. Com-bining thin-plate spline interpolation with a lapse rate model to produce daily air temperature estimates in a data-sparse alpine catchment. Int. J. Climatol., 37(1):214–229. https://doi.org/10.1002/joc.4699

[15] Jung, C., Jian, M., Liu, J., , 2014. Interactive image segmentation via kernel propagation.Patt. Recogn., 47(8): 2745–2755. https://doi.org/10.1016/j.patcog.2014.02.010

[16] Kolmogorov, V., Zabih, R., 2004. What energy functions can be minimized via graph cuts? IEEE Trans. Patt. Anal. Mach. Intell., 26(2):147–159. https://doi.org/10.1109/TPAMI.2004.1262177

[17] Lazaridis, A., Mporas, I., Ganchev, T. , , 2011. Support vector regression fusion scheme in phone duration mod-eling.IEEE Int. Conf. on Acoustics, Speech and Signal Processing, p.4732–4735. https://doi.org/10.1109/ICASSP.2011.5947412

[18] Lee, Y.S., Cho, S.B., 2014. Activity recognition with Android phone using mixture-of-experts co-trained with labeled and unlabeled data. Neurocomputing, 126:106–115. https://doi.org/10.1016/j.neucom.2013.05.044

[19] Li, Y., Sun, J., Tang, C.K., , 2004. Lazy snapping.ACM Trans. Graph., 23(3):303–308. https://doi.org/10.1145/1015706.1015719

[20] Liu, Y., Yu, Y., 2012. Interactive image segmentation based on level sets of probabilities. IEEE Trans. Visual. Comput. Graph., 18(2):202–213. https://doi.org/10.1109/TVCG.2011.77

[21] Martin, D., Fowlkes , C., Tal, D. , , 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring eco-logical statistics.8th IEEE Int. Conf. on Computer Vision, p.416–423. https://doi.org/10.1109/ICCV.2001.937655

[22] Menon, R., Bhat, G., Saade, G.R. , , 2014. Multivariate adaptive regression splines analysis to predict biomarkers of spontaneous preterm birth. Acta Obstetr. Gynecol. Scandinav., 93(4):382–391.https://doi.org/10.1111/aogs.12344

[23] Nguyen, T.N.A., Cai, J., Zhang, J., , 2012. Robust inter-active image segmentation using convex active contours. IEEE Trans. Image Process., 21(8):3734-3743. https://doi.org/10.1109/TIP.2012.2191566

[24] Ning, J., Zhang, L., Zhang, D., , 2010. Interactive image segmentation by maximal similarity based region merg-ing. Patt. Recogn., 43(2):445–456. https://doi.org/10.1016/j.patcog.2009.03.004

[25] Opitz, D., Maclin, R., 1999. Popular ensemble methods: an empirical study.J. Artif. Intell. Res., 11:169–198. https://doi.org/10.1613/jair.614

[26] Pauchard, Y., Fitze, T., Browarnik, D. , , 2016. Interactive graph-cut segmentation for fast creation of finite element models from clinical CT data for hip fracture prediction. Comput. Methods Biomech. Biomed. Eng., 19(16):1693–1703. https://doi.org/10.1080/10255842.2016.1181173

[27] Peng, B., Zhang, L., Zhang, D., 2013. A survey of graph the-oretical approaches to image segmentation. Patt. Recogn., 46(3):1020–1038. https://doi.org/10.1016/j.patcog.2012.09.015

[28] Qin, C., Zhang, G., Zhou, Y., , 2014. Integration of the saliency-based seed extraction and random walks for image segmentation. Neurocomputing, 129:378–391. https://doi.org/10.1016/j.neucom.2013.09.021

[29] Rother, C., Kolmogorov , V., Blake, A. , 2004. GrabCut: in-teractive foreground extraction using iterated graph cuts. ACM Trans. Graph., 23(3):309–314. http://dx.doi.org/10.1145/1015706.1015720

[30] Shahshahani, B.M., Landgrebe , D.A., 1994. The effect of unlabeled samples in reducing the small sample size problem and mitigating the Hughes phenomenon. IEEE Trans. Geosci. Remote Sens., 32(5):1087–1095. https://doi.org/10.1109/36.312897

[31] Tang, M., Gorelick , L., Veksler, O. , , 2013. GrabCut in one cut.IEEE Int. Conf. on Computer Vision, p.1769–1776. https://doi.org/10.1109/ICCV.2013.222

[32] Wang, T., Sun, Q., Ji, Z., , 2016. Multi-layer graph con-straints for interactive image segmentation via game theory. Patt. Recogn., 55:28–44. https://doi.org/10.1016/j.patcog.2016.01.018

[33] Wang, X.Y., Wang, Q.Y., Yang, H.Y. , , 2011a. Color image segmentation using automatic pixel classification with support vector machine.Neurocomputing, 74(18): 3898–3911. https://doi.org/10.1016/j.neucom.2011.08.004

[34] Wang, X.Y., Wang, T., Bu, J., 2011b. Color image segmenta-tion using pixel wise support vector machine classifica-tion. Patt. Recogn., 44(4):777–787. https://doi.org/10.1016/j.patcog.2010.08.008

[35] Wu, J., Zhao, Y., Zhu, J.Y., , 2014. MILCut: a sweeping line multiple instance learning paradigm for interactive image segmentation.IEEE Conf. on Computer Vision and Pattern Recognition, p.256–263. https://doi.org/10.1109/CVPR.2014.40

[36] Xiang, S., Nie, F., Zhang, C., , 2009. Interactive natural image segmentation via spline regression.IEEE Trans. Image Process., 18(7):1623–1632. https://doi.org/10.1109/TIP.2009.2018570

[37] Xiang, S., Nie, F., Zhang, C., 2010. Semi-supervised classifi-cation via local spline regression.IEEE Trans. Patt. Anal. Mach. Intell., 32(11):2039–2053. https://doi.org/10.1109/TPAMI.2010.35

[38] Yang, W., Cai, J., Zheng, J., , 2010. User-friendly inter-active image segmentation through unified combinatorial user inputs. IEEE Trans. Image Process., 19(9):2470–2479. https://doi.org/10.1109/TIP.2010.2048611

[39] Zhang, J., Tang, Z., Liu, J., , 2016. Recognition of flota-tion working conditions through froth image statistical modeling for performance monitoring. Miner. Eng., 86: 116–129. https://doi.org/10.1016/j.mineng.2015.12.008

[40] Zhang, W., Goh, A.T., 2016. Evaluating seismic liquefaction potential using multivariate adaptive regression splines and logistic regression. Geomech. Eng., 10(3):269–284. https://doi.org/10.12989/gae.2016.10.3.269

[41] Zhang, Y., Song, H., Gu, J., , 2010. Interactive object extraction using hierarchical graph cuts.Int. Conf. on Audio Language and Image Processing, p.851–858. https://doi.org/10.1109/ICALIP.2010.5685212

[42] Zhang, Y., Wen, J., Wang, X., , 2014. Semi-supervised learning combining co-training with active learning. Ex-pert Syst. Appl., 41(5):2372–2378. https://doi.org/10.1016/j.eswa.2013.09.035

[43] Zhou, W., Garcia, E.V., 2016. Nuclear image-guided ap-proaches for cardiac resynchronization therapy (CRT). Curr. Cardiol. Rep., 18(1):1–11. https://doi.org/10.1007/s11886-015-0687-4

[44] Zhou, W., Hou, X., Piccinelli, M. , , 2014. 3D fusion of LV venous anatomy on fluoroscopy venograms with epi- cardial surface on SPECT myocardial perfusion images for guiding CRT LV lead placement.JACC Cardiov. Imag., 7(12):1239–1248. https://doi.org/10.1016/j.jcmg.2014.09.002

[45] Zhou, Z.H., 2011. When semi-supervised learning meets en-semble learning. Front. Electr. Electron. Eng. China, 6(1): 6–16. https://doi.org/10.1007/978-3-642-02326-2_53

[46] Zhou, Z.H., Li, M., 2005. Semi-supervised regression with co-training.19th Int. Joint Conf. on Artificial Intelligence, p.908–913.

[47] Zhou, Z.H., Li, M., 2007. Semisupervised regression with cotraining-style algorithms. IEEE Trans. Knowl. Data Eng., 19(11):1479–1493. https://doi.org/10.1109/TKDE.2007.190644

RIGHTS & PERMISSIONS

2017 Zhejiang University and Springer-Verlag Berlin Heidelberg