Semantic image segmentation with fused CNN features

Abstract

Semantic image segmentation is a task to predict a category label for every image pixel. The key challenge of it is to design a strong feature representation. In this paper, we fuse the hierarchical convolutional neural network (CNN) features and the region-based features as the feature representation. The hierarchical features contain more global information, while the region-based features contain more local information. The combination of these two kinds of features significantly enhances the feature representation. Then the fused features are used to train a softmax classifier to produce per-pixel label assignment probability. And a fully connected conditional random field (CRF) is used as a post-processing method to improve the labeling consistency. We conduct experiments on SIFT flow dataset. The pixel accuracy and class accuracy are 84.4% and 34.86%, respectively.

References

[1] BuS, HanP, LiuZ, HanJ. Pattern Recognition, 2016, 59: 188

[2] BengioY. Found. Trends Mach. Learn., 2009, 2: 1

[3] ZhengS, JayasuamanaS, Romera-ParedesB, VineetV, SuZ, DuD, HuangC, TorrP. Conditional Random Fields as Recurrent Neural Networks, 2015, 1529

[4] ChenL-C, PapandreouG, KokkinosI, MurphyK, YuilleAL. Computer Science, 2014, 4: 357

[5] LongJ, ShelhamerE, DarrellT. Fully Convolutional Networks for Semantic Segmentation, IEEE Conference on Computer Vision and Pattern Recognition, 2015,

[6] CaesarH, JasperU, FerrariV. Region-Based Semantic Segmentation with End-to-End Training, Computer Vision–ECCV 2016, 2016,

[7] HariharanB, ArbelaezP, GirshickR, MalikJ. Simultaneous Detection and Segmentation, European Conference on Computer Vision, 2014, 297

[8] CarreiraJ, CaseiroR, BatistaR J, SminchisescuC. Semantic Segmentation with Second-Order Pooling, European Conference on Computer Vision, 2012, 430

[9] UijlingsJ, van de SandeK, GeversT, SmeuldersA. International Journal of Computer Vision, 2013, 104: 154

[10] GirshickR. Fast R-CNN, Computer Science, 2015,

[11] AchantaR, ShajiA, SmithK, LucchiA, FuaP, SusstrunkS. IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34: 2274

[12] KrahenbuhlP, KoltunV. Efficient Inference in Fully Connected CRFS with Gaussian Edge Potentials, 2012, 109

[13] LiuC, YuenJ, TorralbaA. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011, 33: 2368

[14] VedaldiA, LencK. Matconvnet Convolutional Neural Networks for Matlab, 2014, 689

[15] FarabetC, CouprieC, NajmanL, LeCunY. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2013, 35: 1915

[16] SinghG, KoseckaJNonparametric Scene Parsing with Adaptive Feature Relevance and Semantic ContextIEEE Computer Vision and Pattern Recognition, 2013, 3151

[17] GouldS, ZhaoJ, HeXSuperpixel Graph Label Transfer with Learned Distance MetricEuropean Conference on Computer Vision, 2014, 632

[18] GattaC, RomeroA, van de VeijerJUnrolling Loopy Top-Down Semantic Feedback in Convolutional Deep NetworksIEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, 504

[19] PinheiroP, CollobertR. Recurrent Convolutional Neural Networks for Scene Labeling, International Conference on Machine Learning, 2014, 82

[20] ByeonW, BreuelTM, RaueF, LiwickiM. Scene Labeling with LSTM Recurrent Neural Network, IEEE Computer Vision and Pattern Recognition, 2015, 3547

This work has been supported by the National Natural Science Foundation of China (Nos.U1509207, 61325019, 61472278, 61403281 and 61572357), and the Key Project of Natural Science Foundation of Tianjin (No.14JCZDJC31700).