Semantic image segmentation with fused CNN features

Abstract

Semantic image segmentation is a task to predict a category label for every image pixel. The key challenge of it is to design a strong feature representation. In this paper, we fuse the hierarchical convolutional neural network (CNN) features and the region-based features as the feature representation. The hierarchical features contain more global information, while the region-based features contain more local information. The combination of these two kinds of features significantly enhances the feature representation. Then the fused features are used to train a softmax classifier to produce per-pixel label assignment probability. And a fully connected conditional random field (CRF) is used as a post-processing method to improve the labeling consistency. We conduct experiments on SIFT flow dataset. The pixel accuracy and class accuracy are 84.4% and 34.86%, respectively.

References

[1] BuS, HanP, LiuZ, HanJ. Pattern Recognition, 2016, 59: 188

[2] BengioY. Found. Trends Mach. Learn., 2009, 2: 1

[3] ZhengS, JayasuamanaS, Romera-ParedesB, VineetV, SuZ, DuD, HuangC, TorrP. Conditional Random Fields as Recurrent Neural Networks, 2015, 1529

[4] ChenL-C, PapandreouG, KokkinosI, MurphyK, YuilleAL. Computer Science, 2014, 4: 357

[5] LongJ, ShelhamerE, DarrellT. Fully Convolutional Networks for Semantic Segmentation, IEEE Conference on Computer Vision and Pattern Recognition, 2015,

[6] CaesarH, JasperU, FerrariV. Region-Based Semantic Segmentation with End-to-End Training, Computer Vision–ECCV 2016, 2016,

[7] HariharanB, ArbelaezP, GirshickR, MalikJ. Simultaneous Detection and Segmentation, European Conference on Computer Vision, 2014, 297

[8] CarreiraJ, CaseiroR, BatistaR J, SminchisescuC. Semantic Segmentation with Second-Order Pooling, European Conference on Computer Vision, 2012, 430

[9] UijlingsJ, van de SandeK, GeversT, SmeuldersA. International Journal of Computer Vision, 2013, 104: 154

[10] GirshickR. Fast R-CNN, Computer Science, 2015,

[11] AchantaR, ShajiA, SmithK, LucchiA, FuaP, SusstrunkS. IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34: 2274

[12] KrahenbuhlP, KoltunV. Efficient Inference in Fully Connected CRFS with Gaussian Edge Potentials, 2012, 109

[13] LiuC, YuenJ, TorralbaA. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011, 33: 2368

[14] VedaldiA, LencK. Matconvnet Convolutional Neural Networks for Matlab, 2014, 689

[15] FarabetC, CouprieC, NajmanL, LeCunY. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2013, 35: 1915

[16] SinghG, KoseckaJNonparametric Scene Parsing with Adaptive Feature Relevance and Semantic ContextIEEE Computer Vision and Pattern Recognition, 2013, 3151

[17] GouldS, ZhaoJ, HeXSuperpixel Graph Label Transfer with Learned Distance MetricEuropean Conference on Computer Vision, 2014, 632

[18] GattaC, RomeroA, van de VeijerJUnrolling Loopy Top-Down Semantic Feedback in Convolutional Deep NetworksIEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, 504

[19] PinheiroP, CollobertR. Recurrent Convolutional Neural Networks for Scene Labeling, International Conference on Machine Learning, 2014, 82

[20] ByeonW, BreuelTM, RaueF, LiwickiM. Scene Labeling with LSTM Recurrent Neural Network, IEEE Computer Vision and Pattern Recognition, 2015, 3547