Semantic image segmentation with fused CNN features

Abstract

Semantic image segmentation is a task to predict a category label for every image pixel. The key challenge of it is to design a strong feature representation. In this paper, we fuse the hierarchical convolutional neural network (CNN) features and the region-based features as the feature representation. The hierarchical features contain more global information, while the region-based features contain more local information. The combination of these two kinds of features significantly enhances the feature representation. Then the fused features are used to train a softmax classifier to produce per-pixel label assignment probability. And a fully connected conditional random field (CRF) is used as a post-processing method to improve the labeling consistency. We conduct experiments on SIFT flow dataset. The pixel accuracy and class accuracy are 84.4% and 34.86%, respectively.

References

[1] Bu S, Han P, Liu Z, Han J. Pattern Recognition. 2016, 59: 188

[2] Bengio Y. Found. Trends Mach. Learn.. 2009, 2: 1

[3] Zheng S, Jayasuamana S, Romera-Paredes B, Vineet V, Su Z, Du D, Huang C, Torr P. Conditional Random Fields as Recurrent Neural Networks. 20151529

[4] Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Computer Science. 2014, 4: 357

[5] Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation, IEEE Conference on Computer Vision and Pattern Recognition. 2015

[6] Caesar H, Jasper U, Ferrari V. Region-Based Semantic Segmentation with End-to-End Training, Computer Vision–ECCV 2016. 2016

[7] Hariharan B, Arbelaez P, Girshick R, Malik J. Simultaneous Detection and Segmentation, European Conference on Computer Vision. 2014297

[8] Carreira J, Caseiro R, Batista R J, Sminchisescu C. Semantic Segmentation with Second-Order Pooling, European Conference on Computer Vision. 2012430

[9] Uijlings J, van de Sande K, Gevers T, Smeulders A. International Journal of Computer Vision. 2013, 104: 154

[10] Girshick R. Fast R-CNN, Computer Science. 2015

[11] Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S. IEEE Trans. Pattern Anal. Mach. Intell.. 2012, 34: 2274

[12] Krahenbuhl P, Koltun V. Efficient Inference in Fully Connected CRFS with Gaussian Edge Potentials. 2012109

[13] Liu C, Yuen J, Torralba A. IEEE Transactions on Pattern Analysis & Machine Intelligence. 2011, 33: 2368

[14] Vedaldi A, Lenc K. Matconvnet Convolutional Neural Networks for Matlab. 2014689

[15] Farabet C, Couprie C, Najman L, LeCun Y. IEEE Transactions on Pattern Analysis & Machine Intelligence. 2013, 35: 1915

[16] Singh G, Kosecka J. Nonparametric Scene Parsing with Adaptive Feature Relevance and Semantic Context. IEEE Computer Vision and Pattern Recognition. 20133151

[17] Gould S, Zhao J, He X. Superpixel Graph Label Transfer with Learned Distance Metric. European Conference on Computer Vision. 2014632

[18] Gatta C, Romero A, van de Veijer J. Unrolling Loopy Top-Down Semantic Feedback in Convolutional Deep Networks. IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014504

[19] Pinheiro P, Collobert R. Recurrent Convolutional Neural Networks for Scene Labeling, International Conference on Machine Learning. 201482

[20] Byeon W, Breuel TM, Raue F, Liwicki M. Scene Labeling with LSTM Recurrent Neural Network, IEEE Computer Vision and Pattern Recognition. 20153547