Global-local feature attention network with reranking strategy for image caption generation

Abstract

In this paper, a novel framework, named as global-local feature attention network with reranking strategy (GLAN-RS), is presented for image captioning task. Rather than only adopting unitary visual information in the classical models, GLAN-RS explores the attention mechanism to capture local convolutional salient image maps. Furthermore, we adopt reranking strategy to adjust the priority of the candidate captions and select the best one. The proposed model is verified using the Microsoft Common Objects in Context (MSCOCO) benchmark dataset across seven standard evaluation metrics. Experimental results show that GLAN-RS significantly outperforms the state-of-the-art approaches, such as multimodal recurrent neural network (MRNN) and Google NIC, which gets an improvement of 20% in terms of BLEU4 score and 13 points in terms of CIDER score.

References

[1] JINAGY.-f, ZHANGH., XUEY.-b, ZHOUM., XUG.-p, GAOZ.. Journal of Optoelectronics·Laser, 2016, 27: 224

[2] SUNJ.-d, LIH.-h, JINJ.-l. Journal of Optoelectronics·Laser, 2017, 28: 441

[3] AlexK., IlyaS., GeoffreyH.. ImageNet Classification with Deep Convolutional Neural Networks, 2012, 1097

[4] MaoJ., XuW., YangY., WangJ., HuangZ., YuilleA.. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN), 2014,

[5] XuK., BaJ., KirosR., ChoK., CourvilleA. C., SalakhutdinovR., ZemelR. S., BengioY.. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, 2015,

[6] VinyalsO., ToshevA., BengioS., ErhanD.. Show and Tell: A Neural Image Caption Generator, 2015,

[7] SzegedyC., LiuW., JiaY.. Sermanet Pierre, Reed Scott, Anguelov Dragomir, Erhan Dumitru, Vanhoucke Vincent and Rabinovich Andrew, Going Deeper with Convolutions, 2014,

[8] SimonyanK., ZissermanA.. Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014,

[9] JunyoungC., CaglarG., KyungHyunC., YoshuaB.. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, 2014,

[10] DevlinJ., GuptaS., GirshickR., MitchellM., ZitnickC. L.. Exploring Nearest Neighbor Approaches for Image Captioning, 2015,

[11] LinT., MaireM., BelongieS., HaysJ., PeronaP., RamananD., DollarP., ZitnickC. L.. Microsoft COCO: Common Objects in Context, 2014, 740

[12] ChenX., FangH., LinT. Y., VedantamR., GuptaS., DollarP., ZitnickC. L.. Microsoft COCO Captions: Data Collection and Evaluation Server, 2015,

[13] RashtchianC., YoungP., HodoshM., HockenmaierJ.. Collecting Image Captions Using Amazon’s Mechanical Turk, 2010, 139

[14] KishoreP., SalimR., ToddW., ZhuW.. A Method for Automatic Evaluation of Machine Translation, 2002, 311

[15] VedantamR., ZitnickC. L., ParikhD.. CIDEr: Consensus-Based Image Description Evaluation, 2015, 4566

[16] LinC.-Y.. Rouge: A package for automatic evaluation of summaries, 2004,

[17] BanerjeeS., LavieA.. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments, 2005, 65

[18] JiaX., GavvesE., FernandoB., TuytelaarsT.. Guiding the Long-Short Term Memory Model for Image Caption Generation, 2015, 2407

This work has been supported by the Innovative Application and Research Project of Guangdong Province (No.2016KZDXM013), and the Science & Technology Project of Shantou City (No.A201400150). This paper was presented in part at the CCF Chinese Conference on Computer Vision, Tianjin, 2017. This paper was recommended by the program committee.