Improving neural sentence alignment with word translation

Abstract

Sentence alignment is a basic task in natural language processing which aims to extract high-quality parallel sentences automatically. Motivated by the observation that aligned sentence pairs contain a larger number of aligned words than unaligned ones, we treat word translation as one of the most useful external knowledge. In this paper, we show how to explicitly integrate word translation into neural sentence alignment. Specifically, this paper proposes three cross-lingual encoders to incorporate word translation: 1) Mixed Encoder that learns words and their translation annotation vectors over sequences where words and their translations are mixed alternatively; 2) Factored Encoder that views word translations as features and encodes words and their translations by concatenating their embeddings; and 3) Gated Encoder that uses gate mechanism to selectively control the amount of word translations moving forward. Experimentation on NIST MT and Opensubtitles Chinese-English datasets on both non-monotonicity and monotonicity scenarios demonstrates that all the proposed encoders significantly improve sentence alignment performance.

Keywords

sentence alignment / word translation / mixed encoder / factored encoder / gated encoder

References

[1] Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of International Conference on Learning Representations. 2015

[2] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, Kaiser L, Polosukhin I. Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems. 2017, 6000–6010

[3] Hermann K M, Blunsom P. Multilingual models for compositional distributed semantics. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. 2014, 58–68

[4] Nie J Y, Simard M, Isabelle P, Durand R. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the web. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 1999, 74–81

[5] Martino G D S, Romeo S, Barro�n-Cedeno A, Joty S, Marquez L, Moschitti A, Nakov P. Cross-language question re-ranking. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017, 1145–1148

[6] Wu D. Alignment. Handbook of Natural Language Processing. CRC Press. 2010

[7] Gregoire F, Langlais P. A deep neural network approach to parallel sentence extraction. 2017, arXiv preprint arXiv:1709.09783

[8] Grover J, Mitra P. Bilingual word embeddings with bucketed CNN for parallel sentence extraction. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics–Student Research Workshop. 2017, 11–16

[9] Ding Y, Li J, Zhou G. Word-pair relevance network for sentence alignment. Journal of Chinese Information Processing, 2019

[10] Ding Y, Li J H, Gong Z X, Zhou G D. Word-pair relevance modeling with multi-view neural attention mechanism for sentence alignment. Journal of Computer Science and Technology, 2019

[11] Liu L, Utiyama M, Finch A, Sumita E. Neural machine translation with supervised attention. In: Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers. 2016, 3093–3102

[12] Mi H, Wang Z, Ittycheriah A. Supervised attentions for neural machine translation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016, 2283–2288

[13] Arthur P, Neubig G, Nakamura S. Incorporating discrete translation lexicons into neural machine translation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016, 1557–1567

[14] Gale W A, Church K W. A program for aligning sentences in bilingual corpora. In: Proceedings of the Meeting on the Association for Computational Linguistics. 1991, 177–184

[15] Chen S F. Aligning sentences in bilingual corpora using lexical information. Computer Knowledge & Technology, 1993, 46(3): 9–16

[16] Wu D. Aligning a parallel English-Chinese corpus statistically with lexical criteria. Computer Science, 1994, 4(4): 80–87

[17] Moore R C. Fast and accurate sentence alignment of bilingual corpora. In: Processing of the 5th Conference of the Association for Machine Translation in the Americas. 2002, 135–144

[18] Brown P F, Pietra V J D, Pietra S A D, Mercer R L. The mathematics of statistical machine translation: parameter estimation. Computational linguistics, 1993, 19(2): 263–311

[19] Braune F, Fraser A. Improved unsupervised sentence alignment for symmetrical and asymmetrical parallel corpora. In: Processings of the 23rd International Conference on Computational Linguistics. 2010, 81–89

[20] Ma X. Champollion: a robust parallel text sentence aligner. In: Processings of the 5th International Conference on Language Resources and Evaluation. 2006, 489–492

[21] Li P, Sun M, Xue P. Fast-Champollion: a fast and robust sentence alignment algorithm. In: Proceedings of the 23rd International Conference on Computational Linguistics. 2010, 710–718

[22] Quan X, Kit C, Song Y. Non-monotonic sentence alignment via semisupervised learning. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. 2013, 622–630

[23] Chatterjee R, Negri M, Turchi M, Federico M, Specia L, Blain F. Guiding neural machine translation decoding with external knowledge. In: Proceedings of the 2nd Conference on Machine Translation. 2017, 157–168

[24] Nguyen T Q, Chiang D. Improving lexical choice in neural machine translation. 2017, arXiv preprint arXiv:1710.01329

[25] Wang X, Tu Z, Xiong D, Zhang M. Translating phrases in neural machine translation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 1421–1431

[26] Wang X, Lu Z, Tu Z, Li H, Xiong D, Zhang M. Neural machine translation advised by statistical machine translation. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017, 3330–3336

[27] Chen K, Wang R, Utiyama M, Liu L, Tamura A, Sumita E, Zhao T. Neural machine translation with source dependency representation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 2846–2852

[28] Li J, Xiong D, Tu Z, Zhu M, Zhang M, Zhou G. Modeling source syntax for neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 688–697

[29] Eriguchi A, Hashimoto K, Tsuruoka Y. Tree-to-sequence attentional neural machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016, 823–833

[30] Sennrich R, Haddow B. Linguistic input features improve neural machine translation. In: Proceedings of the 1st Conference on Machine Translation. 2016, 83–91

[31] Aharoni R, Goldberg Y. Towards string-to-tree neural machine translation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers). 2017, 132–140

[32] Chen H, Huang S, Chiang D, Chen J. Improved neural machine translation with a syntax-aware encoder and decoder. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 1936–1945

[33] Han D, Li J, Li Y, Zhang M, Zhou G. Explicitly modeling word translations in neural machine translation. ACM Transactions on Asian and Low-Resource Language Information Processing, 2019, 19(1): 1–17

[34] Langlais P, Simard M, Veronis J. Methods and practical issues in evaluating alignment techniques. In: Processings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics. 1998, 711–717

[35] Kit C, Webster J J, Sin K K, Pan H, Li H. Clause alignment for bilingual Hong Kong legal texts: a lexicalbased approach. International Journal of Corpus Linguistics, 2004, 9(1): 29–52

[36] Och F J, Ney H. A systematic comparison of various statistical alignment models. Computational Linguistics, 2003, 29(1): 19–51

[37] Cho K, van Merrienboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014, 1724–1734

[38] Sutskever I, Salakhutdinov R, Tenenbaum J B. Modelling relational data using bayesian clustered tensor factorization. In: Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2009, 1821–1828

[39] Jenatton R, Roux N L, Bordes A, Obozinski G. A latent factor model for highly multi-relational data. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. 2012, 3167–3175

[40] Collobert R, Weston J. A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning. 2008, 160–167

[41] Zou W Y, Socher R, Cer D, Manning C D. Bilingual word embeddings for phrase-based machine translation. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013, 1393–1398

RIGHTS & PERMISSIONS

Higher Education Press