Unsupervised statistical text simplification using pre-trained language modeling for initialization

Abstract

Unsupervised text simplification has attracted much attention due to the scarcity of high-quality parallel text simplification corpora. Recent an unsupervised statistical text simplification based on phrase-based machine translation system (UnsupPBMT) achieved good performance, which initializes the phrase tables using the similar words obtained by word embedding modeling. Since word embedding modeling only considers the relevance between words, the phrase table in UnsupPBMT contains a lot of dissimilar words. In this paper, we propose an unsupervised statistical text simplification using pre-trained language modeling BERT for initialization. Specifically, we use BERT as a general linguistic knowledge base for predicting similar words. Experimental results show that our method outperforms the state-of-the-art unsupervised text simplification methods on three benchmarks, even outperforms some supervised baselines.

Graphical abstract

Keywords

text simplification / pre-trained language modeling / BERT / word embeddings

References

[1] MartinL, dela Clergerie É, SagotB, BordesA. Controllable sentence simplification. In: Proceedings of the 12th Conference on Language Resources and Evaluation. 2020, 4689−4698

[2] NisioiS, ŠtajnerS, PonzettoS P, DinuL P. Exploring neural text simplification models. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 85–91

[3] WubbenS, van den BoschA, KrahmerE. Sentence simplification by monolingual machine translation. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers. 2012, 1015−1024

[4] Xu W , Napoles C , Pavlick E , Chen Q , Callison-Burch C . Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 2016, 4 : 401– 415

[5] ZhangX, LapataM. Sentence simplification with deep reinforcement learning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 584–594

[6] ZhuZ, BernhardD, GurevychI. A monolingual tree-based translation model for sentence simplification. In: Proceedings of the 23rd International Conference on Computational Linguistics. 2010, 1353−1361

[7] Xu W , Callison-Burch C , Napoles C . Problems in current text simplification research: new data can help. Transactions of the Association for Computational Linguistics, 2015, 3 : 283– 297

[8] SuryaS, MishraA, LahaA, JainP, SankaranarayananK. Unsupervised neural text simplification. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 2058−2068

[9] KumarD, MouL, GolabL, VechtomovaO. Iterative edit-based unsupervised sentence simplification. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 7918−7928

[10] Qiang J , Wu X . Unsupervised statistical text simplification. IEEE Transactions on Knowledge and Data Engineering, 2021, 33( 4): 1802– 1806

[11] MengY, ZhangY, HuangJ, XiongC, JiH, ZhangC, HanJ. Text classification using label names only: a language model self-training approach. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, 9006−9017

[12] PetroniF, RocktäschelT, LewisP, BakhtinA, WuY, MillerA H, RiedelS. Language models as knowledge bases?. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019, 2463−2473

[13] RobertsA, RaffelC, ShazeerN. How much knowledge can you pack into the parameters of a language model?. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, 5418−5426

[14] Zhang H, Khashabi D, Song Y, Roth D. TransOMCS: from linguistic graphs to commonsense knowledge. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence. 2020, 4004−4010

[15] KoehnP, HoangH, BirchA, Callison-BurchC, FedericoM, BertoldiN, CowanB, ShenW, MoranC, ZensR, DyerC, BojarO, ConstantinA, HerbstE. Moses: open source toolkit for statistical machine translation. In: Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions. 2007, 177−180

[16] ArtetxeM, LabakaG, AgirreE, ChoK. Unsupervised neural machine translation. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[17] PenningtonJ, SocherR, ManningC. GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014, 1532−1543

[18] Farr J N , Jenkins J J , Paterson D G . Simplification of flesch reading ease formula. Journal of Applied Psychology, 1951, 35( 5): 333– 337

[19] Heafield K. KenLM: faster and smaller language model queries. In: Proceedings of the 6th Workshop on Statistical Machine Translation. 2011, 187−197

[20] LampleG, OttM, ConneauA, DenoyerL, RanzatoM. Phrase-based & neural unsupervised machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 5039−5049

[21] LiD, ZhangY, PengH, ChenL, BrockettC, SunM T, DolanB. Contextualized perturbation for textual adversarial attack. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, 5053−5069

[22] GlavašG, ŠtajnerS. Simplifying lexical simplification: do we need simplified corpora?. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 2015, 63−68

[23] Brysbaert M , New B . Moving beyond Kučera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 2009, 41( 4): 977– 990

[24] Qiang J , Li Y , Zhu Y , Yuan Y , Wu X . Lexical simplification with pretrained encoders. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34( 5): 8649– 8656

[25] Qiang J , Lv X , Li Y , Yuan Y , Wu X . Chinese lexical simplification. IEEE/ACV Transactions on Audio, Speech, and Language Processing, 2021, 29 : 1819– 1828

[26] ZhaoS, MengR, HeD, AndiS, BambangP. Integrating transformer and paraphrase rules for sentence simplification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 3164−3173

[27] NarayanS, GardentC. Hybrid simplification using deep semantics and machine translation. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. 2014, 435−445

[28] GuoH, PasunuruR, BansalM. Dynamic multi-level multi-task learning for sentence simplification. In: Proceedings of the 27th International Conference on Computational Linguistics. 2018, 462−476

[29] DongY, LiZ, RezagholizadehM, CheungJ C K. EditNTS: an neural programmer-interpreter model for sentence simplification through explicit editing. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 3393−3402

[30] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog, 2019, 1(8): 9

[31] YangZ, DaiZ, YangY, CarbonellJ, SalakhutdinovR, LeQ V. XLNet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). 2019, 5754−5764

[32] DevlinJ, ChangM W, LeeK, ToutanovaK. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. 2019, 4171−4186

[33] LanZ, ChenM, GoodmanS, GimpelK, SharmaP, SoricutR. ALBERT: a lite BERT for self-supervised learning of language representations. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[34] LewisM, LiuY, GoyalN, GhazvininejadM, MohamedA, LevyO, StoyanovV, ZettlemoyerL. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2019, 7871−7880

[35] ScartonC, SpeciaL. Learning simplifications for specific target audiences. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018, 712−718

[36] NarayanS, GardentC. Unsupervised sentence simplification using deep semantics. In: Proceedings of the 9th International Natural Language Generation Conference. 2015, 111−120

[37] MartinL, FanA, dela Clergerie É, BordesA, SagotB. MUSS: multilingual unsupervised sentence simplification by mining paraphrases. 2021, arXiv preprint arXiv: 2005.00352

[38] ArtetxeM, LabakaG, AgirreE. Unsupervised statistical machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 3632−3642

[39] WenzekG, LachauxM A, ConneauA, ChaudharyV, GuzmánF, JoulinA, GraveE. CCNET: extracting high quality monolingual datasets from web crawl data. In: Proceedings of the 12th Language Resources and Evaluation Conference. 2020, 4003−4012

[40] PavlickE, Callison-BurchC. Simple PPDB: a paraphrase database for simplification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016, 143−148

RIGHTS & PERMISSIONS

Higher Education Press 2021