Improved paraphrase generation via controllable latent diffusion

Abstract

Paraphrase generation strives to generate high-quality and diverse expressions of a given text, a domain where diffusion models excel. Though SOTA diffusion generation reconciles generation quality and diversity, textual diffusion suffers from a truncation issue that hinders efficiency and quality control. In this work, we propose Latent Diffusion Paraphraser (LDP), a novel paraphrase generation by modeling a controllable diffusion process given a learned latent space. LDP achieves superior generation efficiency compared to its diffusion counterparts. It can facilitate only input segments to ensure paraphrase semantics, improving the results without external features. Experiments show that LDP better reconciles paraphrase generation quality and diversity than baselines. Further analysis shows that our method is also helpful to other similar text generations and domain adaptations

Graphical abstract

Keywords

paraphrase generation / latent diffusion models / controlled generation

References

[1] Zhou J, Bhat S. Paraphrase generation: a survey of the state of the art. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 5075−5086

[2] Dong L, Mallinson J, Reddy S, Lapata M. Learning to paraphrase for question answering. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 875−886

[3] Buck C, Bulian J, Ciaramita M, Gajewski W, Gesmundo A, Houlsby N, Wang W. Ask the right questions: active question reformulation with reinforcement learning. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[4] Kumar A, Bhattamishra S, Bhandari M, Talukdar P. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In: Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019, 3609−3619

[5] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6000−6010

[6] Sun J, Ma X, Peng N. AESOP: paraphrase generation with adaptive syntactic control. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 5176−5189

[7] Chen M, Tang Q, Wiseman S, Gimpel K. Controllable paraphrase generation with a syntactic exemplar. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 5972−5984

[8] Bao Y, Zhou H, Huang S, Li L, Mou L, Vechtomova O, Dai X Y, Chen J. Generating sentences from disentangled syntactic and semantic spaces. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 6008−6019

[9] Du W, Zhao J, Wang L, Ji Y. Diverse text generation via variational encoder-decoder models with Gaussian process priors. 2022, arXiv preprint arXiv: 2204.01227

[10] Chen Y, Jiang H, Liu L, Wang R, Shi S, Xu R. MCPG: a flexible multi-level controllable framework for unsupervised paraphrase generation. In: Proceedings of the Findings of the Association for Computational Linguistics. 2022, 5948−5958

[11] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 574

[12] Nichol A Q, Dhariwal P. Improved denoising diffusion probabilistic models. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 8162−8171

[13] Li X L, Thickstun J, Gulrajani I, Liang P, Hashimoto T B. Diffusion-LM improves controllable text generation. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 313

[14] Austin J, Johnson D D, Ho J, Tarlow D, Van Den Berg R. Structured denoising diffusion models in discrete state-spaces. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 1376

[15] Ye J, Zheng Z, Bao Y, Qian L, Wang M. DINOISER: diffused conditional sequence learning by manipulating noises. 2023, arXiv preprint arXiv: 2302.10025

[16] Karras T, Aittala M, Laine S, Aila T. Elucidating the design space of diffusion-based generative models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1926

[17] Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 10674−10685

[18] Lovelace J, Kishore V, Wan C, Shekhtman E, Weinberger K Q. Latent diffusion for language generation. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2492

[19] Sancheti A, Srinivasan B V, Rudinger R. Entailment relation aware paraphrase generation. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 11258−11266

[20] Yang Q, Huo Z, Shen D, Cheng Y, Wang W, Wang G, Carin L. An end-to-end generative architecture for paraphrase generation. In: Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019, 3132−3142

[21] Hosking T, Tang H, Lapata M. Hierarchical sketch induction for paraphrase generation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 2489−2501

[22] Yang K, Liu D, Lei W, Yang B, Zhang H, Zhao X, Yao W, Chen B. GCPG: a general framework for controllable paraphrase generation. In: Proceedings of Findings of the Association for Computational Linguistics. 2022, 4035−4047

[23] Bowman S R, Vilnis L, Vinyals O, Dai A, Jozefowicz R, Bengio S. Generating sentences from a continuous space. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 2016, 10−21

[24] He Y, Yang T, Zhang Y, Shan Y, Chen Q. Latent video diffusion models for high-fidelity long video generation. 2023, arXiv preprint arXiv: 2211.13221

[25] Liu H, Chen Z, Yuan Y, Mei X, Liu X, Mandic D, Wang W, Plumbley M D. AudioLDM: text-to-audio generation with latent diffusion models. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 21450−21474

[26] Lai Z, Zhu X, Dai J, Qiao Y, Wang W . Mini-DALLE3: interactive text to image by prompting large language models. arXiv preprint arXiv: 2310, 0765, 3: 2023

[27] Song J, Meng C, Ermon S. Denoising diffusion implicit models. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[28] Lu C, Zhou Y, Bao F, Chen J, Li C, Zhu J. DPM-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 418

[29] Lu C, Zhou Y, Bao F, Chen J, Li C, Zhu J. DPM-solver++: fast solver for guided sampling of diffusion probabilistic models. 2022, arXiv preprint arXiv: 2211.01095

[30] Zhang L, Rao A, Agrawala M. Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, 3813−3824

[31] Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 7871−7880

[32] Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu P J . Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020, 21( 1): 140

[33] Xu J, Sun X, Zhang Z, Zhao G, Lin J. Understanding and improving layer normalization. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 394

[34] Peebles W, Xie S. Scalable diffusion models with transformers. In: Proceedings of 2023 IEEE/CVF International Conference on Computer Vision. 2023, 4172−4182

[35] Shazeer N. GLU variants improve transformer. 2020, arXiv preprint arXiv: 2002.05202

[36] Song Y, Sohl-Dickstein J, Kingma D P, Kumar A, Ermon S, Poole B. Score-based generative modeling through stochastic differential equations. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[37] Lan W, Qiu S, He H, Xu W. A continuously growing dataset of sentential paraphrases. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 1224−1234

[38] Zhang Y, Baldridge J, He L. PAWS: paraphrase adversaries from word scrambling. In: Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019, 1298−1308

[39] Wang T, Wan X. T-CVAE: transformer-based conditioned variational autoencoder for story completion. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 5233−5239

[40] Gong S, Li M, Feng J, Wu Z, Kong L. DiffuSeq: sequence to sequence text generation with diffusion models. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[41] Yuan H, Yuan Z, Tan C, Huang F, Huang S. SeqDiffuSeq: text diffusion with encoder-decoder transformers. 2022, arXiv preprint arXiv: 2212.10325

[42] Kumar S, Byrne W. Minimum Bayes-risk decoding for statistical machine translation. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. 2004, 169−176

[43] Touvron H, Martin L, Stone K, Albert P, Almahairi A et al . . Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307, 0928, 8: 2023

[44] Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A et al, . The llama 3 herd of models. 2024, arXiv preprint arXiv: 2407.21783

[45] Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: Low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[46] Papineni K, Roukos S, Ward T, Zhu W J. Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002, 311−318

[47] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI blog, 2019, 9

[48] Deshpande A, Aneja J, Wang L, Schwing A G, Forsyth D. Fast, diverse and accurate image captioning guided by part-of-speech. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 10687–10696

[49] Dou Y, Jiang C, Xu W. Improving large-scale paraphrase acquisition and generation. In: Proceedings of 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 9301–9323

[50] Zhang T, Kishore V, Wu F, Weinberger K Q, Artzi Y. BERTScore: evaluating text generation with BERT. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[51] He P, Liu X, Gao J, Chen W. Deberta: decoding-enhanced bert with disentangled attention. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[52] Shen L, Liu L, Jiang H, Shi S. On the evaluation metrics for paraphrase generation. In: Proceedings of 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 3178–3190

[53] Tan C H, Gu J C, Ling Z H. DiffuSIA: a spiral interaction architecture for encoder-decoder text diffusion. 2023, arXiv preprint arXiv: 2305.11517

[54] Tang Z, Wang P, Zhou K, Li J, Cao Z, Zhang M. Can diffusion model achieve better performance in text generation? Bridging the gap between training and inference! In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 11359–11386

[55] Mahabadi R K, Ivison H, Tae J, Henderson J, Beltagy I, Peters M E, Cohan A. TESS: text-to-text self-conditioned simplex diffusion. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. 2024, 2347–2361

[56] Qian L, Wang M, Liu Y, Zhou H. Diff-glat: diffusion glancing transformer for parallel sequence to sequence learning. 2022, arXiv preprint arXiv: 2212.10240

[57] Vladimir Vorobev M K. Chatgpt paraphrases dataset, 2023

[58] Rajpurkar P, Zhang J, Lopyrev K, Liang P. SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing. 2016, 2383−2392

[59] See A, Liu P J, Manning C D. Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 1073−1083

[60] Dhingra B, Mazaitis K, Cohen W W. Quasar: datasets for question answering by search and reading. 2017, arXiv preprint arXiv: 1707.03904

[61] Xie X, Lu X, Chen B. Multi-task learning for paraphrase generation with keyword and part-of-speech reconstruction. In: Proceedings of Findings of the Association for Computational Linguistics. 2022, 1234−1243

[62] Vijayakumar A K, Cogswell M, Selvaraju R R, Sun Q, Lee S, Crandall D, Batra D. Diverse beam search: decoding diverse solutions from neural sequence models. 2016, arXiv preprint arXiv: 1610.02424

[63] Fan A, Lewis M, Dauphin Y. Hierarchical neural story generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018, 889−898

RIGHTS & PERMISSIONS

Higher Education Press