A multi-projection recurrent model for hypernym detection and discovery

Abstract

Hypernym detection and discovery are fundamental tasks in natural language processing. The former task aims to identify all possible hypernyms of a given hyponym term, whereas the latter attempts to determine whether the given two terms hold a hypernymy relation or not. Existing research on hypernym detection and discovery tasks projects a term into various semantic spaces with single mapping functions. Despite their success, these methods may not be adequate in capturing complex semantic relevance between hyponym/hypernymy pairs in two aspects. First, they may fall short in modeling the hierarchical structure in the hypernymy relations, which may help them learn better term representations. Second, the polysemy phenomenon that hypernyms may express distinct senses is understudied. In this paper, we propose a Multi-Projection Recurrent model (MPR) to simultaneously capture the hierarchical relationships between terms and deal with diverse senses caused by the polysemy phenomenon. Specifically, we build a multi-projection mapping block to deal with the polysemy phenomenon, which learns various word senses by multiple projections. Besides, we adopt a hierarchy-aware recurrent block with the recurrent operation followed by a multi-hop aggregation module to capture the hierarchical structure of hypernym relations. Experiments on 11 benchmark datasets in various task settings illustrate that our multi-projection recurrent model outperforms the baselines. The experimental analysis and case study demonstrate that our multi-projection module and the recurrent structure are effective for hypernym detection and discovery tasks.

Graphical abstract

Keywords

natural language processing / hypernym detection / recurrent model

References

[1] Snow R, Jurafsky D, Ng A Y. Semantic taxonomy induction from heterogenous evidence. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. 2006, 801−808

[2] Navigli R, Velardi P, Faralli S. A graph-based algorithm for inducing lexical taxonomies from scratch. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence. 2011, 1872−1877

[3] Qiang J, Zhang F, Li Y, Yuan Y, Zhu Y, Wu X . Unsupervised statistical text simplification using pre-trained language modeling for initialization. Frontiers of Computer Science, 2023, 17( 1): 171303

[4] Qiang J, Li Y, Li Y, Yuan Y, Zhu Y . Lexical simplification via single-word generation. Frontiers of Computer Science, 2023, 17( 6): 176347

[5] Yahya M, Berberich K, Elbassuoni S, Weikum G. Robust question answering over the web of linked data. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management. 2013, 1107−1116

[6] Gupta D, Pujari R, Ekbal A, Bhattacharyya P, Maitra A, Jain T, Sengupta S. Can taxonomy help? improving semantic question matching using question taxonomy. In: Proceedings of the 27th International Conference on Computational Linguistics. 2018, 499−513

[7] Zhang M, He T, Dong M . Meta-path reasoning of knowledge graph for commonsense question answering. Frontiers of Computer Science, 2024, 18( 1): 181303

[8] Hearst M A. Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the 14th Conference on Computational Linguistics. 1992: 539−545

[9] Geffet M, Dagan I. The distributional inclusion hypotheses and lexical entailment. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. 2005, 107−114

[10] Weeds J, Weir D, McCarthy D. Characterising measures of lexical distributional similarity. In: Proceedings of the 20th International Conference on Computational Linguistics. 2004, 1015−es

[11] Roller S, Kiela D, Nickel M. Hearst patterns revisited: Automatic hypernym detection from large text corpora. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018, 358−363

[12] Dash S, Chowdhury M F M, Gliozzo A, Mihindukulasooriya N, Fauceglia N R. Hypernym detection using strict partial order networks. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 7626−7633

[13] Camacho-Collados J, Bovi C D, Espinosa-Anke L, Oramas S, Pasini T, Santus E, Shwartz V, Navigli R, Saggion H. SemEval-2018 task 9: Hypernym discovery. In: Proceedings of the 12th International Workshop on Semantic Evaluation. 2018, 712−724

[14] Seitner J, Bizer C, Eckert K, Faralli S, Meusel R, Paulheim H, Ponzetto S P. A large database of hypernymy relations extracted from the web. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16). 2016, 360−367

[15] Snow R, Jurafsky D, Ng A Y. Learning syntactic patterns for automatic hypernym discovery. In: Proceedings of the 17th International Conference on Neural Information Processing Systems. 2004, 1297−1304

[16] Weeds J, Weir D. A general framework for distributional similarity. In: Proceedings of 2003 Conference on Empirical Methods in Natural Language Processing. 2003, 81−88

[17] Kotlerman L, Dagan I, Szpektor I, Zhitomirsky-Geffet M . Directional distributional similarity for lexical inference. Natural Language Engineering, 2010, 16( S4): 359–389

[18] Lenci A, Benotto G. Identifying hypernyms in distributional semantic spaces. In: Proceedings of *SEM 2012: The 1st Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012). 2012, 75−79

[19] Santus E, Lenci A, Lu Q, Walde S S I. Chasing hypernyms in vector spaces with entropy. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Volume 2: Short Papers. 2014, 38−42

[20] Roller S, Erk K, Boleda G. Inclusive yet selective: supervised distributional hypernymy detection. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers. 2014: 1025−1036.

[21] Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 3111−3119

[22] Fu R, Guo J, Qin B, Che W, Wang H, Liu T. Learning semantic hierarchies via word embeddings. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014, 1199−1209

[23] Le M, Roller S, Papaxanthos L, Kiela D, Nickel M. Inferring concept hierarchies from text corpora via hyperbolic embeddings. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 3231−3241

[24] Nickel M, Kiela D. Poincaré embeddings for learning hierarchical representations. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6341−6350

[25] Ustalov D, Arefyev N, Biemann C, Panchenko A. Negative sampling improves hypernymy extraction based on projection learning. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. 2017, 543−550

[26] Yamane J, Takatani T, Yamada H, Miwa M, Sasaki Y. Distributional hypernym generation by jointly learning clusters and projections. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 2016, 1871−1879

[27] Bernier-Colborne G, Barrière C. CRIM at SemEval-2018 task 9: a hybrid approach to hypernym discovery. In: Proceedings of the 12th International Workshop on Semantic Evaluation. 2018, 725−731

[28] Held W, Habash N. The effectiveness of simple hybrid systems for hypernym discovery. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 3362−3367

[29] Yu C, Han J, Wang P, Song Y, Zhang H, Ng W, Shi S. When Hearst is not enough: Improving hypernymy detection from corpus with distributional models. In: Proceedings of 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, 6208−6217

[30] Shi H, Xie M, Huang S . Robust AUC maximization for classification with pairwise confidence comparisons. Frontiers of Computer Science, 2024, 18( 4): 184317

[31] Yang M, Liu Q, Sun X, Shi N, Xue H . Towards kernelizing the classifier for hyperbolic data. Frontiers of Computer Science, 2024, 18( 1): 181301

[32] Ba J L, Kiros J R, Hinton G E. Layer normalization. 2016, arXiv preprint arXiv: 1607.06450

[33] Levy O, Remus S, Biemann C, Dagan I. Do supervised distributional methods really learn lexical inference relations?. In: Proceedings of 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015, 970−976

[34] Baroni M, Lenci A. How we BLESSed distributional semantic evaluation. In: Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics. 2011, 1−10

[35] Baroni M, Bernardi R, Do N Q, Shan C C. Entailment above the word level in distributional semantics. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics. 2012, 23−32

[36] Santus E, Yung F, Lenci A, Huang C R. EVALution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In: Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications. 2015, 64−69

[37] Shwartz V, Goldberg Y, Dagan I. Improving hypernymy detection with an integrated path-based and distributional method. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2016, 2389−2398

[38] Weeds J, Clarke D, Reffin J, Weir D, Keller B. Learning to distinguish hypernyms and co-hyponyms. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. 2014, 2249−2259

[39] Espinosa-Anke L, Camacho-Collados J, Bovi C D, Saggion H. Supervised distributional hypernym discovery via domain adaptation. In: Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing. 2016, 424−435

[40] Berend G, Makrai M, Földiák P. 300-sparsans at SemEval-2018 task 9: Hypernymy as interaction of sparse attributes. In: Proceedings of the 12th International Workshop on Semantic Evaluation. 2018, 928−934

[41] Qiu W, Chen M, Li L, Si L. NLP_HZ at SemEval-2018 task 9: a nearest neighbor approach. In: Proceedings of the 12th International Workshop on Semantic Evaluation. 2018, 909−913

[42] Kingma D P, Ba J. Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations. 2015

[43] He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of 2015 IEEE International Conference on Computer Vision. 2015, 1026−1034

[44] Pennington J, Socher R, Manning C. GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014, 1532−1543

RIGHTS & PERMISSIONS

Higher Education Press