A general tail item representation enhancement framework for sequential recommendation

Abstract

Recently advancements in deep learning models have significantly facilitated the development of sequential recommender systems (SRS). However, the current deep model structures are limited in their ability to learn high-quality embeddings with insufficient data. Meanwhile, highly skewed long-tail distribution is very common in recommender systems. Therefore, in this paper, we focus on enhancing the representation of tail items to improve sequential recommendation performance. Through empirical studies on benchmarks, we surprisingly observe that both the ranking performance and training procedure are greatly hindered by the poorly optimized tail item embeddings. To address this issue, we propose a sequential recommendation framework named TailRec that enables contextual information of tail item well-leveraged and greatly improves its corresponding representation. Given the characteristics of the sequential recommendation task, the surrounding interaction records of each tail item are regarded as contextual information without leveraging any additional side information. This approach allows for the mining of contextual information from cross-sequence behaviors to boost the performance of sequential recommendations. Such a light contextual filtering component is plug-and-play for a series of SRS models. To verify the effectiveness of the proposed TailRec, we conduct extensive experiments over several popular benchmark recommenders. The experimental results demonstrate that TailRec can greatly improve the recommendation results and speed up the training process. The codes of our methods have been available ^†

See github.com/Mingyue-Cheng/TailRec website.

.

Graphical abstract

Keywords

sequential recommendation / long-tail distribution / training accelerating

Mingyue Cheng is currently pursuing PhD degree with the Anhui Province Key Laboratory of Big Data Analysis and Application (BDAA), School of Data Science, University of Science and Technology of China (USTC), China. Before that, he received BArts degree from Hefei University of Technology (HFUT), China. His general research interests span machine learning methods and their applications, especially focusing on sequence data mining and prediction, such sequential behaviors analysis, time series analysis. He has published more than 10 papers in referred journals and conference proceedings, e.g., the TKDE, WWW, SIGIR, ICDM. He also serves on the program committees of conferences, including SIGKDD and SIGIR

Qi Liu received the PhD degree from University of Science and Technology of China (USTC), China in 2013. He is currently a professor in the School of Computer Science and Technology at USTC. His general area of research is data mining and knowledge discovery. He has published prolifically in refereed journals and conference proceedings (e.g., TKDE, TOIS, KDD). He is an Associate Editor of IEEE TBD and Neurocomputing. He was the recipient of KDD’18 Best Student Paper Award and ICDM’11 Best Research Paper Award. He is a member of the Alibaba DAMO Academy Young Fellow. He was also the recipient of China Outstanding Youth Science Foundation in 2019

Wenyu Zhang is currently working toward the PhD degree in the School of Computer Science and Technology at University of Science and Technologyof China (USTC), China. He received his bachelor degree in communication engineering from Anhui Agricultural University (AHAU), China in 2021. His main research interests include recommender systems, and computer vision. He has published papers in referred conference proceedings, such as ACM MM’2021 and BMVC’2021

Zhiding Liu received his BE degree in computer science from University of Science and Technology of China (USTC), China in 2021. He is currently working toward the ME degree in the School of Computer Science and Technology at University of Science and Technology of China (USTC), China. His main research interests include data mining, recommender systems, and time series representation learning. He has published papers in referred conference proceedings, such as WWW’2022 and ICDM’2022

Hongke Zhao received the PhD degree from the University of Science and Technology of China (USTC), China. He is an associate professor with the College of Management and Economics, Tianjin University, China. His research interest includes data mining, data-driven management, knowledge and behavior computing. He has published more than 70 papers in refereed journals and conference proceedings, such as INFORMS Journal on Computing, IEEE Transactions on Knowledge and Data Engineering, IEEE Transactions on Evolutionary Computation, ACM Transactions on Information Systems, ACM SIGKDD, ACM WSDM, ACM SIGIR, IJCAI, AAAI, and IEEE ICDM. He was the recipient of Distinguished Dissertation Award Nomination of CAAI (2019), the Best Student Paper Award of CCML-2019

Enhong Chen received the PhD degree from the University of Science and Technology of China (USTC), China. He is a professor and vice dean of the School of Computer Science, USTC, China. His general area of research includes data mining and machine learning, social network analysis, and recommender systems. He has published more than 100 papers in refereed conferences and journals, including the IEEE Transactions on Knowledge and Data Engineering, the IEEE Transactions on Mobile Computing, the IEEE Transactions on Industrial Electronics, the ACM Transactions on Knowledge Discovery from Data, ACM SIGKDD, IEEE ICDM, and NIPS. He was on program committees of numerous conferences including SIGKDD, ICDM, and SDM. He received the Best Application Paper Award on KDD’2008, the Best Research Paper Award on ICDM’2011, and the Best of SDM’2015. His research is supported by the National Science Foundation for Distinguished Young Scholars of China

References

[1] Wang S, Hu L, Wang Y, Cao L, Sheng Q Z, Orgun M A. Sequential recommender systems: challenges, progress and prospects. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 6332−6338

[2] Wu S, Tang Y, Zhu Y, Wang L, Xie X, Tan T. Session-based recommendation with graph neural networks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. 2019, 43

[3] Xu E, Yu Z, Li N, Cui H, Yao L, Guo B . Quantifying predictability of sequential recommendation via logical constraints. Frontiers of Computer Science, 2023, 17( 5): 175612

[4] Zhaok X, Liu H, Fan W, Liu H, Tang J, Wang C, Chen M, Zheng X, Liu X, Yang X. AutoEmb: automated embedding dimensionality search in streaming recommendations. In: Proceedings of 2021 IEEE International Conference on Data Mining. 2021, 896−905

[5] Cheng M, Liu Q, Liu Z, Li Z, Luo Y, Chen E. FormerTime: hierarchical multi-scale representations for multivariate time series classification. In: Proceedings of the ACM Web Conference. 2023, 1437−1445

[6] Cheng M, Liu Q, Liu Z, Zhang H, Zhang R, Chen E. TimeMAE: self-supervised representations of time series with decoupled masked autoencoders. 2023, arXiv preprint arXiv: 2303.00320

[7] Sun Y, Yuan F, Yang M, Wei G, Zhao Z, Liu D. A generic network compression framework for sequential recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020, 1299−1308

[8] Chen L, Yuan F, Yang J, Ao X, Li C, Yang M. A user-adaptive layer selection framework for very deep sequential recommender models. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 3984−3991

[9] Zhang S, Yao D, Zhao Z, Chua T S, Wu F. CauseRec: counterfactual user sequence synthesis for sequential recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021, 367−377

[10] Yin J, Liu C, Wang W, Sun J, Hoi S C H. Learning transferrable parameters for long-tailed sequential user behavior modeling. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 359−367

[11] Kim Y, Kim K, Park C, Yu H. Sequential and diverse recommendation with long tail. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 2740−2746

[12] Fan Z, Liu Z, Zhang J, Xiong Y, Zheng L, Yu P S. Continuous-time sequential recommendation with temporal graph collaborative transformer. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, 433−442

[13] Zipf G K. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Xue C F, trans. Shanghai: Shanghai People’s Publishing House, 2016

[14] Yin H, Cui B, Li J, Yao J, Chen C . Challenging the long tail recommendation. Proceedings of the VLDB Endowment, 2012, 5( 9): 896–907

[15] Liu Z, Cheng M, Li Z, Liu Q, Chen E. One person, one model-learning compound router for sequential recommendation. In: Proceedings of IEEE International Conference on Data Mining. 2022, 289−298

[16] Wei T, Feng F, Chen J, Wu Z, Yi J, He X. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 1791−1800

[17] Liu S, Zheng Y. Long-tail session-based recommendation. In: Proceedings of the 14th ACM Conference on Recommender Systems. 2020, 509−514

[18] Jang S, Lee H, Cho H, Chung S. CITIES: contextual inference of tail-item embeddings for sequential recommendation. In: Proceedings of the 20th IEEE International Conference on Data Mining. 2020, 202−211

[19] Kang W C, McAuley J. Self-attentive sequential recommendation. In: Proceedings of 2018 IEEE International Conference on Data Mining. 2018, 197−206

[20] Rendle S, Freudenthaler C, Schmidt-Thieme L. Factorizing personalized Markov chains for next-basket recommendation. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 811−820

[21] Hidasi B, Karatzoglou A, Baltrunas L, Tikk D. Session-based recommendations with recurrent neural networks. In: Proceedings of the 4th International Conference on Learning Representations. 2016

[22] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6000−6010

[23] Cheng M, Liu Z, Liu Q, Ge S, Chen E. Towards automatic discovering of deep hybrid network architecture for sequential recommendation. In: Proceedings of the ACM Web Conference 2022. 2022, 1923−1932

[24] Sun F, Liu J, Wu J, Pei C, Lin X, Ou W, Jiang P. BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019, 1441−1450

[25] Cheng M, Yuan F, Liu Q, Ge S, Li Z, Yu R, Lian D, Yuan S, Chen E. Learning recommender systems with implicit feedback via soft target enhancement. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021, 575−584

[26] Xue F, He X, Wang X, Xu J, Liu K, Hong R . Deep item-based collaborative filtering for top-N recommendation. ACM Transactions on Information Systems, 2019, 37( 3): 33

[27] Cai Y, Cui Z, Wu S, Lei Z, Ma X. Represent items by items: An enhanced representation of the target item for recommendation. 2021, arXiv preprint arXiv: 2104.12483

[28] He K, Fan H, Wu Y, Xie S, Girshick R. Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 9726−9735

[29] Liu Q, Zeng Y, Mokhosi R, Zhang H. STAMP: short-term attention/memory priority model for session-based recommendation. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018, 1831−1839

[30] De Souza Pereira Moreira G, Rabhi S, Lee J M, Ak R, Oldridge E. Transformers4Rec: bridging the gap between NLP and sequential/session-based recommendation. In: Proceedings of the 15th ACM Conference on Recommender Systems. 2021, 143−153

[31] Tang J, Wang K. Personalized top-n sequential recommendation via convolutional sequence embedding. In: Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 2018, 565−573

[32] Cheng M, Yuan F, Liu Q, Xin X, Chen E. Learning transferable user representations with sequential behaviors via contrastive pre-training. In: Proceedings of 2021 IEEE International Conference on Data Mining (ICDM). 2021, 51−60

[33] Li J, Ren P, Chen Z, Ren Z, Lian T, Ma J. Neural attentive session-based recommendation. In: Proceedings of 2017 ACM on Conference on Information and Knowledge Management. 2017, 1419−1428

[34] Yuan F, Karatzoglou A, Arapakis I, Jose J M, He X. A simple convolutional generative network for next item recommendation. In: Proceedings of the 12th ACM International Conference on Web Search and Data Mining. 2019, 582−590

[35] Rendle S, Freudenthaler C, Gantner Z, Schmidt-Thieme L. BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence. 2012, 452−461

[36] Zhang Y, Cheng D Z, Yao T, Yi X, Hong L, Chi E H. A model of two tales: dual transfer learning framework for improved long-tail item recommendation. In: Proceedings of the Web Conference 2021. 2021, 2220−2231

[37] Zhao Z, Chen J, Zhou S, He X, Cao X, Zhang F, Wu W. Popularity bias is not always evil: disentangling benign and harmful bias for recommendation. IEEE Transactions on Knowledge and Data Engineering, 2022, doi: 10.1109/TKDE.2022.3218994

[38] He X, Chua T S. Neural factorization machines for sparse predictive analytics. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017, 355−364

[39] Pi Q, Zhou G, Zhang Y, Wang Z, Ren L, Fan Y, Zhu X, Gai K. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020, 2685−2692

[40] He R, Fang C, Wang Z, McAuley J. Vista: a visually, socially, and temporally-aware model for artistic recommendation. In: Proceedings of the 10th ACM Conference on Recommender Systems. 2016, 309–316

[41] Ying H, Zhuang F, Zhang F, Liu Y, Xu G, Xie X, Xiong H, Wu J. Sequential recommender system based on hierarchical attention network. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018, 3926–3932

[42] Gu Y, Lei T, Barzilay R, Jaakkola T. Learning to refine text based recommendations. In: Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing. 2016, 2103−2108

[43] Tan Y K, Xu X, Liu Y. Improved recurrent neural networks for session-based recommendations. In: Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. 2016, 17−22

[44] Huang J, Zhao W X, Dou H, Wen J R, Chang E Y. Improving sequential recommendation with knowledge-enhanced memory networks. In: Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018, 505−514

[45] Wang J, Yuan F, Chen J, Wu Q, Yang M, Sun Y, Zhang G. StackRec: efficient training of very deep sequential recommender models by iterative stacking. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021, 357−366

[46] Liang Y, Song Q, Zhao Z, Zhou H, Gong M . BA-GNN: Behavior-aware graph neural network for session-based recommendation. Frontiers of Computer Science, 2023, 17( 6): 176613

[47] Brynjolfsson E, Hu Y J, Smith M D . From niches to riches: anatomy of the long tail. Sloan Management Review, 2006, 47( 4): 67–71

[48] Liang D, Charlin L, Blei D M. Causal inference for recommendation. In: Proceedings of Causation: Foundation to Application, Workshop at UAI. 2016

[49] Abdollahpouri H, Burke R, Mobasher B. Controlling popularity bias in learning-to-rank recommendation. In: Proceedings of the 11th ACM Conference on Recommender Systems. 2017, 42−46

[50] Adomavicius G, Kwon Y . Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on Knowledge and Data Engineering, 2012, 24( 5): 896–911

[51] Bai B, Fan Y, Tan W, Zhang J . DLTSR: a deep learning framework for recommendations of long-tail web services. IEEE Transactions on Services Computing, 2020, 13( 1): 73–85

[52] Li J, Lu K, Huang Z, Shen H T. Two birds one stone: on both cold-start and long-tail recommendation. In: Proceedings of the 25th ACM International Conference on Multimedia. 2017, 898−906

Acknowledgements

This research was supported by grant from the National Key R&D Program of China (No. 2021YFF0901003). The authors also appreciate the partial support from Hefei AI Computing Center (Project Team).

Competing interests

The authors declare that they have no competing interests or financial conflicts to disclose.

RIGHTS & PERMISSIONS

2024 Higher Education Press