Personal Style Guided Outfit Recommendation with Multi-Modal Fashion Compatibility Modeling

Abstract

A personalized outfit recommendation has emerged as a hot research topic in the fashion domain.However, existing recommendations do not fully exploit user style preferences.Typically, users prefer particular styles such as casual and athletic styles, and consider attributes like color and texture when selecting outfits.To achieve personalized outfit recommendations in line with user style preferences, this paper proposes a personal style guided outfit recommendation with multi-modal fashion compatibility modeling, termed as PSGNet.Firstly, a style classifier is designed to categorize fashion images of various clothing types and attributes into distinct style categories.Secondly, a personal style prediction module extracts user style preferences by analyzing historical data.Then, to address the limitations of single-modal representations and enhance fashion compatibility, both fashion images and text data are leveraged to extract multi-modal features.Finally, PSGNet integrates these components through Bayesian personalized ranking(BPR) to unify the personal style and fashion compatibility, where the former is used as personal style features and guides the output of the personalized outfit recommendation tailored to the target user.Extensive experiments on large-scale datasets demonstrate that the proposed model is efficient on the personalized outfit recommendation.

Keywords

personalized outfit recommendation / fashion compatibility modeling / style preference / multi-modal representation / Bayesian personalized ranking(BPR) / style classifier

References

[1] CHEN H J, SHUAI H H, CHENG W H. A survey of artificial intelligence in fashion[J]. IEEE Signal Processing Magazine, 2023, 40(3):64-73.

[2] CHEN X, CHEN H, XU H, et al. Personalized fashion recommendation with visual explanations based on multi-modal attention network:towards visually explainable recommendation[C]//PProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York,USA: ACM, 2019:765-774.

[3] CHENG W H, SONG S J, CHEN C Y, et al. Fashion meets computer vision:a survey[EB/OL].(2021-01-28)[2024-04-01]. https://arxiv.org/abs/2003.13988.

[4] JEON Y, JIN S, HAN K. FANCY:human-centered, FANCY:human-centered,deep learning-based framework for fashion style analysis[C]//Proceedings of the Web Conference 2021. New York,USA: ACM, 2021:2367-2378.

[5] LIU Z, LUO P, QIU S, et al. DeepFashion:powering robust clothes recognition and retrieval with rich annotations[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. New York,USA: IEEE, 2016:1096-1104.

[6] HE R N, MCAULEY J. VBPR:visual Bayesian personalized ranking from implicit feedback[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1):144-150.

[7] SONG X M, HAN X J, LI Y K, et al. GP-BPR:personalized compatibility modeling for clothing matching[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York,USA: ACM, 2019:320-328.

[8] LU Z, HU Y, YU C, et al. Personalized fashion recommendation with discrete content-based tensor factorization[J]. IEEE Transactions on Multimedia, 2023,25:5053-5064.

[9] SAGAR D, GARG J, KANSAL P, et al. PAI-BPR:personalized outfit recommendation scheme with attribute-wise interpretability[C]//2020 IEEE Sixth International Conference on Multimedia Big Data. New York,USA: IEEE, 2020:221-230.

[10] RENDLE S, FREUDENTHALER C, GANTNER Z, et al. BPR:Bayesian personalized ranking from implicit feedback[C]//Proceedings of the International Conference on Uncertainty in Artificial Intelligence. [S.l.]: AUAI Press, 2009,452-461.

[11] GONG W, KHALID L. Aesthetics, Aesthetics,personalization and recommendation:a survey on deep learning in fashion[EB/OL].(2021-01-20)[2024-04-01]. https://arxiv.org/abs/2101.08301v1.

[12] GU X L, GAO F, TAN M, et al. Fashion analysis and understanding with artificial intelligence[J]. Information Processing & Management, 2020, 57(5):102276.

[13] DELDJOO Y, NAZARY F, RAMISA A, et al. A review of modern fashion recommender systems[EB/OL].(2022-02-06) [2024-04-01]. https://arxiv.org/abs/2202.02757v3.

[14] HAN X T, WU Z X, JIANG Y G, et al. Learning fashion compatibility with bidirectional LSTMs[EB/OL].(2017-07-08) [2024-04-01]. http://arxiv.org/abs/1707.05691.

[15] MCAULEY J, TARGETT C, SHI Q F, et al. Image-based recommendations on styles and substitutes[EB/OL].(2015-06-15) [2024-04-01]. https://arxiv.org/abs/1506.04757v1.

[16] SONG X M, FENG F L, LIU J H, et al. NeuroStylist:neural compatibility modeling for clothing matching[C]//PProceedings of the 25th ACM International Conference on Multimedia. New York,USA: ACM, 2017:753-761.

[17] CUCURULL G, TASLAKIAN P, VAZQUEZ D. Context-aware visual compatibility prediction[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York,USA: IEEE, 2019:12609-12618.

[18] LIN Y S, MOOSAEI M, YANG H. OutfitNet:fashion outfit recommendation with attention-based multiple instance learning[C]//Proceedings of the Web Conference 2020. New York,USA: ACM, 2020:77-87.

[19] FENG Z L, YU Z Y, YANG Y Z, et al. Interpretable partitioned embedding for customized fashion outfit composition[EB/OL].(2018-06-21)[2024-03-28]. https://arxiv.org/abs/1806.04845.

[20] ZHAN H J, LIN J, AK K E, et al. A³-FKG:attentive attribute-aware fashion knowledge graph for outfit preference prediction[J]. IEEE Transactions on Multimedia, 2022,24:819-831.

[21] WANG Y Z, LIU L, FU X D, et al. MCCP:multi-modal fashion compatibility and conditional preference model for personalized clothing recommendation[J]. Multimedia Tools and Applications, 2024, 83(4):9621-9645.

[22] DING Y J, MOK P Y, MA Y S, et al. Personalized fashion outfit generation with user coordination preference learning[J]. Information Processing & Management, 2023, 60(5):103434.

[23] GODART F C. Why is style not in fashion? Using the concept of ‘style’ to understand the creative industries[M]//JONES C,MAORET M,eds.Research in the Sociology of Organizations. [S.l.]: Emerald Publishing Limited, 2018:103-128.

[24] DE DIVITIIS L, BECATTINI F, BAECCHI C, et al. Style-based outfit recommendation[C]//2021 International Conference on Content-Based Multimedia Indexing. New York,USA: IEEE, 2021:1-4.

[25] SUN Z, ZHOU Y H, HE H H, et al. SGDiff:a style guided diffusion model for fashion synthesis[EB/OL].(2023-08-15)[2023-11-14]. https://arxiv.org/abs/2308.07605v1.

[26] AN H, LEE K Y, CHOI Y, et al. Conceptual framework of hybrid style in fashion image datasets for machine learning[J]. Fashion and Textiles, 2023, 10(1):18.

[27] BANERJEE D, DHAKAD L, MAHESHWARI H, et al. Recommendation of compatible outfits conditioned on style[EB/OL].(2022-03-30)[2023-11-14]. https://arxiv.org/abs/2203.16161v1.

[28] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[EB/OL].(2015-12-10)[2023-11-14]. https://arxiv.org/abs/1512.03385v1.

[29] FRALEY C. How many clusters? Which clustering method? Answers via model-based cluster analysis[J]. The Computer Journal, 1998, 41(8):578-588.

[30] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[EB/OL].(2014-12-14)[2023-11-06]. https://arxiv.org/abs/1409.3215.

[31] HAN X, WU Z, JIANG Y G, et al. Learning fashion compatibility with bidirectional LSTMs[C]//Proceedings of the 25th ACM International Conference on Multimedia. New York,USA: ACM, 2017:1078-1086.

[32] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6):84-90.

[33] VASILEVA M I, PLUMMER B A, DUSAD K, et al. Learning type-aware embeddings for fashion compatibility[EB/OL].(2018-03-25)[2024-01-22]. https://arxiv.org/abs/1803.09196v2.

[34] WANG X, WU B, ZHONG Y. Outfit compatibility prediction and diagnosis with multi-layered comparison network[C]//Proceedings of the 27th ACM International Conference on Multimedia.Nice, France: ACM, 2019:329-337.

Funding

Foundation items: Shanghai Frontier Science Research Center for Modern Textiles, Donghua University, China

Open Project of Henan Key Laboratory of Intelligent Manufacturing of Mechanical Equipment, Zhengzhou University of Light Industry, China(IM202303)

National Key Research and Development Program of China(2019YFB1706300)