ARtiVision: bridging expert knowledge and visitor experience through gaze-guided artifact interpretation in AR

Wei ZHANG , Xing LIU , Biying XU , Xinzhuo DENG , Kam-Kwai WONG , Wenjie NING , Wei CHEN

Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109709

PDF (3494KB)
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) :2109709 DOI: 10.1007/s11704-026-60114-x
Image and Graphics
RESEARCH ARTICLE
ARtiVision: bridging expert knowledge and visitor experience through gaze-guided artifact interpretation in AR
Author information +
History +
PDF (3494KB)

Abstract

Museums play a vital role in promoting public art education and cultural appreciation. However, traditional guidance methods often lack interactivity and fail to convey curatorial insights effectively, resulting in fragmented visitor attention and superficial understanding of artifacts. To address these challenges, we present ARtiVision, a gaze-guided augmented reality framework designed to deliver immersive and personalized museum experiences. Based on interviews with 2 curators and a visitor behavior study with 12 museum visitors, we construct a hierarchical artifact appreciation model that bridges expert knowledge with diverse visitor needs, enabling multi-level engagement with cultural exhibits. ARtiVision integrates two core components: an expert-driven annotation system for structuring interpretive content, and an AR application that presents this content interactively based on visitor preferences and attention. This design allows visitors to explore artifacts at varying depths, enhancing autonomy and comprehension. A follow-up study with 12 museum visitors and 2 curators demonstrates that ARtiVision improves engagement duration, enhances knowledge retention, and streamlines curatorial workflows. Our results highlight ARtiVision’s potential as a scalable, user-adaptive solution for enriching museum experiences.

Graphical abstract

Keywords

augmented reality / artifact interpretation / gaze interaction / museum experience

Cite this article

Download citation ▾
Wei ZHANG, Xing LIU, Biying XU, Xinzhuo DENG, Kam-Kwai WONG, Wenjie NING, Wei CHEN. ARtiVision: bridging expert knowledge and visitor experience through gaze-guided artifact interpretation in AR. Front. Comput. Sci., 2027, 21 (9) : 2109709 DOI:10.1007/s11704-026-60114-x

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

McManus P M . Oh, yes, they do: how museum visitors read labels and interact with exhibit texts. Curator: The Museum Journal, 1989, 32( 3): 174–189

[2]

Reitstätter L, Galter K, Bakondi F . Looking to read: how visitors use exhibit labels in the art museum. Visitor Studies, 2022, 25( 2): 127–150

[3]

Mandolesi S, Gambelli D, Naspetti S, Zanoli R . Exploring visitors’ visual behavior using eye-tracking: the case of the “Studiolo Del Duca”. Journal of Imaging, 2022, 8( 1): 8

[4]

Tzortzi K . Movement in museums: mediating between museum intent and visitor experience. Museum Management and Curatorship, 2014, 29( 4): 327–348

[5]

Krause A E, Davidson J W . Investigating the development and reception of an art exhibition on the theme of early modern representations of love. Curator: The Museum Journal, 2021, 64( 4): 693–714

[6]

Ryffel M, Zünd F, Aksoy Y, Marra A, Nitti M, Aydın T O, Sumner B. AR museum: a mobile augmented reality application for interactive painting recoloring. In: Proceedings of the International Conference on Interfaces and Human Computer Interaction 2017 - Part of the Multi Conference on Computer Science and Information Systems. 2017, 54−60

[7]

Gong Z, Wang R, Xia G . Augmented reality (AR) as a tool for engaging museum experience: a case study on Chinese art pieces. Digital, 2022, 2( 1): 33–45

[8]

Cheng A, Ma D, Pan Y, Qian H . Enhancing museum visiting experience: investigating the relationships between augmented reality quality, immersion, and TAM using PLS-SEM. International Journal of Human-Computer Interaction, 2024, 40( 17): 4521–4532

[9]

Zhang W, Zhang J W, Wong K K, Wang Y F, Feng Y C J, Wang L W, Chen W . Computational approaches for traditional Chinese painting: from the “six principles of painting” perspective. Journal of Computer Science and Technology, 2024, 39( 2): 269–285

[10]

Yang S, Hou M, Li S . Three-dimensional point cloud semantic segmentation for cultural heritage: a comprehensive review. Remote Sensing, 2023, 15( 3): 548

[11]

De Bonis M, Nguyen H, Bourdot P. Context-based annotation visualisation in virtual reality: a use case in archaeological data exploration. In: Proceedings of the 21st EuroXR International Conference on Virtual Reality and Mixed Reality. 2024, 95−119

[12]

Li J, Zheng X, Watanabe I, Ochiai Y . A systematic review of digital transformation technologies in museum exhibition. Computers in Human Behavior, 2024, 161: 108407

[13]

López-Martínez A, Carrera Á, Iglesias C A . Empowering museum experiences applying gamification techniques based on linked data and smart objects. Applied Sciences, 2020, 10( 16): 5419

[14]

Zhang W, Wong K K, Xu B, Ren Y, Li Y, Feng Y, Zhu M, Chen W. CultiVerse: towards cross-cultural understanding for paintings with large language model. In: Proceedings of the 33rd ACM International Conference on Multimedia. 2025, 6710−6719

[15]

Fisseler D, Weichert F, Müller G G W, Cammarosano M. Extending philological research with methods of 3D computer graphics applied to analysis of cultural heritage. In: Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage. 2014, 165−172

[16]

Liarokapis F, Voulodimos A, Doulamis N, Doulamis A. Visual Computing for Cultural Heritage. Cham: Springer, 2020

[17]

Villaespesa E, Murphy O . This is not an apple! Benefits and challenges of applying computer vision to museum collections. Museum Management and Curatorship, 2021, 36( 4): 362–383

[18]

Rodríguez-Ortega N. Image processing and computer vision in the field of art history. In: Brown K, ed. The Routledge Companion to Digital Humanities and Art History. New York: Routledge, 2020, 338−357

[19]

Cai H, Wu Q, Corradi T, Hall P. The cross-depiction problem: computer vision algorithms for recognising objects in artwork and in photographs. 2015, arXiv preprint arXiv: 1505.00110

[20]

Lecoutre A, Negrevergne B, Yger F. Recognizing art style automatically in painting with deep learning. In: Proceedings of the 9th Asian Conference on Machine Learning. 2017, 327−342

[21]

Cetinic E, She J . Understanding and creating art with AI: review and outlook. ACM Transactions on Multimedia Computing, Communications, and Applications, 2022, 18( 2): 66

[22]

Tonini F, Dall’Asen N, Vaquero L, Beyan C, Ricci E. AL-GTD: deep active learning for gaze target detection. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 2360−2369

[23]

Hu Z, Chen Y, Zhao Z, Liu J, Ke B, Ni B. Towards artist-like painting agents with multi-granularity semantic alignment. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 10191−10199

[24]

Zhang W, Wong J K, Wang X, Gong Y, Zhu R, Liu K, Yan Z, Tan S, Qu H, Chen S, Chen W . CohortVA: a visual analytic system for interactive exploration of cohorts based on historical data. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 1): 756–766

[25]

Meyer L, Aaen J E, Tranberg A R, Kun P, Freiberger M, Risi S, Løvlie A S. Algorithmic ways of seeing: using object detection to facilitate art exploration. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 29

[26]

Mermet A, Kitamoto A, Suzuki C, Takagishi A. Face detection on pre-modern Japanese artworks using R-CNN and image patching for semi-automatic annotation. In: Proceedings of the 2nd Workshop on Structuring and Understanding of Multimedia HeritAge Contents. 2020, 23−31

[27]

Mzoughi O, Bigand A, Renaud C. Face detection in painting using deep convolutional neural networks. In: Proceedings of the 19th International Conference on Advanced Concepts for Intelligent Vision Systems. 2018, 333−341

[28]

Madhu P, Marquart T, Kosti R, Bell P, Maier A, Christlein V. Understanding compositional structures in art historical images using pose and gaze priors. In: Proceedings of the 16th European Conference on Computer Vision. 2020, 109−125

[29]

Seirafi K, Wiencek F. HoloMuse-a concept for augmented learning in museums. In: Proceedings of the 10th Forum Media Technology and 3rd All Around Audio Symposium. 2017, 53−57

[30]

Ridel B, Reuter P, Laviole J, Mellado N, Couture N, Granier X . The revealing flashlight: interactive spatial augmented reality for detail exploration of cultural heritage artifacts. Journal on Computing and Cultural Heritage (JOCCH), 2014, 7( 2): 6

[31]

Cetinic E, Grgic S. Automated painter recognition based on image feature extraction. In: Proceedings of ELMAR-2013. 2013, 19−22

[32]

Shamir L, Macura T, Orlov N, Eckley D M, Goldberg I G . Impressionism, expressionism, surrealism: automated recognition of painters and schools of art. ACM Transactions on Applied Perception (TAP), 2010, 7( 2): 8

[33]

Agarwal S, Karnick H, Pant N, Patel U. Genre and style based painting classification. In: Proceedings of 2015 IEEE Winter Conference on Applications of Computer Vision. 2015, 588−594

[34]

Feng Y, Chen J, Huang K, Wong J K, Ye H, Zhang W, Zhu R, Luo X, Chen W . iPoet: interactive painting poetry creation with visual multimodal analysis. Journal of Visualization, 2022, 25( 3): 671–685

[35]

Zhang W, Wong K K, Chen Y, Jia A, Wang L, Zhang J W, Cheng L, Qu H, Chen W . ScrollTimes: tracing the provenance of paintings as a window into history. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 6): 2981–2994

[36]

Zhang J, Synave R, Delepoulle S, Cozot R . Reconstructing image composition: computation of leading lines. Journal of Imaging, 2024, 10( 1): 5

[37]

Niebur E . Saliency map. Scholarpedia, 2007, 2( 8): 2675–2703

[38]

Yu Z, Zhang X, Zhao L, Bin Y, Xiao G. Exploring deeper! Segment anything model with depth perception for camouflaged object detection. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 4322−4330

[39]

Zhu L, Zhou J, Liu Y, Hao X, Liu W, Wang X. WeakSAM: segment anything meets weakly-supervised instance-level recognition. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 7947−7956

[40]

Shibasaki F, Fujishiro I. Topological analysis and approximate identification of leading lines in artworks based on discrete Morse theory. In: Proceedings of 2023 IEEE Visualization and Visual Analytics. 2023, 6−10

[41]

Itti L, Koch C, Niebur E . A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20( 11): 1254–1259

[42]

Mochizuki I, Toyoura M, Mao X . Visual attention prediction for images with leading line structure. The Visual Computer, 2018, 34( 6−8): 1031–1041

[43]

Yun Y K, Lin W. SelfReformer: self-refined network with transformer for salient object detection. 2022, arXiv preprint arXiv: 2205.11283

[44]

Debnath S, Roy R, Changder S . A novel approach using deep convolutional neural network to classify the photographs based on leading-line by fine-tuning the pre-trained VGG16 neural network. Multimedia Tools and Applications, 2024, 83( 1): 3189–3214

[45]

Lee J T, Kim H U, Lee C, Kim C S . Photographic composition classification and dominant geometric element detection for outdoor scenes. Journal of Visual Communication and Image Representation, 2018, 55: 91–105

[46]

Zhao K, Han Q, Zhang C B, Xu J, Cheng M M . Deep Hough transform for semantic line detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44( 9): 4793–4806

[47]

Tennent P, Martindale S, Benford S, Darzentas D, Brundell P, Collishaw M . Thresholds: embedding virtual reality in the museum. Journal on Computing and Cultural Heritage (JOCCH), 2020, 13( 2): 12

[48]

Bekele M K . Clouds-based collaborative and multi-modal mixed reality for virtual heritage. Heritage, 2021, 4( 3): 1447–1459

[49]

He Z, Wu L, Li X . When art meets tech: the role of augmented reality in enhancing museum experiences and purchase intentions. Tourism Management, 2018, 68: 127–139

[50]

Giariskanis F, Kritikos Y, Protopapadaki E, Papanastasiou A, Papadopoulou E, Mania K. The augmented museum: a multimodal, game-based, augmented reality narrative for cultural heritage. In: Proceedings of 2022 ACM International Conference on Interactive Media Experiences. 2022, 281−286

[51]

Papagiannakis G, Geronikolakis E, Pateraki M, López-Menchero V M, Tsioumas M, Sylaiou S, Liarokapis F, Grammatikopoulou A, Dimitropoulos K, Grammalidis N, Partarakis N, Margetis G, Drossis G, Vassiliadi M, Chalmers A, Stephanidis C, Magnenat-Thalmann N. Mixed reality, gamified presence, and storytelling for virtual museums. In: Lee N, ed. Encyclopedia of Computer Graphics and Games. Cham: Springer International Publishing, 2018, 1−13

[52]

Bekele M K, Pierdicca R, Frontoni E, Malinverni E S, Gain J . A survey of augmented, virtual, and mixed reality for cultural heritage. Journal on Computing and Cultural Heritage (JOCCH), 2018, 11( 2): 7

[53]

Boboc R G, Băutu E, Gîrbacia F, Popovici N, Popovici D M . Augmented reality in cultural heritage: an overview of the last decade of applications. Applied Sciences, 2022, 12( 19): 9859

[54]

Innocente C, Ulrich L, Moos S, Vezzetti E . A framework study on the use of immersive XR technologies in the cultural heritage domain. Journal of Cultural Heritage, 2023, 62: 268–283

[55]

Quinto Lima S, Buraglia G, Wong K K, Roberts J. Data bias recognition in museum settings: framework development and contributing factors. In: Proceedings of 2025 CHI Conference on Human Factors in Computing Systems. 2025, 229

[56]

Zhao M, Jin L . A re-presentation of the Chinese traditional art by the VR technology in the digital age: taking Mogao Grottoes in Dunhuang as an example. Journal of Physics: Conference Series, 2021, 1961( 1): 012075

[57]

Jin S, Fan M, Kadir A . Immersive Spring Morning in the Han Palace: learning traditional Chinese art via virtual reality and multi-touch tabletop. International Journal of Human-Computer Interaction, 2022, 38( 3): 213–226

[58]

Carrozzino M, Bergamasco M . Beyond virtual museums: experiencing immersive virtual reality in real museums. Journal of Cultural Heritage, 2010, 11( 4): 452–458

[59]

Jin S, Fan M, Wang Y, Liu Q. Reconstructing traditional Chinese paintings with immersive virtual reality. In: Proceedings of Extended Abstracts of 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1−8

[60]

Wang X, Li Y, Liang H N. MagicMap: enhancing indoor navigation experience in VR museums. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 881−891

[61]

Takahira K, Wong K K, Yang L, Xu X, Fujiwara T, Qu H. TangibleNet: synchronous network data storytelling through tangible interactions in augmented reality. In: Proceedings of 2025 CHI Conference on Human Factors in Computing Systems. 2025, 233

[62]

Wong K K, Ye Y L, Tong W, Li H, Takahira K, Bhatta A, Poudyal S, Ng C W W, Qu H, Yang L. LandSAR: visceralizing landslide data for enhanced situational awareness in immersive analytics. 2026, arXiv preprint arXiv: 2604.09241

[63]

Aggarwal R, Singhal A. Augmented reality and its effect on our life. In: Proceedings of the 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence). 2019, 510−515

[64]

Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B . Recent advances in augmented reality. IEEE Computer Graphics and Applications, 2001, 21( 6): 34–47

[65]

Zhong Q, Wang Q, Chen H, Han M, Wang G. Research on virtual restoration display of Chinese paintings based on augmented reality. In: Proceedings of the 4th International Conference on Computer Engineering and Application. 2023, 133−137

[66]

Zhang Y, Wang Y. Augmented reality technology in painting display. In: Proceedings of the 12th International Conference on Design, User Experience, and Usability. 2023, 227−235

[67]

Flanagan J C . The critical incident technique. Psychological Bulletin, 1954, 51( 4): 327–358

[68]

Salvucci D D, Goldberg J H. Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications. 2000, 71−78

[69]

Wooding D S . Eye movements of large populations: Ⅱ. Deriving regions of interest, coverage, and similarity using fixation maps. Behavior Research Methods, Instruments, & Computers, 2002, 34( 4): 518–528

[70]

Goldberg J H, Kotval X P . Computer interface evaluation using eye movements: methods and constructs. International Journal of Industrial Ergonomics, 1999, 24( 6): 631–645

[71]

Blascheck T, Kurzhals K, Raschke M, Burch M, Weiskopf D, Ertl T . Visualization of eye tracking data: a taxonomy and survey. Computer Graphics Forum, 2017, 36( 8): 260–284

[72]

Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, van de Weijer J. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press, 2011

[73]

Rayner K . Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 1998, 124( 3): 372–422

[74]

Lee B, Sedlmair M, Schmalstieg D . Design patterns for situated visualization in augmented reality. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 1): 1324–1335

[75]

Catmull E, Rom R. A class of local interpolating splines. In: Barnhill R E, Riesenfeld R F, eds. Computer Aided Geometric Design. Amsterdam: Elsevier, 1974, 317−326

[76]

Rogers D F, Adams J A. Mathematical Elements for Computer Graphics. 2nd ed. New York: McGraw-Hill Higher Education, 1990

[77]

Brooke J. SUS: a ‘quick and dirty’ usability scale. In: Jordan P W, Thomas B, McClelland I L, Weerdmeester B, eds. Usability Evaluation in Industry. London: CRC Press, 1996, 189−194

[78]

Lewis J R . The system usability scale: past, present, and future. International Journal of Human-Computer Interaction, 2018, 34( 7): 577–590

[79]

Ch’ng E, Cai S, Feng P, Cheng D . Social augmented reality: communicating via cultural heritage. ACM Journal on Computing and Cultural Heritage, 2023, 16( 2): 29

[80]

Borys M, Plechawska-Wójcik M . Eye-tracking metrics in perception and visual attention research. European Journal of Medical Technologies, 2017, 3( 16): 11–23

[81]

Clark N, Dabkowski M, Driscoll P J, Kennedy D, Kloo I, Shi H . Empirical decision rules for improving the uncertainty reporting of small sample system usability scale scores. International Journal of Human-Computer Interaction, 2021, 37( 13): 1191–1206

Rights & permissions

Higher Education Press

PDF (3494KB)

Supplementary files

Highlights

841

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉