About the journal
Browse
Collections
Multimedia collections
Authors & reviewers
ARtiVision: bridging expert knowledge and visitor experience through gaze-guided artifact interpretation in AR
Wei ZHANG , Xing LIU , Biying XU , Xinzhuo DENG , Kam-Kwai WONG , Wenjie NING , Wei CHEN
Front. Comput. Sci. ›› 2027, Vol. 21 ›› Issue (9) : 2109709
Museums play a vital role in promoting public art education and cultural appreciation. However, traditional guidance methods often lack interactivity and fail to convey curatorial insights effectively, resulting in fragmented visitor attention and superficial understanding of artifacts. To address these challenges, we present ARtiVision, a gaze-guided augmented reality framework designed to deliver immersive and personalized museum experiences. Based on interviews with 2 curators and a visitor behavior study with 12 museum visitors, we construct a hierarchical artifact appreciation model that bridges expert knowledge with diverse visitor needs, enabling multi-level engagement with cultural exhibits. ARtiVision integrates two core components: an expert-driven annotation system for structuring interpretive content, and an AR application that presents this content interactively based on visitor preferences and attention. This design allows visitors to explore artifacts at varying depths, enhancing autonomy and comprehension. A follow-up study with 12 museum visitors and 2 curators demonstrates that ARtiVision improves engagement duration, enhances knowledge retention, and streamlines curatorial workflows. Our results highlight ARtiVision’s potential as a scalable, user-adaptive solution for enriching museum experiences.
augmented reality / artifact interpretation / gaze interaction / museum experience
| [1] |
|
| [2] |
|
| [3] |
|
| [4] |
|
| [5] |
|
| [6] |
Ryffel M, Zünd F, Aksoy Y, Marra A, Nitti M, Aydın T O, Sumner B. AR museum: a mobile augmented reality application for interactive painting recoloring. In: Proceedings of the International Conference on Interfaces and Human Computer Interaction 2017 - Part of the Multi Conference on Computer Science and Information Systems. 2017, 54−60 |
| [7] |
|
| [8] |
|
| [9] |
|
| [10] |
|
| [11] |
De Bonis M, Nguyen H, Bourdot P. Context-based annotation visualisation in virtual reality: a use case in archaeological data exploration. In: Proceedings of the 21st EuroXR International Conference on Virtual Reality and Mixed Reality. 2024, 95−119 |
| [12] |
|
| [13] |
|
| [14] |
Zhang W, Wong K K, Xu B, Ren Y, Li Y, Feng Y, Zhu M, Chen W. CultiVerse: towards cross-cultural understanding for paintings with large language model. In: Proceedings of the 33rd ACM International Conference on Multimedia. 2025, 6710−6719 |
| [15] |
Fisseler D, Weichert F, Müller G G W, Cammarosano M. Extending philological research with methods of 3D computer graphics applied to analysis of cultural heritage. In: Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage. 2014, 165−172 |
| [16] |
Liarokapis F, Voulodimos A, Doulamis N, Doulamis A. Visual Computing for Cultural Heritage. Cham: Springer, 2020 |
| [17] |
|
| [18] |
Rodríguez-Ortega N. Image processing and computer vision in the field of art history. In: Brown K, ed. The Routledge Companion to Digital Humanities and Art History. New York: Routledge, 2020, 338−357 |
| [19] |
Cai H, Wu Q, Corradi T, Hall P. The cross-depiction problem: computer vision algorithms for recognising objects in artwork and in photographs. 2015, arXiv preprint arXiv: 1505.00110 |
| [20] |
Lecoutre A, Negrevergne B, Yger F. Recognizing art style automatically in painting with deep learning. In: Proceedings of the 9th Asian Conference on Machine Learning. 2017, 327−342 |
| [21] |
|
| [22] |
Tonini F, Dall’Asen N, Vaquero L, Beyan C, Ricci E. AL-GTD: deep active learning for gaze target detection. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 2360−2369 |
| [23] |
Hu Z, Chen Y, Zhao Z, Liu J, Ke B, Ni B. Towards artist-like painting agents with multi-granularity semantic alignment. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 10191−10199 |
| [24] |
|
| [25] |
Meyer L, Aaen J E, Tranberg A R, Kun P, Freiberger M, Risi S, Løvlie A S. Algorithmic ways of seeing: using object detection to facilitate art exploration. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 29 |
| [26] |
Mermet A, Kitamoto A, Suzuki C, Takagishi A. Face detection on pre-modern Japanese artworks using R-CNN and image patching for semi-automatic annotation. In: Proceedings of the 2nd Workshop on Structuring and Understanding of Multimedia HeritAge Contents. 2020, 23−31 |
| [27] |
Mzoughi O, Bigand A, Renaud C. Face detection in painting using deep convolutional neural networks. In: Proceedings of the 19th International Conference on Advanced Concepts for Intelligent Vision Systems. 2018, 333−341 |
| [28] |
Madhu P, Marquart T, Kosti R, Bell P, Maier A, Christlein V. Understanding compositional structures in art historical images using pose and gaze priors. In: Proceedings of the 16th European Conference on Computer Vision. 2020, 109−125 |
| [29] |
Seirafi K, Wiencek F. HoloMuse-a concept for augmented learning in museums. In: Proceedings of the 10th Forum Media Technology and 3rd All Around Audio Symposium. 2017, 53−57 |
| [30] |
|
| [31] |
Cetinic E, Grgic S. Automated painter recognition based on image feature extraction. In: Proceedings of ELMAR-2013. 2013, 19−22 |
| [32] |
|
| [33] |
Agarwal S, Karnick H, Pant N, Patel U. Genre and style based painting classification. In: Proceedings of 2015 IEEE Winter Conference on Applications of Computer Vision. 2015, 588−594 |
| [34] |
|
| [35] |
|
| [36] |
|
| [37] |
|
| [38] |
Yu Z, Zhang X, Zhao L, Bin Y, Xiao G. Exploring deeper! Segment anything model with depth perception for camouflaged object detection. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 4322−4330 |
| [39] |
Zhu L, Zhou J, Liu Y, Hao X, Liu W, Wang X. WeakSAM: segment anything meets weakly-supervised instance-level recognition. In: Proceedings of the 32nd ACM International Conference on Multimedia. 2024, 7947−7956 |
| [40] |
Shibasaki F, Fujishiro I. Topological analysis and approximate identification of leading lines in artworks based on discrete Morse theory. In: Proceedings of 2023 IEEE Visualization and Visual Analytics. 2023, 6−10 |
| [41] |
|
| [42] |
|
| [43] |
Yun Y K, Lin W. SelfReformer: self-refined network with transformer for salient object detection. 2022, arXiv preprint arXiv: 2205.11283 |
| [44] |
|
| [45] |
|
| [46] |
|
| [47] |
|
| [48] |
|
| [49] |
|
| [50] |
Giariskanis F, Kritikos Y, Protopapadaki E, Papanastasiou A, Papadopoulou E, Mania K. The augmented museum: a multimodal, game-based, augmented reality narrative for cultural heritage. In: Proceedings of 2022 ACM International Conference on Interactive Media Experiences. 2022, 281−286 |
| [51] |
Papagiannakis G, Geronikolakis E, Pateraki M, López-Menchero V M, Tsioumas M, Sylaiou S, Liarokapis F, Grammatikopoulou A, Dimitropoulos K, Grammalidis N, Partarakis N, Margetis G, Drossis G, Vassiliadi M, Chalmers A, Stephanidis C, Magnenat-Thalmann N. Mixed reality, gamified presence, and storytelling for virtual museums. In: Lee N, ed. Encyclopedia of Computer Graphics and Games. Cham: Springer International Publishing, 2018, 1−13 |
| [52] |
|
| [53] |
|
| [54] |
|
| [55] |
Quinto Lima S, Buraglia G, Wong K K, Roberts J. Data bias recognition in museum settings: framework development and contributing factors. In: Proceedings of 2025 CHI Conference on Human Factors in Computing Systems. 2025, 229 |
| [56] |
|
| [57] |
|
| [58] |
|
| [59] |
Jin S, Fan M, Wang Y, Liu Q. Reconstructing traditional Chinese paintings with immersive virtual reality. In: Proceedings of Extended Abstracts of 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1−8 |
| [60] |
Wang X, Li Y, Liang H N. MagicMap: enhancing indoor navigation experience in VR museums. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 881−891 |
| [61] |
Takahira K, Wong K K, Yang L, Xu X, Fujiwara T, Qu H. TangibleNet: synchronous network data storytelling through tangible interactions in augmented reality. In: Proceedings of 2025 CHI Conference on Human Factors in Computing Systems. 2025, 233 |
| [62] |
Wong K K, Ye Y L, Tong W, Li H, Takahira K, Bhatta A, Poudyal S, Ng C W W, Qu H, Yang L. LandSAR: visceralizing landslide data for enhanced situational awareness in immersive analytics. 2026, arXiv preprint arXiv: 2604.09241 |
| [63] |
Aggarwal R, Singhal A. Augmented reality and its effect on our life. In: Proceedings of the 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence). 2019, 510−515 |
| [64] |
|
| [65] |
Zhong Q, Wang Q, Chen H, Han M, Wang G. Research on virtual restoration display of Chinese paintings based on augmented reality. In: Proceedings of the 4th International Conference on Computer Engineering and Application. 2023, 133−137 |
| [66] |
Zhang Y, Wang Y. Augmented reality technology in painting display. In: Proceedings of the 12th International Conference on Design, User Experience, and Usability. 2023, 227−235 |
| [67] |
|
| [68] |
Salvucci D D, Goldberg J H. Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications. 2000, 71−78 |
| [69] |
|
| [70] |
|
| [71] |
|
| [72] |
Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, van de Weijer J. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press, 2011 |
| [73] |
|
| [74] |
|
| [75] |
Catmull E, Rom R. A class of local interpolating splines. In: Barnhill R E, Riesenfeld R F, eds. Computer Aided Geometric Design. Amsterdam: Elsevier, 1974, 317−326 |
| [76] |
Rogers D F, Adams J A. Mathematical Elements for Computer Graphics. 2nd ed. New York: McGraw-Hill Higher Education, 1990 |
| [77] |
Brooke J. SUS: a ‘quick and dirty’ usability scale. In: Jordan P W, Thomas B, McClelland I L, Weerdmeester B, eds. Usability Evaluation in Industry. London: CRC Press, 1996, 189−194 |
| [78] |
|
| [79] |
|
| [80] |
|
| [81] |
|
Higher Education Press
/
| 〈 |
|
〉 |