VIS+AI: integrating visualization with artificial intelligence for efficient data analysis

Abstract

Visualization and artificial intelligence (AI) are well-applied approaches to data analysis. On one hand, visualization can facilitate humans in data understanding through intuitive visual representation and interactive exploration. On the other hand, AI is able to learn from data and implement bulky tasks for humans. In complex data analysis scenarios, like epidemic traceability and city planning, humans need to understand large-scale data and make decisions, which requires complementing the strengths of both visualization and AI. Existing studies have introduced AI-assisted visualization as AI4VIS and visualization-assisted AI as VIS4AI. However, how can AI and visualization complement each other and be integrated into data analysis processes are still missing. In this paper, we define three integration levels of visualization and AI. The highest integration level is described as the framework of VIS+AI, which allows AI to learn human intelligence from interactions and communicate with humans through visual interfaces. We also summarize future directions of VIS+AI to inspire related studies.

Graphical abstract

Keywords

visualization / artificial intelligence / data analysis / knowledge generation

Xumeng Wang is a lecturer in the College of Computer Science, Nankai University, China. She earned the PhD in computer science and technology from Zhejiang University, China in 2021. Her research interests include visual analytics and privacy preservation

Ziliang Wu received the BS degree in computer science from Zhejiang University, China in 2020. He is currently working toward the PhD in the College of Computer Science and Technology, Zhejiang University, China. His research interests include visual analysis powered by AI, visualization recommendation

Wenqi Huang is the leader of Artificial Intelligence and Intelligent Software Team in R&D Center, Digital Grid Research Institute, China Southern Power Grid, China. She holds BS (2010) and PhD (2015) degrees from the Department of Information Science and Electronic Engineering, Zhejiang University, China. Her research interests span artificial intelligence, data mining and blockchain application technology in the field of power industry

Yating Wei is a PhD student in the State Key Lab of CAD&CG, Zhejiang University, China. She earned the BS degree in software engineering from Central South University, China in 2017. Her research interests include visual analytics and perceptual consistency

Zhaosong Huang is with Huawei Cloud, China. He received his PhD in the College of Computer Science and Technology from Zhejiang University, China. He was a joint PhD student at Arizona State University, USA in 2017−2018. His research interests include visualization and visual analysis of urban data

Mingliang Xu is a professor and director of the School of Computer and Artificial Intelligence, Zhengzhou University, China, and the director of the Engineering Research Center of Ministry of Education on Intelligent Swarm Systems, China. He received his PhD degree from the State Key Lab of CAD&CG from Zhejiang University, China. He was awarded as the National Science Foundation for Excellent Young Scholars. His current research interests include computer graphics, multimedia and artificial intelligence. He has published 100+ papers in ACM/IEEE Transactions and full papers on top-tier conferences, such as CVPR and ACM Multimedia

Wei Chen is a professor of the State Key Lab of CAD&CG, Zhejiang University, China. His research interests include visualization and visual analysis. He has published more than 70 IEEE/ACM Transactions and IEEE VIS papers. He actively served as guest or associate editors of the ACM Transactions on Intelligent System and Technology, the IEEE Computer Graphics and Applications, and Journal of Visualization

References

[1] Kwon B C, Choi M J, Kim J T, Choi E, Kim Y B, Kwon S, Sun J, Choo J . RetainVis: visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 299–309

[2] Zhang Y, Chanana K, Dunne C . IDMVis: temporal event sequence visualization for type 1 diabetes treatment decision support. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 512–522

[3] Wu Y, Chen Z, Sun G, Xie X, Cao N, Liu S, Cui W . StreamExplorer: a multi-stage system for visually exploring events in social streams. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 10): 2758–2772

[4] Chen W, Xia J, Wang X, Wang Y, Chen J, Chang L . RelationLines: visual reasoning of egocentric relations from heterogeneous urban data. ACM Transactions on Intelligent Systems and Technology, 2019, 10( 1): 2

[5] Leite R A, Gschwandtner T, Miksch S, Kriglstein S, Pohl M, Gstrein E, Kuntner J . EVA: visual analytics to identify fraudulent events. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 1): 330–339

[6] Wang X-M, Zhang T-Y, Ma Y-X, Xia J, Chen W . A survey of visual analytic pipelines. Journal of Computer Science and Technology, 2016, 31( 4): 787–804

[7] Xia J, Ye F, Chen W, Wang Y, Chen W, Ma Y, Tung A K H . LDSScanner: exploratory analysis of low-dimensional structures in high-dimensional datasets. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 1): 236–245

[8] Giovannangeli L, Bourqui R, Giot R, Auber D . Toward automatic comparison of visualization techniques: application to graph visualization. Visual Informatics, 2020, 4( 2): 86–98

[9] Riveiro M, Lebram M, Elmer M . Anomaly detection for road traffic: a visual analytics framework. IEEE Transactions on Intelligent Transportation Systems, 2017, 18( 8): 2260–2270

[10] Dodge S, Karam L. A study and comparison of human and deep learning recognition performance under visual distortions. In: Proceedings of the 26th International Conference on Computer Communication and Networks. 2017, 1–7

[11] Tang T, Li R, Wu X, Liu S, Knittel J, Koch S, Ertl T, Yu L, Ren P, Wu Y . PlotThread: creating expressive storyline visualizations using reinforcement learning. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 2): 294–303

[12] Wang Q, Chen Z, Wang Y, Qu H . A survey on ML4VIS: applying machine learning advances to data visualization. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 12): 5134–5153

[13] Sperrle F, El-Assady M, Guo G, Borgo R, Chau D H, Endert A, Keim D . A survey of human-centered evaluations in human-centered machine learning. Computer Graphics Forum, 2021, 40( 3): 543–568

[14] Yuan J, Chen C, Yang W, Liu M, Xia J, Liu S . A survey of visual analytics techniques for machine learning. Computational Visual Media, 2021, 7( 1): 3–36

[15] Domova V, Vrotsou K. A model for types and levels of automation in visual analytics: a survey, a taxonomy, and examples. IEEE Transactions on Visualization and Computer Graphics, DOI: 10.1109/TVCG.2022.3163765, 2022

[16] Wang Q, Chen Z, Wang Y, Qu H. Applying machine learning advances to data visualization: a survey on ml4vis. 2020, arXiv preprint arXiv: 2012.00467

[17] Sacha D, Stoffel A, Stoffel F, Kwon B C, Ellis G, Keim D A . Knowledge generation model for visual analytics. IEEE Transactions on Visualization and Computer Graphics, 2014, 20( 12): 1604–1613

[18] Keim D, Andrienko G, Fekete J D, Görg C, Kohlhammer J, Melançon G. Visual analytics: definition, process, and challenges. In: Kerren A, Stasko J T, Fekete J D, North C, eds. Information Visualization. Berlin: Springer, 2008, 154–175

[19] Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. In: Bederson B B, Shneiderman B, eds. The Craft of Information Visualization. Amsterdam: Elsevier, 2003, 364–371

[20] Alemzadeh S, Niemann U, Ittermann T, Völzke H, Schneider D, Spiliopoulou M, Bühler K, Preim B . Visual analysis of missing values in longitudinal cohort study data. Computer Graphics Forum, 2020, 39( 1): 63–75

[21] Arbesser C, Spechtenhauser F, Mühlbacher T, Piringer H . Visplause: visual data quality assessment of many time series using plausibility checks. IEEE Transactions on Visualization and Computer Graphics, 2017, 23( 1): 641–650

[22] Bäuerle A, Neumann H, Ropinski T . Classifier-guided visual correction of noisy labels for image classification tasks. Computer Graphics Forum, 2020, 39( 3): 195–205

[23] Willett W, Ginosar S, Steinitz A, Hartmann B, Agrawala M . Identifying redundancy and exposing provenance in crowdsourced data analysis. IEEE Transactions on Visualization and Computer Graphics, 2013, 19( 12): 2198–2206

[24] Krause J, Perer A, Bertini E . INFUSE: interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics, 2014, 20( 12): 1614–1623

[25] Chegini M, Bernard J, Berger P, Sourin A, Andrews K, Schreck T . Interactive labelling of a multivariate dataset for supervised machine learning using linked visualisations, clustering, and active learning. Visual Informatics, 2019, 3( 1): 9–17

[26] Yang F, Harrison L T, Rensink R A, Franconeri S L, Chang R . Correlation judgment and visualization features: a comparative study. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 3): 1474–1488

[27] Wongsuphasawat K, Smilkov D, Wexler J, Wilson J, Mané D, Fritz D, Krishnan D, Viégas F B, Wattenberg M . Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 1): 1–12

[28] Wang Z J, Turko R, Shaikh O, Park H, Das N, Hohman F, Kahng M, Polo Chau D H . CNN explainer: learning convolutional neural networks with interactive visualization. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 2): 1396–1406

[29] Smilkov D, Carter S, Sculley D, Viégas F B, Wattenberg M. Direct-manipulation visualization of deep networks. 2017, arXiv preprint arXiv: 1708.03788

[30] Krause J, Dasgupta A, Swartz J, Aphinyanaphongs Y, Bertini E. A workflow for visual diagnostics of binary classifiers using instance-level explanations. In: Proceedings of 2017 IEEE Conference on Visual Analytics Science and Technology. 2017, 162–172

[31] Kahng M, Andrews P Y, Kalro A, Chau D H . ActiVis: visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 1): 88–97

[32] Zhang J, Wang Y, Molino P, Li L, Ebert D S . Manifold: a model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 364–373

[33] Strobelt H, Gehrmann S, Behrisch M, Perer A, Pfister H, Rush A M . Seq2seq-Vis: a visual debugging tool for sequence-to-sequence models. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 353–363

[34] Wexler J, Pushkarna M, Bolukbasi T, Wattenberg M, Viégas F, Wilson J . The what-if tool: interactive probing of machine learning models. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 56–65

[35] Wang X, Chen W, Xia J, Chen Z, Xu D, Wu X, Xu M, Schreck T. ConceptExplorer: visual analysis of concept drifts in multi-source time-series data. In: Proceedings of 2020 IEEE Conference on Visual Analytics Science and Technology. 2020, 1–11

[36] Ahn Y, Lin Y-R . FairSight: visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 1086–1095

[37] Ma Y, Xie T, Li J, Maciejewski R . Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 1075–1085

[38] Gogolou A, Tsandilas T, Palpanas T, Bezerianos A . Comparing similarity perception in time series visualizations. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 523–533

[39] Kieffer S, Dwyer T, Marriott K, Wybrow M . HOLA: human-like orthogonal network layout. IEEE Transactions on Visualization and Computer Graphics, 2016, 22( 1): 349–358

[40] Pohl M, Schmitt M, Diehl S. Comparing the readability of graph layouts using eyetracking and task-oriented analysis. In: Proceedings of the 5th Eurographics Conference on Computational Aesthetics in Graphics, Visualization and Imaging. 2009, 49–56

[41] Xu K, Rooney C, Passmore P, Ham D H, Nguyen P H . A user study on curved edges in graph visualization. IEEE Transactions on Visualization and Computer Graphics, 2012, 18( 12): 2449–2456

[42] Etemadpour R, Motta R, de Souza Paiva J G, Minghim R, De Oliveira M C F, Linsen L . Perception-based evaluation of projection methods for multidimensional data visualization. IEEE Transactions on Visualization and Computer Graphics, 2015, 21( 1): 81–94

[43] Fu X, Wang Y, Dong H, Cui W, Zhang H. Visualization assessment: a machine learning approach. In: Proceedings of 2019 IEEE Visualization Conference. 2019, 126–130

[44] Ding R, Han S, Xu Y, Zhang H, Zhang D. QuickInsights: quick and automatic discovery of insights from multi-dimensional data. In: Proceedings of 2019 International Conference on Management of Data. 2019, 317–332

[45] Zhao Y, Ge L, Xie H, Bai G, Zhang Z, Wei Q, Lin Y, Liu Y, Zhou F . ASTF: visual abstractions of time-varying patterns in radio signals. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 1): 214–224

[46] Wang H, Ondřej J, O’Sullivan C . Trending paths: a new semantic-level metric for comparing simulated and real crowd data. IEEE Transactions on Visualization and Computer Graphics, 2017, 23( 5): 1454–1464

[47] Haleem H, Wang Y, Puri A, Wadhwa S, Qu H . Evaluating the readability of force directed graph layouts: a deep learning approach. IEEE Computer Graphics and Applications, 2019, 39( 4): 40–53

[48] Fujiwara T, Chou J K, Shilpika S, Xu P, Ren L, Ma K-L . An incremental dimensionality reduction method for visualizing streaming multidimensional data. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 418–428

[49] Kim Y, Wongsuphasawat K, Hullman J, Heer J. GraphScape: a model for automated reasoning about visualization similarity and sequencing. In: Proceedings of 2017 CHI Conference on Human Factors in Computing Systems. 2017, 2628–2638

[50] Wang Y, Jin Z, Wang Q, Cui W, Ma T, Qu H . DeepDrawing: a deep learning approach to graph drawing. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 676–686

[51] Chen C, Wang C, Bai X, Zhang P, Li C . GenerativeMap: visualization and exploration of dynamic density maps via generative learning model. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 216–226

[52] Han J, Wang C . TSR-TVD: temporal super-resolution for time-varying data analysis and visualization. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 205–215

[53] Blascheck T, Kurzhals K, Raschke M, Burch M, Weiskopf D, Ertl T . Visualization of eye tracking data: a taxonomy and survey. Computer Graphics Forum, 2017, 36( 8): 260–284

[54] Müller N H, Liebold B, Pietschmann D, Ohler P, Rosenthal P. Hierarchy visualization designs and their impact on perception and problem solving strategies. In: Proceedings of the 10th International Conference on Advances in Computer-Human Interactions. 2017, 93–101

[55] Bryan C, Mishra A, Shidara H, Ma K-L . Analyzing gaze behavior for text-embellished narrative visualizations under different task scenarios. Visual Informatics, 2020, 4( 3): 41–50

[56] Blascheck T, John M, Kurzhals K, Koch S, Ertl T . VA²: a visual analytics approach for evaluating visual analytics applications. IEEE Transactions on Visualization and Computer Graphics, 2016, 22( 1): 61–70

[57] Pandey A V, Krause J, Felix C, Boy J, Bertini E. Towards understanding human similarity perception in the analysis of large sets of scatter plots. In: Proceedings of 2016 CHI Conference on Human Factors in Computing Systems, 2016, 3659–3669

[58] Jo J, Seo J. Disentangled representation of data distributions in scatterplots. In: Proceedings of 2019 IEEE Visualization Conference. 2019, 136–140

[59] Fan C, Hauser H . Fast and accurate CNN-based brushing in scatterplots. Computer Graphics Forum, 2018, 37( 3): 111–120

[60] Brehmer M, Munzner T . A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 2013, 19( 12): 2376–2385

[61] Siegel N, Horvitz Z, Levin R, Divvala S, Farhadi A. FigureSeer: parsing result-figures in research papers. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 664–680

[62] Al-Zaidy R A, Choudhury S R, Giles C L. Automatic summary generation for scientific data charts. In: Proceedings of 2016 AAAI Workshop. 2016, 658–663

[63] Harper J, Agrawala M . Converting basic D3 charts into reusable style templates. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 3): 1274–1286

[64] Hoque E, Agrawala M . Searching the visual style and structure of D3 visualizations. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 1236–1245

[65] Bryan C, Ma K-L, Woodring J . Temporal summary images: an approach to narrative visualization via interactive annotation generation and placement. IEEE Transactions on Visualization and Computer Graphics, 2017, 23( 1): 511–520

[66] Liu C, Xie L, Han Y, Wei D, Yuan X. AutoCaption: an approach to generate natural language description from visualization automatically. In: Proceedings of 2020 IEEE Pacific Visualization Symposium. 2020, 191–195

[67] Obeid J, Hoque E. Chart-to-text: generating natural language descriptions for charts by adapting the transformer model. 2020, arXiv preprint arXiv: 2010.09142

[68] Micallef L, Palmas G, Oulasvirta A, Weinkauf T . Towards perceptual optimization of the visual design of scatterplots. IEEE Transactions on Visualization and Computer Graphics, 2017, 23( 6): 1588–1599

[69] Ragan E D, Endert A, Sanyal J, Chen J . Characterizing provenance in visualization and data analysis: an organizational framework of provenance types and purposes. IEEE Transactions on Visualization and Computer Graphics, 2016, 22( 1): 31–40

[70] Ottley A, Garnett R, Wan R . Follow the clicks: learning and anticipating mouse interactions during exploratory data analysis. Computer Graphics Forum, 2019, 38( 3): 41–52

[71] Li Y, Qi Y, Shi Y, Chen Q, Cao N, Chen S . Diverse interaction recommendation for public users exploring multi-view visualization using deep learning. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 1): 95–105

[72] Torrey L, Shavlik J. Transfer learning. In: Olivas E S, Guerrero J D M, Martinez-Sober M, Magdalena-Benedito J R, López A J S, eds. Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. Hershey: IGI Global, 2010, 242–264

[73] Van Den Elzen S, Van Wijk J J. BaobabView: interactive construction and analysis of decision trees. In: Proceedings of 2011 IEEE conference on Visual Analytics Science and Technology. 2011, 151–160

[74] Chen C, Yuan J, Lu Y, Liu Y, Su H, Yuan S, Liu S . OoDAnalyzer: interactive analysis of out-of-distribution samples. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 7): 3335–3349

[75] Cavallo M, Demiralp Ç . Clustrophile 2: guided visual clustering analysis. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 267–276

[76] Pister A, Buono P, Fekete J-D, Plaisant C, Valdivia P . Integrating prior knowledge in mixed-initiative social network clustering. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 2): 1775–1785

[77] Yang W, Wang X, Lu J, Dou W, Liu S . Interactive steering of hierarchical clustering. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 10): 3953–3967

[78] Sedlmair M, Aupetit M . Data-driven evaluation of visual quality measures. Computer Graphics Forum, 2015, 34( 3): 201–210

[79] Ma Y, Tung A K H, Wang W, Gao X, Pan Z, Chen W . ScatterNet: a deep subjective similarity model for visual analysis of scatterplots. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 3): 1562–1576

[80] Abbas M M, Aupetit M, Sedlmair M, Bensmail H . ClustMe: a visual quality measure for ranking monochrome scatterplots based on cluster patterns. Computer Graphics Forum, 2019, 38( 3): 225–236

[81] Luo Y, Qin X, Tang N, Li G. DeepEye: towards automatic data visualization. In: Proceedings of the 34th IEEE International Conference on Data Engineering. 2018, 101–112

[82] Yu Y, Kruyff D, Jiao J, Becker T, Behrisch M . PSEUDo: interactive pattern search in multivariate time series with locality-sensitive hashing and relevance feedback. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 1): 33–42

[83] Wang Y, Feng K, Chu X, Zhang J, Fu C-W, Sedlmair M, Yu X, Chen B . A perception-driven approach to supervised dimensionality reduction for visualization. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 5): 1828–1840

[84] Gramazio C C, Huang J, Laidlaw D H . An analysis of automated visual analysis classification: interactive visualization task inference of cancer genomics domain experts. IEEE Transactions on Visualization and Computer Graphics, 2018, 24( 8): 2270–2283

[85] Gotz D, Wen Z. Behavior-driven visualization recommendation. In: Proceedings of the 14th International Conference on Intelligent User Interfaces. 2009, 315–324

[86] Milo T, Somech A. Next-step suggestions for modern interactive data analysis platforms. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018, 576–585

[87] Chen Z, Zeng W, Yang Z, Yu L, Fu C-W, Qu H . LassoNet: deep lasso-selection of 3D point clouds. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 195–204

[88] Brown E T, Yarlagadda S, Cook K A, Chang R, Endert A. ModelSpace: visualizing the trails of data models in visual analytics systems. In: Proceedings of 2018 IEEE Workshop on Machine Learning from User Interaction for Visualization and Analytics. 2018, 1–11

[89] Kahou S E, Michalski V, Atkinson A, Kádár Á, Trischler A, Bengio Y. FigureQA: an annotated figure dataset for visual reasoning. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[90] Kafle K, Price B, Cohen S, Kanan C. DVQA: understanding data visualizations via question answering. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 5648–5656

[91] Zhang Y, Pasupat P, Liang P. Macro grammars and holistic triggering for efficient semantic parsing. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 1214–1223

[92] Kim D H, Hoque E, Agrawala M. Answering questions about charts and generating visual explanations. In: Proceedings of 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1–13

[93] Martinez-Maldonado R, Echeverria V, Fernandez Nieto G, Buckingham Shum S. From data to insights: a layered storytelling approach for multimodal learning analytics. In: Proceedings of 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1–15

[94] Lai C, Lin Z, Jiang R, Han Y, Liu C, Yuan X. Automatic annotation synchronizing with textual description for visualization. In: Proceedings of 2020 CHI Conference on Human Factors in Computing Systems. 2020, 1–13

[95] Srinivasan A, Drucker S M, Endert A, Stasko J . Augmenting visualizations with interactive data facts to facilitate interpretation and communication. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 1): 672–681

[96] Wang Y, Sun Z, Zhang H, Cui W, Xu K, Ma X, Zhang D . DataShot: automatic generation of fact sheets from tabular data. IEEE Transactions on Visualization and Computer Graphics, 2020, 26( 1): 895–905

[97] Zhao Y, Shi J, Liu J, Zhao J, Zhou F, Zhang W, Chen K, Zhao X, Zhu C, Chen W . Evaluating effects of background stories on graph perception. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 12): 4839–4854

[98] Xu K, Ottley A, Walchshofer C, Streit M, Chang R, Wenskovitch J . Survey on the analysis of user interactions and visualization provenance. Computer Graphics Forum, 2020, 39( 3): 757–783

[99] Gotz D, Zhou M X . Characterizing users’ visual analytic activity for insight provenance. Information Visualization, 2009, 8( 1): 42–55

[100] Xu G, Li H, Ren H, Yang K, Deng R H . Data security issues in deep learning: attacks, countermeasures, and opportunities. IEEE Communications Magazine, 2019, 57( 11): 116–122

[101] Pitropakis N, Panaousis E, Giannetsos T, Anastasiadis E, Loukas G . A taxonomy and survey of attacks against machine learning. Computer Science Review, 2019, 34: 100199

[102] Bhagoji A N, Chakraborty S, Mittal P, Calo S. Analyzing federated learning through an adversarial lens. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 634–643

[103] Yang Q, Liu Y, Chen T, Tong Y . Federated machine learning: concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10( 2): 12

[104] Liu Y, Zhang W, Wang J. Source-free domain adaptation for semantic segmentation. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 1215–1224

[105] Abadi M, Chu A, Goodfellow I J, McMahan H B, Mironov I, Talwar K, Zhang L. Deep learning with differential privacy. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 308–318

[106] Pan Y-H . On visual knowledge. Frontiers of Information Technology & Electronic Engineering, 2019, 20( 8): 1021–1025

[107] Wongsuphasawat K, Qu Z, Moritz D, Chang R, Ouk F, Anand A, Mackinlay J, Howe B, Heer J. Voyager 2: augmenting visual analysis with partial view specifications. In: Proceedings of 2017 CHI Conference on Human Factors in Computing Systems. 2017, 2648–2659

[108] Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J . Vega-lite: a grammar of interactive graphics. IEEE Transactions on Visualization and Computer Graphics, 2017, 23( 1): 341–350

[109] Koonchanok R, Baser P, Sikharam A, Raveendranath N K, Reda K. Data prophecy: exploring the effects of belief elicitation in visual analytics. In: Proceedings of 2021 CHI Conference on Human Factors in Computing Systems. 2021, 18

[110] Zhang P, Li C, Wang C . VisCode: embedding information in visualization images using encoder-decoder network. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 2): 326–336

[111] Fu J, Zhu B, Cui W, Ge S, Wang Y, Zhang H, Huang H, Tang Y, Zhang D, Ma X . Chartem: reviving chart images with data embedding. IEEE Transactions on Visualization and Computer Graphics, 2021, 27( 2): 337–346

[112] Jiang A, Nacenta M A, Terzic K, Ye J. Visualization as intermediate representations (VLAIR) for human activity recognition. In: Proceedings of the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare. 2020, 201–210

[113] Shneiderman B . Human-centered artificial intelligence: reliable, safe & trustworthy. International Journal of Human–Computer Interaction, 2020, 36( 6): 495–504

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 62202244, 62132017, and 62036010).

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit creativecommons.org/licenses/by/4.0/

RIGHTS & PERMISSIONS

2023 The Author(s) 2023. This article is published with open access at link.springer.com and journal.hep.com.cn