Boosting cross-domain and cross-task generalization for text-attributed graphs from structural perspective

Abstract

Graph models based on large language models (LLMs) have recently garnered considerable attention due to its significant success. Although existing methods resort to LLMs to learn unified semantic representations across domains, they disregard the unique structural characteristics of graphs from different domains. To address this problem, in this paper, we boost graph models from structural perspective and propose BooG. The model constructs virtual super nodes to unify structural characteristics of graph data from different domains. Specifically, the super nodes fuse the information of anchor nodes and class labels, where each anchor node captures the information of a node or a graph instance to be classified. Instead of using the raw graph structure, the super nodes, along with virtual edges, establish a standardized aggregation mechanism that fuses rich information from neighborhoods and associated class labels, accommodating graph structural characteristics inherent to different domains. Additionally, we propose a novel pre-training objective based on contrastive learning, which learns more expressive representations for graph data and generalizes effectively to different domains and downstream tasks. Experimental results on various datasets and tasks demonstrate the superior performance of BooG. We provide our code and data here at the website of github.com/cy623/BooG.

Graphical abstract

Keywords

graph learning / graph foundation model / pre-trained graph models

References

[1] Sankar A, Liu Y, Yu J, Shah N. Graph neural networks for friend ranking in large-scale social platforms. In: Proceedings of the Web Conference 2021. 2021, 2535−2546

[2] Leskovec J, Huttenlocher D, Kleinberg J. Predicting positive and negative links in online social networks. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 641−650

[3] Lv Q, Ding M, Liu Q, Chen Y, Feng W, He S, Zhou C, Jiang J, Dong Y, Tang J. Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 1150−1160

[4] Dwivedi V P, Joshi C K, Luu A T, Laurent T, Bengio Y, Bresson X . Benchmarking graph neural networks. The Journal of Machine Learning Research, 2023, 24( 1): 43

[5] Zitnik M, Leskovec J . Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 2017, 33( 14): i190–i198

[6] He X, Deng K, Wang X, Li Y, Zhang Y, Wang M. LightGCN: simplifying and powering graph convolution network for recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020, 639−648

[7] Song X, Huang H, Lian J, Jin H . XGCN: a library for large-scale graph neural network recommendations. Frontiers of Computer Science, 2024, 18( 3): 183343

[8] Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y. Graph attention networks. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[9] Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th International Conference on Learning Representations. 2017

[10] Xu K, Hu W, Leskovec J, Jegelka S. How powerful are graph neural networks? In: Proceedings of the 7th International Conference on Learning Representations. 2019

[11] Zeng Y, Li Z, Chen Z, Ma H . Aspect-level sentiment analysis based on semantic heterogeneous graph convolutional network. Frontiers of Computer Science, 2023, 17( 6): 176340

[12] Luo J, He M, Pan W, Ming Z . BGNN: behavior-aware graph neural network for heterogeneous session-based recommendation. Frontiers of Computer Science, 2023, 17( 5): 175336

[13] Klicpera J, Bojchevski A, Günnemann S. Predict then propagate: graph neural networks meet personalized PageRank. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[14] Hamilton W L, Ying R, Leskovec J. Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 1025−1035

[15] Abu-El-Haija S, Perozzi B, Kapoor A, Alipourfard N, Lerman K, Harutyunyan H, Ver Steeg G, Galstyan A. MixHop: higher-order graph convolutional architectures via sparsified neighborhood mixing. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 21−29

[16] He D, Liang C, Liu H, Wen M, Jiao P, Feng Z. Block modeling-guided graph convolutional neural networks. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 4022−4029

[17] Jin D, Yu Z, Huo C, Wang R, Wang X, He D, Han J. Universal graph convolutional networks. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 815

[18] Gong C H, Cheng Y, Yu J X, Xu C, Shan C H, Luo S Q, Li X . A survey on learning from graphs with heterophily: recent advances and future directions. Frontiers of Computer Science, 2026, 20( 2): 2002314

[19] Zhu Y, Zhuang F, Zhang X, Qi Z, Shi Z, Cao J, He Q . Combat data shift in few-shot learning with knowledge graph. Frontiers of Computer Science, 2023, 17( 1): 171305

[20] Yan C, Ma H, Li Q, Yang F, Li Z . Efficient multi-scale community search method based on spectral graph wavelet. Frontiers of Computer Science, 2023, 17( 5): 175335

[21] Xiao S, Bai T, Cui X, Wu B, Meng X, Wang B . A graph-based contrastive learning framework for medicare insurance fraud detection. Frontiers of Computer Science, 2023, 17( 2): 172341

[22] Wu S, Xiong Y, Weng C . Dynamic depth-width optimization for capsule graph convolutional network. Frontiers of Computer Science, 2023, 17( 6): 176346

[23] Zhang M, He T, Dong M . Meta-path reasoning of knowledge graph for commonsense question answering. Frontiers of Computer Science, 2024, 18( 1): 181303

[24] Liu J, Yu Z, Guo B, Deng C, Fu L, Wang X, Zhou C . EvolveKG: a general framework to learn evolving knowledge graphs. Frontiers of Computer Science, 2024, 18( 3): 183309

[25] Liu Q, Zhang Q, Zhao F, Wang G . Uncertain knowledge graph embedding: an effective method combining multi-relation and multi-path. Frontiers of Computer Science, 2024, 18( 3): 183311

[26] Tang J, Song R, Huang Y, Gao S, Yu Z . Semantic-aware entity alignment for low resource language knowledge graph. Frontiers of Computer Science, 2024, 18( 4): 184319

[27] Wu Z, Gan Y, Xu T, Wang F . Graph-Segmenter: graph transformer with boundary-aware attention for semantic segmentation. Frontiers of Computer Science, 2024, 18( 5): 185327

[28] Wang T, Jin D, Wang R, He D, Huang Y. Powerful graph convolutional networks with adaptive propagation mechanism for homophily and heterophily. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 4210−4218

[29] Liu Y, Zheng Y, Zhang D, Lee V C S, Pan S. Beyond smoothing: unsupervised graph representation learning with edge heterophily discriminating. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 4516−4524

[30] Pandit S, Chau D H, Wang S, Faloutsos C. Netprobe: a fast and scalable system for fraud detection in online auction networks. In: Proceedings of the 16th International Conference on World Wide Web. 2007, 201−210

[31] Zhu J, Rossi R A, Rao A, Mai T, Lipka N, Ahmed N K, Koutra D. Graph neural networks with heterophily. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 11168−11176

[32] Ding Y, Yao Q, Zhao H, Zhang T. DiffMG: differentiable meta graph search for heterogeneous graph neural networks. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 279−288

[33] Oloulade B M, Gao J, Chen J, Lyu T, Al-Sabri R . Graph neural architecture search: a survey. Tsinghua Science and Technology, 2022, 27( 4): 692–708

[34] Zhang P, Yuan Y, Song J, Gu Y, Qu Q, Bai Y . Introducing on-chain graph data to consortium blockchain for commercial transactions. Frontiers of Computer Science, 2024, 18( 2): 182608

[35] Zhang M, Chen Y. Link prediction based on graph neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 5171−5181

[36] Jin W, Liu X, Zhao X, Ma Y, Shah N, Tang J. Automated self-supervised learning for graphs. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[37] Veličković P, Fedus W, Hamilton W L, Liò P, Bengio Y, Hjelm R D. Deep graph infomax. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[38] You Y, Chen T, Sui Y, Chen T, Wang Z, Shen Y. Graph contrastive learning with augmentations. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 488

[39] Hu Z, Dong Y, Wang K, Chang K W, Sun Y. GPT-GNN: generative pre-training of graph neural networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 1857−1867

[40] Qiu J, Chen Q, Dong Y, Zhang J, Yang H, Ding M, Wang K, Tang J. GCC: graph contrastive coding for graph neural network pre-training. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 1150−1160

[41] Hou Z, Liu X, Cen Y, Dong Y, Yang H, Wang C, Tang J. GraphMAE: self-supervised masked graph autoencoders. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022, 594−604

[42] Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G . Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2023, 55( 9): 195

[43] Gong C, Li X, Yu J, Yao C, Tan J, Yu C, Yin D. Prompt tuning for multi-view graph contrastive learning. 2023, arXiv preprint arXiv: 2310.10362

[44] Liu Z, Yu X, Fang Y, Zhang X. GraphPrompt: unifying pre-training and downstream tasks for graph neural networks. In: Proceedings of the ACM Web Conference 2023. 2023, 417−428

[45] Sun X, Cheng H, Li J, Liu B, Guan J. All in one: multi-task prompting for graph neural networks. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 2120−2131

[46] Liu H, Feng J, Kong L, Liang N, Tao D, Chen Y, Zhang M. One for all: towards training one graph model for all classification tasks. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[47] Xia L, Kao B, Huang C. OpenGraph: towards open graph foundation models. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024. 2024, 2365−2379

[48] He Y, Hooi B. UniGraph: learning a cross-domain graph foundation model from natural language. 2024, arXiv preprint arXiv: 2402.13630

[49] Wu F, Souza A H Jr, Zhang T, Fifty C, Yu T, Weinberger K Q. Simplifying graph convolutional networks. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 6861−6871

[50] Suresh S, Budde V, Neville J, Li P, Ma J. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 1541−1551

[51] Yang L, Li M, Liu L, Niu B, Wang C, Cao X, Guo Y. Diverse message passing for attribute with heterophily. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 363

[52] Jiang Y, Ma H, Zhang X, Li Z, Chang L . Incorporating metapath interaction on heterogeneous information network for social recommendation. Frontiers of Computer Science, 2024, 18( 1): 181302

[53] Jiang W, Ning B, Li G, Bai M, Jia X, Wei F . Graph-decomposed k-NN searching algorithm on road network. Frontiers of Computer Science, 2024, 18( 3): 183609

[54] Wu S, Xiong Y, Liang H, Weng C . D²-GCN: a graph convolutional network with dynamic disentanglement for node classification. Frontiers of Computer Science, 2025, 19( 1): 191305

[55] Liu J, Yang C, Lu Z, Chen J, Li Y, Zhang M, Bai T, Fang Y, Sun L, Yu P S, Shi C. Towards graph foundation models: a survey and beyond. 2023, arXiv preprint arXiv: 2310.11829

[56] Li Y, Wang P, Li Z, Yu J X, Li J. ZeroG: investigating cross-dataset zero-shot transferability in graphs. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024, 1725−1735

[57] Zhao J, Mostafa H, Galkin M, Bronstein M, Zhu Z, Tang J. GraphAny: a foundation model for node classification on any graph. 2024, arXiv preprint arXiv: 2405.20445

[58] Shi C, Chen J, Liu J, Yang C . Graph foundation model. Frontiers of Computer Science, 2024, 18( 6): 186355

[59] Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019, 3982−3992

[60] Chen T, Kornblith S, Norouzi M, Hinton G E. A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 1597−1607

[61] He X, Bresson X, Laurent T, Perold A, LeCun Y, Hooi B. Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph representation learning. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[62] Hu W, Fey M, Zitnik M, Dong Y, Ren H, Liu B, Catasta M, Leskovec J. Open graph benchmark: datasets for machine learning on graphs. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1855

[63] Zhu J, Zhou Y, Qian S, He Z, Zhao T, Shah N, Koutra D. Multimodal graph benchmark. 2024, arXiv preprint arXiv: 2406.16321

[64] Pei H, Wei B, Chang K C C, Lei Y, Yang B. Geom-GCN: geometric graph convolutional networks. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[65] Mernyei P, Cangea C. Wiki-CS: a Wikipedia-based benchmark for graph neural networks. 2020, arXiv preprint arXiv: 2007.02901

[66] Zhao H, Liu S, Ma C, Xu H, Fu J, Deng Z H, Kong L, Liu Q. GIMLET: a unified graph-text model for instruction-based molecule zero-shot learning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 257

[67] Kingma D P, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations. 2015

[68] van der Maaten L, Hinton G . Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9( 86): 2579–2605

RIGHTS & PERMISSIONS

Higher Education Press