An efficient parallel algorithm of N-hop neighborhoods on graphs in distributed environment

Abstract

N-hop neighborhoods information is very useful in analytic tasks on large-scale graphs, like finding clique in a social network, recommending friends or advertising links according to one’s interests, predicting links among websites and etc. To get the N-hop neighborhoods information on a large graph, such as a web graph, a twitter social graph, the most straightforward method is to conduct a breadth first search (BFS) on a parallel distributed graph processing framework, such as Pregel and GraphLab. However, due to the massive volume of message transfer, the BFS method results in high communication cost and has low efficiency.

In this work, we propose a key/value based method, namely KVB, which perfectly fits into the prevailing parallel graph processing framework and computes N-hop neighborhoods on a large scale graph efficiently. Unlike the BFS method, our method need not transfer large amount of neighborhoods information, thus, significantly reduces the overhead on both the communication and intermediate results in the distributed framework.We formalize the N-hop neighborhoods query processing as an optimization problem based on a set of quantitative cost metrics of parallel graph processing. Moreover, we propose a solution to efficiently load only the relevant neighborhoods for computation. Specially, we prove the optimal partial neighborhoods load problem is NP-hard and carefully design a heuristic strategy. We have implemented our algorithm on a distributed graph framework- Spark GraphX and validated our solution with extensive experiments over a number of real world and synthetic large graphs on a modest indoor cluster. Experiments show that our solution generally gains an order of magnitude speedup comparing to the state-of-art BFS implementation.

Keywords

N-hop neighborhoods / graph mining / parallel computing / distributed computing

References

[1] Quamar A, Deshpande A, Lin J. NScale: neighborhood-centric largescale graph analytics in the cloud. The VLDB Journal—The International Journal on Very Large Data Bases, 2016, 25(2): 125–150

[2] Fang Y, Cheng R, Luo S, Hu J. Effective community search for large attributed graphs. Proceedings of the VLDB Endowment, 2016, 9(12): 1233–1244

[3] Xu S, Su S, Xiong L, Cheng X, Xiao K. Differentially private frequent subgraph mining. In: Proceedings of the 32nd IEEE International Conference on Data Engineering. 2016, 229–240

[4] Tadimety P R. Six Degrees of Separation. OSPF: A Network Routing Protocol, Apress, Berkeley, 2015, 1–2

[5] Calinescu G. Computing 2-hop neighborhoods in Ad Hoc wireless networks. In: Proceedings of the International Conference on Ad-Hoc Networks and Wireless. 2003, 175–186

[6] Gui J, Zhou K. Flexible adjustments between energy and capacity for topology control in heterogeneous wireless multi-hop networks. Journal of Network and Systems Management, 2016, 24(4): 789–812

[7] Diop M, Pham C, Thiaré O. 2-hop neighborhood information for cover set selection in mission-critical surveillance with wireless image sensor networks. In: Proceedings of IFIP Wireless Days International Conference. 2013, 1–7

[8] Malewicz G, Austern M H, Bik A J, Dehnert J C, Horn I, Leiser N, Czajkowski G. Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. 2010, 135–146

[9] Low Y, Gonzalez J E, Kyrola A, Bickson D, Guestrin C E, Hellerstein J. Graphlab: a new framework for parallel machine learning. 2014, arXiv preprint arXiv:1408.2041

[10] Lu Y, Cheng J, Yan D, Wu H. Large-scale distributed graph computing systems: an experimental evaluation. Proceedings of the VLDB Endowment, 2014, 8(3): 281–292

[11] Liu H, Huang H H, Hu Y. IBFS: concurrent breadth-first search on gpus. In: Proceedings of the 2016 International Conference on Management of Data. 2016, 403–416

[12] Clauset A, Shalizi C R, Newman M E. Power-law distributions in empirical data. SIAM Review, 2009, 51(4): 661–703

[13] Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In: Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST). 2010, 1–10

[14] Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin M J, Shenker S, Stoica I. Resilient distributed datasets: a faulttolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. 2012, 2

[15] Bernaschi M, Carbone G, Mastrostefano E, Vella F. Solutions to the stconnectivity problem using a GPU-based distributed BFS. Journal of Parallel and Distributed Computing, 2015, 76: 145–153

[16] Hair J F, Black W C, Babin B J, Anderson R E, Tatham R L. Multivariate Data Analysis. Pearson Prentice Hall Upper Saddle River, NJ, 2006

[17] Ketchen D J, Shook C L. The application of cluster analysis in strategic management research: an analysis and critique. Strategic Management Journal, 1996, 17(6): 441–458

[18] Akaike H. Information Theory and an Extension of the Maximum Likelihood Principle. Selected Papers of Hirotugu Akaike,Springer, New York, 1998, 199–213

[19] Bhat H, Kumar N. On the derivation of the bayesian information criterion. School of Natural Sciences, University of California, 2010

[20] Linde A. DIC in variable selection. Statistica Neerlandica, 2005, 59(1): 45–56

[21] Vukotic A, Watt N, Abedrabbo T, Fox D, Partner J. Neo4j in Action. Manning Publications Co., 2014

[22] Xin R S, Gonzalez J E, Franklin M J, Stoica I. Graphx: a resilient distributed graph system on spark. In: Proceedings of the International Workshop on Graph Data Management Experiences and Systems. 2013, 1–6

[23] Csardi G. The igraph software package for complex network research. InterJournal Complex Systems, 2006, 1695(5): 1–9

[24] Avery C. Giraph: large-scale graph processing infrastructure on hadoop. Proceedings of the Hadoop Summit. Santa Clara, 2011, 11(3): 5–9

[25] Shang H, Kitsuregawa M. Efficient breadth-first search on large graphs with skewed degree distributions. In: Proceedings of the 16th International Conference on Extending Database Technology. 2013, 311–322

[26] Yan D, Cheng J, Lu Y, Ng W. Blogel: a block-centric framework for distributed computation on real-world graphs. Proceedings of the VLDB Endowment, 2014, 7(14): 1981–1992

[27] Ugander J, Karrer B, Backstrom L, Marlow C. The anatomy of the facebook social graph. 2011, arXiv preprint arXiv:1111.4503

RIGHTS & PERMISSIONS

2018 Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature