Improved parallel processing function for high-performance large-scale astronomical cross-matching

Qing Zhao; Jizhou Sun; Ce Yu; Jian Xiao; Chenzhou Cui; Xiao Zhang

doi:10.1007/s12209-011-1491-x

Transactions of Tianjin University ›› 2011, Vol. 17 ›› Issue (1) : 62 -67. DOI: 10.1007/s12209-011-1491-x

Article

Improved parallel processing function for high-performance large-scale astronomical cross-matching

Author information +

History +

PDF

Abstract

Astronomical cross-matching is a basic method for aggregating the observational data of different wavelengths. By data aggregation, the properties of astronomical objects can be understood comprehensively. Aiming at decreasing the time consumed on I/O operations, several improved methods are introduced, including a processing flow based on the boundary growing model, which can reduce the database query operations; a concept of the biggest growing block and its determination which can improve the performance of task partition and resolve data-sparse problem; and a fast bitwise algorithm to compute the index numbers of the neighboring blocks, which is a significant efficiency guarantee. Experiments show that the methods can effectively speed up cross-matching on both sparse datasets and high-density datasets.

Keywords

astronomical cross-matching / boundary growing model / HEALPix / task partition / data-sparse problem

Cite this article

Download citation ▾

Qing Zhao, Jizhou Sun, Ce Yu, Jian Xiao, Chenzhou Cui, Xiao Zhang. Improved parallel processing function for high-performance large-scale astronomical cross-matching. Transactions of Tianjin University, 2011, 17(1): 62-67 DOI:10.1007/s12209-011-1491-x

登录浏览全文

4963

注册一个新账户忘记密码

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	Nieto-Santisteban M A, Thakar A R, Szalay A S. Cross-Matching Very Large Datasets[R/OL]. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.129.3240&rep=rep1&type=pdf. 2006.

[2]	Gray J, Szalay A, Budavri T et al. Cross-Matching Multiple Spatial Observations and Dealing with Missing Data[R/OL]. Microsoft Technical Report, MSR-TR-2006-175. Redmond WA. http://research.microsoft.com/pubs/64519/tr-2006-175pdf. 2006.

[3]	Gray J, Szalay A, Fekete, G. Using Table Valued Functions in SQL Server 2005 to Implement a Spatial Data Library[R/OL]. Microsoft Technical Report, MSR-TR-2005-122. Redmond WA. 2005.

[4]	Gray J, Nieto-Santisteban M A, Szalay A S. The Zones Algorithm for Finding Points-near-a-Point or Cross-Matching Spatial Datasets[R/OL]. Microsoft Technical Report, MSR-TR-2006-52. Redmond WA. http://arxiv.org/ftp/cs/papers/0701/0701171pdf. 2006.

[5]	Clive Page: Indexing the Sky[R/OL]. Technical Report, AstroGrid. http://www.star.le.ac.ud/~cgp/ag/skyindex.html. 2002.

[6]	Clive Page: Comments on the XMATCH function in ADQL[R/OL]. Technical Report. AstroGrid. 2004.

[7]	Report on Cross-Matching Catalogues[R/OL]. Technical Report. AstroGrid, http://wiki.astrogrid.org/pub/Astrogrid/DataFederationandDataMining/cross.htm, 2003.

[8]	Spatial Joins and Spatial Indexing Revisted [R/OL]. Technical Report of AstroGrid. AstroGrid, http://wiki.astrogrid.org/bin/view/Astrogrid/SpatialIndexing, 2003.

[9]	Gao D., Zhang Y., Zhao Yongheng. The realization of cross-identification based on huge multi-wavelength catalog data[J]. Astronomical Research and Technology, 2005, 2(3): 186-193.

[10]	Gao Dan. Very Large Astronomical Data Sets Fusion System’s Development and Data Mining Algorithms’ Research[D]. National Astronomical Observatories, Chinese Academy of Sciences, 2008 (in Chinese).

[11]	Zhao Qing, Sun Jizhou, Yu Ce et al. A paralleled largescale astronomical cross-matching function[C]. In: The 9th International Conference on Algorithms and Architectures for Parallel Processing. ICA3PP 2009. Taipei, China, 2009. Vol. 5574 LNCS. 604–614.