A cautionary note on using STRUCTURE to detect hybridization in a phylogenetic context

Abstract

Population genetic clustering methods are widely used to detect hybridization events between closely related populations within species, as well as between deeply diverged lineages across phylogenetic time scales. Their strengths and limitations in the latter cases, however, remain poorly explored. This study presents a systematic evaluation of the performance of the most popular population clustering method, STRUCTURE, under a variety of cross-species hybridization scenarios, including hybrid speciation, as well as introgression involving ghost (i.e., extinct or unsampled) lineages or otherwise. Our simulations demonstrate that STRUCTURE performs well in identifying hybrids and their parental donors when admixture happens very recently between sampled extant lineages, but generally fails to detect signals of admixture when hybridization occurs in deep time or when gene flow stems from ghost lineages. We find that symmetrical parental contribution in cases of hybrid speciation will often be revealed as extremely asymmetrical in STRUCTURE, especially when the admixture event occurred a long time ago. Our results suggest that population genetic clustering methods may be inefficient for detecting ancient or ghost admixtures, which may partly explain why ghost introgression has escaped the attention of evolutionary biologists until recently.

Keywords

ancient hybridization / ghost lineages / hybrid speciation / introgression / simulations / STRUCTURE

RIGHTS & PERMISSIONS

2025 Institute of Botany, Chinese Academy of Sciences.