Taxi demand prediction is a crucial component of intelligent transportation system research. Compared to region-based demand prediction, origin-destination (OD) demand prediction has a wide range of potential applications, including real-time matching, idle vehicle allocation, ride-sharing services, and dynamic pricing, among others. However, because OD demand involves complex spatiotemporal dependence, research in this area has been limited thus far. In this paper, we first review existing research from four perspectives: topology construction, temporal and spatial feature processing, and other relevant factors. We then elaborate on the advantages and limitations of OD prediction methods based on deep learning architecture theory. Next, we discuss ongoing challenges in OD prediction, such as dynamics, spatiotemporal dependence, semantic differentiation, time window selection, and data sparsity problems, and summarize and compare potential solutions to each challenge. These findings offer valuable insights for model selection in OD demand prediction. Finally, we provide public datasets and open-source code, along with suggestions for future research directions.