ProbDiffFlow: an efficient learning-free framework for probabilistic single-image optical flow estimation

Abstract

This paper studies optical flow estimation, a critical task in motion analysis with applications in autonomous navigation, action recognition, and film production. Traditional optical flow methods require consecutive frames, which are often unavailable due to limitations in data acquisition or real-world scene disruptions. Thus, single-frame optical flow estimation is emerging in the literature. However, existing single-frame approaches suffer from two major limitations: (1) they rely on labeled training data, making them task-specific, and (2) they produce deterministic predictions, failing to capture motion uncertainty. To overcome these challenges, we propose ProbDiffFlow, a training-free framework that estimates optical flow distributions from a single image. Instead of directly predicting motion, ProbDiffFlow follows an estimation-by-synthesis paradigm: it first generates diverse plausible future frames using a diffusion-based model, then estimates motion from these synthesized samples using a pre-trained optical flow model, and finally aggregates the results into a probabilistic flow distribution. This design eliminates the need for task-specific training while capturing multiple plausible motions. Experiments on both synthetic and real-world datasets demonstrate that ProbDiffFlow achieves superior accuracy, diversity, and efficiency, outperforming existing single-image and two-frame baselines.

Graphical abstract

Keywords

single image optical flow / stable diffusion

References

[1] Beauchemin S S, Barron J L . The computation of optical flow. ACM Computing Surveys (CSUR), 1995, 27( 3): 433–466

[2] Wang T, Li J, Wu H N, Li C, Snoussi H, Wu Y . ResLNet: deep residual LSTM network with longer input for action recognition. Frontiers of Computer Science, 2022, 16( 6): 166334

[3] Shah S T H, Xuezhi X . Traditional and modern strategies for optical flow: an investigation. SN Applied Sciences, 2021, 3( 3): 289

[4] Butler D J, Wulff J, Stanley G B, Black M J. A naturalistic open source movie for optical flow evaluation. In: Proceedings of the 12th European Conference on Computer Vision. 2012, 611−625

[5] Teed Z, Deng J. RAFT: recurrent all-pairs field transforms for optical flow. 2020, arXiv preprint arXiv: 2003.12039

[6] Sun D, Yang X, Liu M Y, Kautz J. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 8934−8943

[7] Morimitsu H, Zhu X, Ji X, Yin X C. Recurrent partial kernel network for efficient optical flow estimation. In: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence. 2024, 4278−4286

[8] Morimitsu H, Zhu X, Cesar R M, Ji X, Yin X C. RAPIDFlow: recurrent adaptable pyramids with iterative decoding for efficient optical flow estimation. In: Proceedings of IEEE International Conference on Robotics and Automation. 2024, 2946−2952

[9] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015, 234−241

[10] Argaw D M, Kim J, Rameau F, Cho J W, Kweon I S. Optical flow estimation from a single motion-blurred image. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 891−900

[11] Horn B K P, Schunck B G . Determining optical flow. Artificial Intelligence, 1981, 17( 1−3): 185–203

[12] Walker J, Gupta A, Hebert M. Dense optical flow prediction from a static image. In: Proceedings of 2015 IEEE International Conference on Computer Vision. 2015, 2443−2451

[13] Aleotti F, Poggi M, Mattoccia S. Learning optical flow from still images. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 15196−15206

[14] Sun D, Roth S, Lewis J P, Black M J. Learning optical flow. In: Proceedings of the 10th European Conference on Computer Vision. 2008, 83−97

[15] Denton E, Fergus R. Stochastic video generation with a learned prior. In: Proceedings of the 35th International Conference on Machine Learning. 2018, 1174−1183

[16] Wang A, Wen S, Gao Y, Li Q, Deng K, Pang C . Toward enhancing room layout estimation by feature pyramid networks. Data Science and Engineering, 2022, 7( 3): 213–224

[17] Wang T, Qiao M, Zhu A, Shan G, Snoussi H . Abnormal event detection via the analysis of multi-frame optical flow information. Frontiers of Computer Science, 2020, 14( 2): 304–313

[18] Xu H, Chen Z, Zhang Y, Geng X, Mi S, Yang Z . Weakly supervised temporal action localization with proxy metric modeling. Frontiers of Computer Science, 2023, 17( 2): 172309

[19] Wang J, Wang K, Zhang Y, Zhang W, Xu X, Lin X. On LLM-enhanced mixed-type data imputation with high-order message passing. 2025, arXiv preprint arXiv: 2501.02191

[20] Wang J, Wang K, Lin X, Zhang W, Zhang Y . Efficient unsupervised community search with pre-trained graph transformer. Proceedings of the VLDB Endowment, 2024, 17( 9): 2227–2240

[21] Ilg E, Çiçek Ö, Galesso S, Klein A, Makansi O, Hutter F, Brox T. Uncertainty estimates and multi-hypotheses networks for optical flow. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 677−693

[22] Yang K, Ding X, Zhang Y, Chen L, Zheng B, Gao Y . Distributed similarity queries in metric spaces. Data Science and Engineering, 2019, 4( 2): 93–108

[23] Yang G, Ramanan D. Volumetric correspondence networks for optical flow. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 72

[24] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 574

[25] Song J, Meng C, Ermon S. Denoising diffusion implicit models. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[26] Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 10674−10685

[27] Radford A, Kim J W, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 8748−8763

[28] Chai Y, Sapp B, Bansal M, Anguelov D. MultiPath: multiple probabilistic anchor trajectory hypotheses for behavior prediction. In: Proceedings of the 3rd Annual Conference on Robot Learning. 2019, 86−99

[29] Ilg E, Saikia T, Keuper M, Brox T. Occlusions, motion and depth boundaries with a generic network for disparity, optical flow or scene flow estimation. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 626−643

[30] Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3354−3361

[31] Mehl L, Schmalfuss J, Jahedi A, Nalivayko Y, Bruhn A. Spring: a high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 4981−4991

[32] Blattmann A, Dockhorn T, Kulal S, Mendelevitch D, Kilian M, Lorenz D, Levi Y, English Z, Voleti V, Letts A, Jampani V, Rombach R. Stable video diffusion: scaling latent video diffusion models to large datasets. 2023, arXiv preprint arXiv: 2311.15127

[33] Fortun D, Bouthemy P, Kervrann C . Optical flow modeling and computation: a survey. Computer Vision and Image Understanding, 2015, 134: 1–21

[34] Baker S, Scharstein D, Lewis J P, Roth S, Black M J, Szeliski R. A database and evaluation methodology for optical flow. In: Proceedings of IEEE 11th International Conference on Computer Vision. 2007, 1−8

[35] Brox T, Malik J . Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33( 3): 500–513

[36] Chen Q, Koltun V. Full flow: optical flow estimation by global optimization over regular grids. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4706−4714

[37] Menze M, Heipke C, Geiger A. Discrete optimization for optical flow. In: Proceedings of the 37th German Conference on Pattern Recognition. 2015, 16−28

[38] Xu J, Ranftl R, Koltun V. Accurate optical flow via direct cost volume processing. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5807−5815

[39] Sui X, Li S, Geng X, Wu Y, Xu X, Liu Y, Goh R, Zhu H. CRAFT: cross-attentional flow transformer for robust optical flow. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 17581−17590

[40] Huang Z, Shi X, Zhang C, Wang Q, Cheung K C, Qin H, Dai J, Li H. FlowFormer: a transformer architecture for optical flow. In: Proceedings of the 17th European Conference on Computer Vision. 2022, 668−685

[41] Lu Y, Wang Q, Ma S, Geng T, Chen Y V, Chen H, Liu D. TransFlow: transformer as flow learner. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 18063−18073

[42] Croitoru F A, Hondru V, Ionescu R T, Shah M . Diffusion models in vision: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45( 9): 10850–10869

[43] Chen S, Sun P, Song Y, Luo P. DiffusionDet: diffusion model for object detection. In: Proceedings of IEEE/CVF International Conference on Computer Vision. 2023, 19773−19786

RIGHTS & PERMISSIONS

Higher Education Press