Background and aims: Reproducible external benchmarks for pneumothorax-related lung ultrasound (LUS) AI are scarce, and binary lung-sliding classification may obscure clinically important signs. We therefore developed a manifest-based external benchmark and used it to test both cross-domain generalisation and task validity.
Methods: We curated 280 clips from 190 publicly accessible LUS source videos and released a reconstruction manifest containing URLs, timestamps, crop coordinates, labels, and probe shape. Labels were normal lung sliding, absent lung sliding, lung point, and lung pulse. A previously published single-site binary classifier was evaluated on this benchmark; challenge-state analysis examined lung point and lung pulse using the predicted probability of absent sliding, P(absent).
Results: The single-site comparator achieved Receiver Operating Characteristic–Area Under the Curve (ROC-AUC) 0.9625 in-domain but 0.7050 on the heterogeneous external benchmark; restricting external evaluation to linear clips still yielded ROC-AUC 0.7212. In challenge-state analysis, mean P(absent) ranked absent (0.504) > lung point (0.313) > normal (0.186) > lung pulse (0.143). Lung pulse differed from absent clips (p = 0.000470) but not from normal clips (p = 0.813), indicating that the binary model treated pulse as normal-like despite absent sliding. Lung point differed from both absent (p = 0.000468) and normal (p = 0.000026), supporting its interpretation as an intermediate ambiguity state rather than a clean binary class.
Conclusion: A manifest-based, multi-source benchmark can support reproducible external evaluation without redistributing source videos. Binary lung-sliding classification is an incomplete proxy for pneumothorax reasoning because it obscures blind-spot and ambiguity states, such as lung pulse and lung point.
| [1] |
Baloescu C, Bailitz J, Cheema B, et al. Artificial intelligence-guided lung ultrasound by nonexperts. JAMA Cardiol. 2025;10(3):245-253.
|
| [2] |
Kossoff J, Duncan S, Acharya J, Davis D. Automated analysis of ultrasound for the diagnosis of pneumothorax: a systematic review. Cureus. 2024;16(11):e72896.
|
| [3] |
Wu D, Smith D, VanBerlo B, et al. Improving the generalizability and performance of an ultrasound deep learning model using limited multicenter data for lung sliding artifact identification. Diagnostics. 2024;14(11):1081.
|
| [4] |
Qiang X, Wang Q, Liu G, et al. Use video comprehension technology to diagnose ultrasound pneumothorax like a doctor would. Front Physiol. 2025;16:1530808.
|
| [5] |
Hliboký M, Magyar J, Bundzel M, et al. Artifact detection in lung ultrasound: an analytical approach. Electronics. 2023;12(7):1551.
|
| [6] |
Chen L, Zhang Z. Bedside ultrasonography for diagnosis of pneumothorax. Quant Imaging Med Surg. 2015;5(4):618-623.
|
| [7] |
Skulec R, Parizek T, David M, Matousek V, Cerny V. Lung point sign in ultrasound diagnostics of pneumothorax: imitations and variants. Emerg Med Int. 2021;2021:6897946.
|
| [8] |
Jaščur M, Bundzel M, Malík M, Dzian A, Ferenčík N, Babič F. Detecting the absence of lung sliding in lung ultrasounds using deep learning. Appl Sci. 2021;11(15):6976.
|
| [9] |
Prager R, Clausdorff Fiedler H, Smith D, Wu D, Arntfield R. Interrater agreement of physicians identifying lung sliding artifact on B-mode and M-mode point of care ultrasound (POCUS). POCUS J. 2025;10(1):92-98.
|
| [10] |
Biasucci DG, Cina A, Sandroni C, Moscato U, Dauri M, Vetrugno L, et al. Influence of intercostal muscles contraction on sonographic evaluation of lung sliding: a physiological study on healthy subjects. J Anesth Analg Crit Care. 2024;4:31.
|
| [11] |
British Medical Ultrasound Society. Best practice statement on recording and storage of point-of-care ultrasound examinations. BMUS; 2021. Accessed March 23, 2026. Available from: https://www.bmus.org/static/uploads/resources/Best_practice_PoCUS_governance_statement.pdf
|
| [12] |
Aziz S, Edmunds CT, Barratt J. Implementation of a point-of-care ultrasound archiving system and governance framework in a UK physician-paramedic staffed helicopter emergency medical service. Scand J Trauma Resusc Emerg Med. 2024;32(1):49.
|
| [13] |
Clausdorff Fiedler H, Prager R, Smith D, et al. Automated real-time detection of lung sliding using artificial intelligence: a prospective diagnostic accuracy study. Chest. 2024;166(2):362-370.
|
| [14] |
VanBerlo B, Wu D, Li B, et al. Accurate assessment of the lung sliding artefact on lung ultrasonography using a deep learning approach. Comput Biol Med. 2022;148:105953.
|
| [15] |
Kim K, Macruz F, Wu D, et al. Point-of-care AI-assisted stepwise ultrasound pneumothorax diagnosis. Phys Med Biol. 2023;68(20):205013.
|
RIGHTS & PERMISSIONS
2026 The Author(s). Clinical and Translational Discovery published by John Wiley & Sons Australia, Ltd on behalf of Shanghai Institute of Clinical Bioinformatics.
PDF
(716KB)
