Post-hoc Calibration under Domain Shift: Leveraging Unlabeled Data to Estimate Temperatures
Deng-Bao Wang , Chen-Meng Qiu , Xi Cheng , Min-Ling Zhang
Modern deep neural networks (DNNs) have been empirically observed to exhibit poor calibration properties. To address this issue, researchers have proposed post-hoc calibration techniques, such as temperature scaling (TS),which involve refining model outputs using additional validation data. While these methods demonstrate superior calibration performance on in-domain (ID) testing data, their practical application under domain shift scenarios remains limited due to the distribution mismatch between ID validation data and out-of-domain (OOD) testing data. In this study, we explore a realistic scenario where it is possible to access an unlabeled dataset that conforms to the distribution of OOD testing data. Our investigation reveals that this unlabeled OOD set can serve as an anchor to perturb a labeled ID validation set, thereby creating a surrogate dataset that enables temperature estimation for OOD calibration. Experimental results validate the effectiveness of our proposed approach in accurately estimating the optimal temperature specific to OOD testing data, and incorporating this estimated temperature in the TS technique generally reduces expected calibration error relative to source TS and prior baselines. Furthermore, we examine several regularization methods previously proposed to enhance generalization and ID uncertainty calibration. Intriguingly, we observe that these methods may adversely affect OOD uncertainty calibration within the context of post-hoc calibration, highlighting the need for cautious utilization of regularization techniques during training in uncertainty-aware applications.
Machine learning / Uncertainty quantification / Out-of-domain calibration / Deep neural networks / Post-hoc calibration / Temperature scaling
Higher Education Press 2026
/
| 〈 |
|
〉 |
PDF
(2370KB)
