Difference-in-Differences research designs for energy and environmental policy evaluation: A critical and practical guide

Abstract

The Difference-in-Differences (DID) method relying on observational data has become a well-established tool for causal inference in the evaluation of energy and environmental policies. Despite its popularity and rapid methodological advances, there is still a lack of practical guidelines for the full DID research design, which may in turn lead to limited credibility and misleading policy implications. This study presents a comprehensive, up-to-date, critical, and practical guide to the DID research design, which is intended to help early-career researchers improve the credibility, transparency, and replicability of policy evaluation studies. This guide offers a step-by-step framework covering real-world questions, clean identification strategies, appropriate controls, proper standard errors, transparent data, robust checks, and insightful analyses of heterogeneity and mechanism. By focusing on research logic and design principles rather than complex methodological details, this study helps researchers and policymakers obtain more credible evidence for policy learning and optimization.

Graphical abstract

Keywords

policy evaluation / Difference-in-Differences / research design / causal inference / guidelines / energy and environment

References

[1] Abadie A, Athey S, Imbens G W, Wooldridge J M, ((2023)). When should you adjust standard errors for clustering?. Quarterly Journal of Economics, 138( 1): 1–35

[2] Abadie A, Diamond A, Hainmueller J, (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105( 490): 493–505

[3] Abadie A, Diamond A, Hainmueller J, (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59( 2): 495–510

[4] Acemoglu D, Wolitzky A, (2011). The economics of labor coercion. Econometrica, 79( 2): 555–600

[5] An C, Zhou P, Ma C, (2025). Regional abatement cost-saving effects of industrial agglomeration. Energy Economics, 2025: 108903

[6] Angrist J D, Pischke J S, (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24( 2): 3–30

[7] Arkhangelsky D, Athey S, Hirshberg D A, Imbens G W, Wager S, (2021). Synthetic difference-in-differences. American Economic Review, 111( 12): 4088–4118

[8] Baker ACallaway BCunningham SGoodman-Bacon ASant’Anna P H (2025). Difference-in-differences designs: A practitioner’s guide. Preprint at arXiv. arXiv: 2503.13323

[9] Beck T, Levine R, Levkov A, (2010). Big bad banks? The winners and losers from bank deregulation in the United States. Journal of Finance, 65( 5): 1637–1667

[10] Bellemare M F, Masaki T, Pepinsky T B, (2017). Lagged explanatory variables and the estimation of causal effect. Journal of Politics, 79( 3): 949–963

[11] Breuer M, DeHaan E D, (2024). Using and interpreting fixed effects models. Journal of Accounting Research, 62( 4): 1183–1226

[12] Brodeur A, Cook N, Heyes A, (2020). Methods matter: P-hacking and publication bias in causal analysis in economics. American Economic Review, 110( 11): 3634–3660

[13] Cai C, (2020). Literature on recent advances in applied micro methods. Journal of Environmental Engineering,

[14] Callaway B (2023). Difference-in-differences for policy evaluation. Handbook of Labor, Human Resources and Population Economics: 1–61

[15] Callaway BDyal DSant’Anna P HTsyawo E S (2025). Beyond parallel trends: An identification-strategy-robust approach to causal inference with panel data. Preprint at arXiv. arXiv: 2511.21977

[16] Callaway B, Li T, (2019). Quantile treatment effects in difference in differences models with panel data. Quantitative Economics, 10( 4): 1579–1618

[17] Callaway B, Sant’Anna P H, (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225( 2): 200–230

[18] Cameron A C, Miller D L, (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50( 2): 317–372

[19] Chen Y, Fan Z, Gu X, Zhou L A, (2020). Arrival of young talent: The send-down movement and rural education in China. American Economic Review, 110( 11): 3393–3430

[20] Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J, (2018). Double/debiased machine learning for treatment and structural parameters. Econometrics Journal, 21( 1): C1–C68

[21] Chu W, Fan L, Zhou P, (2025). Examining the interactions of carbon, electricity, and natural gas markets. Engineering Management, 12( 03): 1–15

[22] Cinelli CFeller AImbens GKennedy EMagliacane SZubizarreta J (2025). Challenges in statistics: A dozen challenges in causality and causal inference. Preprint at arXiv. arXiv: 2508.17099

[23] Cinelli C, Forney A, Pearl J, (2024). A crash course in good and bad controls. Sociological Methods & Research, 53( 3): 1071–1104

[24] Cinelli C, Hazlett C, (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society. Series B, Statistical Methodology, 82( 1): 39–67

[25] De Chaisemartin C, d’Haultfoeuille X, (2018). Fuzzy differences-in-differences. Review of Economic Studies, 85( 2): 999–1028

[26] De Chaisemartin C, d’Haultfoeuille X, (2023). Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: A survey. Econometrics Journal, 26( 3): C1–C30

[27] Dechezleprêtre A, Nachtigall D, Venmans F, (2023). The joint impact of the European Union emissions trading system on carbon emissions and economic performance. Journal of Environmental Economics and Management, 118: 102758

[28] Di Falco S, Veronesi M, Yesuf M, (2011). Does adaptation to climate change provide food security? A micro-perspective from Ethiopia. American Journal of Agricultural Economics, 93( 3): 829–846

[29] Elliott G, Kudrin N, Wüthrich K, (2022). Detecting p-hacking. Econometrica, 90( 2): 887–906

[30] Feng S, Ganguli I, Lee Y, Poe J, Ryan A, Bilinski A, (2025). Difference-in-differences for health policy and practice: A review of modern methods. Statistics in Medicine, 44( 23-24): e70247

[31] Frake J, Gibbs A, Goldfarb B, Hiraiwa T, Starr E, Yamaguchi S, (2025). From perfect to practical: Partial identification methods for causal inference in strategic management research. Strategic Management Journal, 46( 8): 1894–1929

[32] Goldsmith-Pinkham P, Hull P, Kolesár M, (2024). Contamination bias in linear regressions. American Economic Review, 114( 12): 4015–4051

[33] Goodman-Bacon A, (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225( 2): 254–277

[34] Greenstone M, Gayer T, (2009). Quasi-experimental and experimental approaches to environmental economics. Journal of Environmental Economics and Management, 57( 1): 21–44

[35] Heckman J, Pinto R, (2024). Econometric causality: The central role of thought experiments. Journal of Econometrics, 243( 1-2): 105719

[36] Henningsen A, Low G, Wuepper D, Dalhaus T, Storm H, Belay D, Hirsch S, ((2025)). Estimating causal effects with observational data: Guidelines for agricultural and applied economists.. Journal of Agricultural Economics,

[37] Holland P W, (1986). Statistics and causal inference. Journal of the American Statistical Association, 81( 396): 945–960

[38] Imbens G W, (2020). Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics. Journal of Economic Literature, 58( 4): 1129–1179

[39] Imbens G W, (2024). Causal inference in the social sciences. Annual Review of Statistics and Its Application, 11( 1): 123–152

[40] Imbens G W, Rubin D B (2015). Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge: Cambridge University Press

[41] Keane M P, Neal T, (2024). A practical guide to weak instruments. Annual Review of Economics, 16( 1): 185–212

[42] Kédagni D, Mourifié I, (2020). Generalized instrumental inequalities: testing the instrumental variable independence assumption. Biometrika, 107( 3): 661–675

[43] Lal A, Lockhart M, Xu Y, Zu Z, (2024). How much should we trust instrumental variable estimates in political science? Practical advice based on 67 replicated studies. Political Analysis, 32( 4): 521–540

[44] Li Y, Zhao J, ((2025)). The effectiveness of carbon emission trading system: Evidence from China’s regional markets.. Journal of Deve-lopment Economics, 103631

[45] MacKinnon J G, Nielsen M Ø, Webb M D, (2023). Cluster-robust inference: A guide to empirical practice. Journal of Econometrics, 232( 2): 272–299

[46] Naguib C, (2025). P-hacking and significance stars. Economic Journal, 136( 675): 1140–1154

[47] Nunn N, Qian N, (2011). The potato’s contribution to population and urbanization: evidence from a historical experiment. Quarterly Journal of Economics, 126( 2): 593–650

[48] Olden A, Møen J, (2022). The triple difference estimator. Econometrics Journal, 25( 3): 531–553

[49] Ortiz-Villavicencio MSant’Anna P H (2025). Better understanding triple differences estimators. Preprint at arXiv. arXiv: 2505.09942

[50] Oster E, (2019). Unobservable selection and coefficient stability: Theory and evidence. Journal of Business & Economic Statistics, 37( 2): 187–204

[51] Persson T, Tabellini G, (2009). Democratic capital: The nexus of political and economic change. American Economic Journal. Macroeconomics, 1( 2): 88–126

[52] Picchetti P (2025). Breakdown analysis for instrumental variables with binary outcomes. Preprint at arXiv. arXiv: 2507.10242

[53] Rambachan A, Roth J, (2023). A more credible approach to parallel trends. Review of Economic Studies, 90( 5): 2555–2591

[54] Ren Y S, Huynh T L D, Liu P Z, Narayan S, (2024). Is the carbon emission trading scheme conducive to promoting energy transition? Some empirical evidence from China. Energy Economics, 134: 107629

[55] Roth J, Sant’Anna P H, Bilinski A, Poe J, (2023). What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235( 2): 2218–2244

[56] Sant’Anna P H, Zhao J, (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219( 1): 101–122

[57] Sun L, Abraham S, (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225( 2): 175–199

[58] Wang G, Hamad R, White J S, (2024). Advances in difference-in-differences methods for policy evaluation research. Epidemiology. 35( 5): 628–637

[59] Wing C, Yozwiak M, Hollingsworth A, Freedman S, Simon K, (2024). Designing difference-in-difference studies with staggered treatment adoption: key concepts and practical guidelines. Annual Review of Public Health, 45( 1): 485–505

[60] Wooldridge J M, (2025). Two-way fixed effects, the two-way mundlak regression, and difference-in-differences estimators. Empirical Economics, 69( 5): 2545–2587

[61] Yao Y, Du H, Zou H, Zhou P, Antunes C H, Neumann A, Yeh S, (2023). Fifty years of Energy Policy: A bibliometric overview. Energy Policy, 183: 113769

[62] Zha D, Zhang S, Cao Y, (2025). Can extremely high-temperature weather forecast oil prices. Frontiers of Engineering Management, 12( 03): 1–14

RIGHTS & PERMISSIONS

Higher Education Press