Evaluation of Fire Service Command Unit Trainings

Abstract

The lack of routine and training of command units and emergency managers is among the main causes of suboptimal decisions and could lead to serious consequences. To ensure optimal standards of emergency management training, specific and valid evaluation tools are needed—but are lacking. Thus, the present study’s purpose is to validate instruments for the evaluation of tactical and strategic leader trainings, in particular command unit trainings, based on survey data of n = 288 German Command Unit members. Resulting questionnaires were named “FIRE-CU” (Feedback Instrument for Rescue forces Education – Command Unit) and “FIRE-CPX” (Feedback Instrument for Rescue forces Education – Command Post eXercise scale). Results of confirmatory factor analyses show a good fit for the postulated four-dimensional structure of process scales in the FIRE-CU (trainer’s behavior, structure, overextension, group), for the two-dimensional structure of outcome scales in the FIRE-CU (self-rated competence, transfer), and for the one-dimensional structure of the FIRE-CPX. Further, strong evidence is found for reliability as well as for convergent, divergent, and concurrent validity of both the FIRE-CU and FIRE-CPX. Implications for research and practical application are also discussed to enable broad applicability in various educational programs for public security and crisis management.

Keywords

Command post exercise / Command staff / Crisis management / Emergency management / Rescue forces / Vocational training evaluation

References

[1] Alliger GM, Tannenbaum SI, Bennett W, Traver H, Shotland A. A meta-analysis of the relations among training criteria. Personnel Psychology, 1997, 50(2): 341-358.

[2] American Educational Research Association, American Psychological Association, and National Council on Measurement in Education Standards for educational and psychological testing, 2014, Washington, DC: American Educational Research Association

[3] Arthur W, Bennett W, Edens PS, Bell ST. Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 2003, 88(2): 234-245.

[4] Badke-Schaub P, Hofinger G, Lauche K. Human factors, 2012, Berlin: Springer

[5] Baran BE, Scott CW. Organizing ambiguity: A grounded theory of leadership and sensemaking within dangerous contexts. Military Psychology, 2010, 22(sup1): S42-S69.

[6] Berlin JM, Carlström ED. Collaboration exercises – The lack of collaborative benefits. International Journal of Disaster Risk Science, 2014, 5(3): 192-205.

[7] Blanchard PN, Thacker JW. Effective training: Systems, strategies, and practices, 2013 5 Boston: Pearson

[8] Boon A, Raes E, Kyndt E, Dochy F. Team learning beliefs and behaviours in response teams. European Journal of Training and Development, 2013, 37(4): 357-379.

[9] Cannon-Bowers JA, Salas E, Converse S. Castellan NJ. Shared mental models in expert team decision making. Individual and group decision making: Current issues, 2013, Hoboken: Taylor and Francis 221-246.

[10] Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological Assessment, 1995, 7(3): 309-319.

[11] Dixon DP, Weeks M, Boland R, Perelli S. Making sense when it matters most: An exploratory study of leadership in extremis. Journal of Leadership & Organizational Studies, 2017, 24(3): 294-317.

[12] Evers A. The revised Dutch rating system for test quality. International Journal of Testing, 2001, 1(2): 155-182.

[13] Feuerwehr-Dienstvorschrift (FwDV) 2. 2012. Fire service regulations 2: Training of volunteer fire service (Ausbildung der Freiwilligen Feuerwehr). http://www.bbk.bund.de/SharedDocs/Downloads/BBK/DE/FIS/DownloadsRechtundVorschriften/Volltext_Fw_Dv/FwDV_2_Stand_Jan2012.pdf?__blob=publicationFile. Accessed 8 May 2020.

[14] Gagne P, Hancock GR. Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 2006, 41(1): 65-83.

[15] Grohmann A, Kauffeld S. Evaluating training programs: Development and correlates of the questionnaire for professional training evaluation. International Journal of Training and Development, 2013, 17(2): 135-155.

[16] Grunnan T, Fridheim H. Planning and conducting crisis management exercises for decision-making: The do’s and don’ts. EURO Journal on Decision Processes, 2017, 5(1–4): 79-95.

[17] Hagemann V, Kluge A, Greve J. Measuring the effects of team resource management training for the fire service. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2012, 56(1): 2442-2446.

[18] Hagemann V, Kluge A, Ritzmann S. Flexibility under complexity: Work contexts, task profiles and team processes of high responsibility teams. Employee Relations, 2012, 34(3): 322-338.

[19] Harrell Jr, F.E. 2018. Hmisc: Harrell miscellaneous (Version 4.1–1). https://CRAN.R-project.org/package=Hmisc. Accessed 8 May 2020.

[20] Haus M, Adler C, Hagl M, Maragkos M, Duschek S. Stress and stress management in European crisis managers. International Journal of Emergency Services, 2016, 5(1): 66-81.

[21] Heath R. Dealing with the complete crisis – The crisis management shell structure. Safety Science, 1998, 30(1–2): 139-150.

[22] Hofinger G, Heimann R. Handbook staff work (Handbuch Stabsarbeit), 2016, Berlin: Springer

[23] Hu L-T, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 1999, 6(1): 1-55.

[24] Irwing P, Hughes DJ. Irwing P, Booth T, Hughes DJ. Test development. The Wiley handbook of psychometric testing, 2018, Chichester, UK: John Wiley and Sons 3-48.

[25] Jäger R. Construction of a rating scale with smilies as symbolic marks (Konstruktion einer Ratingskala mit Smilies als symbolische Marken). Diagnostica, 2004, 50(1): 31-38.

[26] Kaplan, D., and R. Pruim. 2018. ggformula. https://CRAN.R-project.org/package=ggformula. Accessed 8 May 2020.

[27] Karsten A. Leading an operational-tactical unit: The tasks of the head of a command unit at D level in civil protection (Leiten eines operativ-taktischen Stabes: Die Aufgaben der Leiterin/des Leiters eines operativ-taktischen Stabes der Führungsstufe D im Bevölkerungsschutz). Bevölkerungsschutz, 2012, 3: 16-19.

[28] Kirkpatrick DL. Techniques for evaluating training programs. Training and Development Journal., 1979, 33: 78-92.

[29] Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: The four levels, 2006 3 San Francisco: Berrett-Koehler

[30] Kraiger K, Ford JK, Salas E. Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. The Journal of Applied Psychology, 1993, 78(2): 311-328.

[31] Lamers C. Command units in civil protection: History, analysis and suggestions for optimisation (Stabsarbeit im Bevölkerungsschutz: Historie, Analyse und Vorschläge zur Optimierung), 2016, Edewecht: S + K Verlagsgesellschaft Stumpf + Kossendey mbH

[32] Mohammed S, Ferzandi L, Hamilton K. Metaphor no more: A 15-year review of the team mental model construct. Journal of Management, 2010, 36(4): 876-910.

[33] Nunnally JC. Psychometric theory, 1978 2 New-York: McGraw-Hill

[34] Paechter, M., B. Maier, and D. Macher. 2011. Evaluation of university teaching by means of assessments of subjective competence acquisition (Evaluation universitärer Lehre mittels Einschätzungen des subjektiven Kompetenzerwerbs). Psychologie in Erziehung und Unterricht 58(2): Article 128.

[35] R Core Team. 2018. The R project for statistical computing. http://www.R-project.org/. Accessed 2 May 2020.

[36] Revelle, W. 2018. Psych: Procedures for psychological, psychometric, and personality research. https://CRAN.R-project.org/package=psych. Accessed 2 May 2020.

[37] Rindermann H. Spiel C. Student assessment of courses – Object of research and implications (Die studentische Beurteilung von Lehrveranstaltungen – Forschungsgegenstand und Implikationen). Evaluation of university teaching: Between quality management and self purpose (Evaluation universitärer Lehre: Zwischen Qualitätsmanagement und Selbstzweck), 2001, Münster: Waxmann 61-88.

[38] Rindermann, H., and M. Amelang. 1994. The Heidelberg Inventory for Course Evaluation (HILVE). Instruction manual (Das Heidelberger Inventar zur Lehrveranstaltungs-Evaluation (HILVE). Handanweisung). Heidelberg: Asanger.

[39] Ritzmann S, Hagemann V, Kluge A. The Training Evaluation Inventory (TEI) – Evaluation of training design and measurement of training outcomes for predicting training success. Vocations and Learning, 2014, 7(1): 41-73.

[40] Röseler S, Thölking H, Hagel M, Thielsch MT. Feedback Instrument for Rescue forces Education – Mission exercises (FIRE-E) (Feedback-Instrument zur Rettungskräfte-Entwicklung – Einsatzübungen). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS), 2020

[41] Rosseel Y. Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 2012, 48(2): 1-36.

[42] Salas E, Cannon-Bowers JA. The science of training: A decade of progress. Annual Review of Psychology, 2001, 52: 471-499.

[43] Schaub H. Hofinger G, Heimann R. Command units in the German army (Militärische Stäbe in der Bundeswehr). Handbook staff work (Handbuch Stabsarbeit), 2016, Berlin: Springer 33-38.

[44] Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research, 2003, 8(2): 23-74.

[45] Schönbrodt FD, Perugini M. At what sample size do correlations stabilize?. Journal of Research in Personality, 2013, 47(5): 609-612.

[46] Schulte N, Thielsch MT. Evaluation of firefighter leadership trainings. International Journal of Emergency Services, 2019, 8(1): 34-49.

[47] Scriven, M. 1999. The nature of evaluation part I: Relation to psychology. Practical Assessment, Research & Evaluation 6(11). https://doi.org/10.7275/egax-6010.

[48] Sjoberg M, Wallenius C, Larsson G. Leadership in complex, stressful rescue operations: A quantitative test of a qualitatively developed model. Disaster Prevention and Management: An International Journal, 2011, 20(2): 199-212.

[49] Sommer M, Njå O. Learning amongst Norwegian fire-fighters. Journal of Workplace Learning, 2011, 23(7): 435-455.

[50] Thielsch MT, Busjan JN, Frerichs K. Feedback Instrument for Rescue forces Education – Examinations (FIRE-P) (Feedback-Instrument zur Rettungskräfte-Entwicklung – Prüfungen). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS), 2018

[51] Thielsch MT, Kläpker L, Streppel L. Feedback Instrument for Rescue forces Education – Basic Training (FIRE-B) (Feedback-Instrument zur Rettungskräfte-Entwicklung – Basisausbildung). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS), 2019

[52] Thieme U, Hofinger G. Badke-Schaub P, Hofinger G, Lauche K. Command units and “Standing staffs” in the police: Security through professionalisation (Stabsarbeit und » Ständige Stäbe « bei der Polizei: Sicherheit durch Professionalisierung). Human factors, 2012, Berlin: Springer 275-291.

[53] Thompson B. Exploratory and confirmatory factor analysis: Understanding concepts and applications, 2004, Washington, DC: American Psychological Association

[54] Trizano-Hermosilla, I., and J.M. Alvarado. 2016. Best alternatives to Cronbach’s alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology 7: Article 769.

[55] Useem M, Cook J, Sutton L. Developing leaders for decision making under stress: Wildland firefighters in the south canyon fire and its aftermath. Academy of Management Learning and Education, 2005, 4(4): 461-485.

[56] Uttl B, White CA, Gonzalez DW. Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 2017, 54: 22-42.

[57] Van Buuren S, Groothuis-Oudshoorn K. Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 2011, 45(3): 1-67.

[58] Van den Bossche P, Gijselaers WH, Segers M, Kirschner PA. Social and cognitive factors driving teamwork in collaborative learning environments. Small Group Research, 2006, 37(5): 490-521.

[59] Wickham H. Plyr: The Split-Apply-Combine Strategy for data analysis. Journal of Statistical Software, 2011, 40(1): 1-29.

[60] Wroblewski, A., and A. Leitner. 2009. Between scientific standards and claims to efficiency: Expert interviews in programme evaluation. In Research methods series. Interviewing experts, ed. A. Bogner, B. Littig, and W. Menz, 235–251. Basingstoke: Palgrave Macmillan.

[61] Wybo JL, Kowalski KM. Command centers and emergency management support. Safety Science, 1998, 30(1–2): 131-138.