Qualimetric assessment of research and development performance of a higher education institution

Abstract

The study presents an original methodology for assessing the quality of scientific activity of the units of the Military Medical Academy named after S.M. Kirov. As a tool for its solution, we used qualimetric evaluation of all possible areas of scientific activity of departments and criteria that characterize these areas. Qualimetric methods of expert survey and weighting of performance indicators and descriptive statistics were used. Twenty-five experts took part in structuring the scientific performance of the academy. The resulting structure included six main sections, each described by a specific set of attributes. The attributes were both quantitative and qualitative with certain graduations and units of measurement, each of which was assigned a weighting coefficient. The values of the coefficients were expressed in fractions of 1, and the sum was 1. The consistency of the experts proved to be quite high, and the coefficient of concordance was 0.82 (p < 0.001). Based on the comprehensive assessment of each division of the academy, a scale score was calculated, which helped in determining the rating of a division. A sigma (using the mean square deviation) estimate was chosen as a scaling tool, which shows that 68% of all complex estimates of the units being evaluated are expected in the range from the arithmetic mean minus sigma to the arithmetic mean plus sigma. That is, units with the value of a complex indicator that falls within this interval are recognized as units with an average assessment of scientific activity. Approximately 16% of units whose value of the complex indicator will be greater than the calculated interval will make up a group of units with scientific activity above the average, and the same number of units whose value of the complex indicator will be less than the calculated interval will make up a group of units with scientific activity as below the average. In general, following the structuring and weighting coefficients, a calculation complex was developed to determine the scientific status of a unit, which was performed using an Excel spreadsheet and can be used, along with the evaluation of educational and methodological activities, in the general evaluation of units.

Keywords

educational organization / scientific rating / qualimetry / weighting coefficient / concordance coefficient / scientific performance / sigma assessment / expert assessment

References

[1] Zubkova TM, Natochaya EN. Rating assessment of academic staff in a university based on an automated information system. Software and Systems. 2019;(3):525–533. (In Russ.). DOI: 10.15827/0236-235X.127.525-533

[2] Зубкова Т.М., Наточая Е.Н. Оценка рейтинга научно-педагогических работников университета на основе автоматизированной информационной системы // Программные продукты и системы. 2019. № 3. С. 525–533. DOI: 10.15827/0236-235X.127.525-533

[3] Laptev VV, Potemkin MN. Vnutrivuzovskie reitingi kafedr v sfere nauchnoi deyatel’nosti kak mekhanizm aktivizatsii nauchnoi raboty v universitete. Byulleten’ Uchenogo Soveta. 2006;(8(34)):3–20. (In Russ.).

[4] Лаптев В.В., Потемкин М.Н. Внутривузовские рейтинги кафедр в сфере научной деятельности как механизм активизации научной работы в университете // Бюллетень ученого совета. 2006. № 8 (34). С. 3–20.

[5] Sharshov IA. Otsenka effektivnosti nauchno-issledovatel’skoi deyatel’nosti v sovremennom universitete. Psychology-Pedagogical Journal Gaudeamus. 2014;(1(23)):38–44. (In Russ.).

[6] Шаршов И.А. Оценка эффективности научно-исследовательской деятельности в современном университете // Психолого-педагогический журнал Гаудеамус. 2014. № 1 (23). С. 38–44.

[7] Azgaldov GG. Immeasurable or unmeasured. Product quality control. 2021;11:44–47. (In Russ.).

[8] Азгальдов Г.Г. Неизмеримое или неизмерявшееся // Контроль качества продукции. 2021. № 11. С. 44–47.

[9] Azgaldov GG, Kostin AV, Metrology and qualimetry: identification issues. The world of torments. 2021;2:69–73. (In Russ.).

[10] Азгальдов Г.Г., Костин А.В. Метрология и квалиметрия: вопросы идентификации // Мир измирений. 2021. № 2. С. 69–73.

[11] Pankova LA, Petrovsky AM, Schneiderman MV. Organizatsiya ekspertiz i analiz ekspertnoi informatsii. Moscow: Science; 1984. 120 р. (In Russ.).

[12] Панкова Л.А., Петровский А.М., Шнейдерман М.В Организация экспертиз и анализ экспертной информации. Москва: Наука, 1984. 120 с.

[13] Cherepanov VS, Lyubimova OV. Kvalimetricheskii podkhod v obrazovanii. Education and Science Journal. 2009;63(6):36–43. (In Russ.).

[14] Черепанов В.С., Любимова О.В. Квалиметрический подход в образовании // Образование и наука. 2009. Т. 63, № 6. С. 36–43.

[15] Sheshukova TG. Sergeeva NV. Formirovanie sistemy pokazatelei dlya otsenki effektivnosti nauchnoi deyatel’nosti natsional’nykh issledovatel’skikh universitetov. Economic Analysis: Theory and Practice. 2012;(4(259)):53–63. (In Russ.).

[16] Шешукова Т.Г., Сергеева Н.В. Формирование системы показателей для оценки эффективности научной деятельности национальных исследовательских университетов // Экономический анализ: теория и практика. 2012. № 4 (259). С. 53–63.

[17] Ivchenko EV, Ovchinnikov DV, Zubenko AI, etc. Istoricheskii ocherk nauchnogo otdela Voenno-meditsinskoi akademii. Saint Petersburg: VMA; 2016. 152 р. (In Russ.).

[18] Ивченко Е.В., Овчинников Д.В., Зубенко А.И., и др. Исторический очерк научного отдела Военно-медицинской академии. Санкт-Петербург: ВМА, 2016. 152 с.

[19] Ovchinnikov DV. Scientific research of military medicine and training of scientific personnel in its interests (to the 90th anniversary of the department of organization of scientific work and training of scientific and pedagogical personnel of the Military Medical Academy). Izvestia of the Russian Military Medical Academy. 2021;40(3):5–12. (In Russ.). DOI: 10.17816/rmmar76030

[20] Овчинников Д.В. Научные исследования военной медицины и подготовка научных кадров в ее интересах (к 90-летию отдела организации научной работы и подготовки научно-педагогических кадров Военно-медицинской академии) // Известия Российской военно-медицинской академии. 2021 Т. 40, № 3. C. 5–12. DOI: 10.17816/rmmar76030

[21] Ekspertnye otsenki. Beshelev SD, Gurvich FG, editors. Moscow: Nauka; 1973. 157 р. (In Russ.).

[22] Экспертные оценки / под ред. С.Д. Бешелева, Ф.Г. Гурвича. Москва: Наука, 1973. 157 с.

[23] Fadeeva OM. Primenenie kvalimetrii kak teoreticheskoi bazy otsenki kachestva sistemy strategicheskogo upravleniya vysshim uchebnym zavedeniem. Sotsial’naya Politika i Sotsiologiya. 2013; (4-1(96)):24–35. (In Russ.).

[24] Фадеева О.М. Применение квалиметрии как теоретической базы оценки качества системы стратегического управления высшим учебным заведением // Социальная политика и социология. 2013. № 4-1 (96). С. 24–35.

[25] Korneshchuk NG, Rubin GSh, Abramova TV. Kvalimetriya kak instrument otsenki kachestva obrazovaniya. Standarty i Monitoring v Obrazovanii. 2006;(3):16–9. (In Russ.).

[26] Корнещук Н.Г., Рубин Г.Ш., Абрамова Т.В. Квалиметрия как инструмент оценки качества образования // Стандарты и мониторинг в образовании. 2006. № 3. С. 16–19.

[27] Junkerov VI, Grigoriev SG, Rezvantsev MV. Matematiko-statisticheskaya obrabotka dannykh meditsinskikh issledovanii. Saint Petersburg: VMA; 2011. 318 р. (In Russ.).

[28] Юнкеров В.И., Григорьев С.Г., Резванцев М.В. Математико-статистическая обработка данных медицинских исследований. Санкт-Петербург: ВМА, 2011. 318 с.

RIGHTS & PERMISSIONS

Ivchenko E.V., Grigoriev S.G., Ovchinnikov D.V., Latypov I.F.