New development of cognitive diagnosis models

Abstract

Cognitive diagnosis is the judgment of the student’s cognitive ability, is a wide-spread concern in educational science. The cognitive diagnosis model (CDM) is an essential method to realize cognitive diagnosis measurement. This paper presents new research on the cognitive diagnosis model and introduces four individual aspects of probability-based CDM and deep learning-based CDM. These four aspects are higher-order latent trait, polytomous responses, polytomous attributes, and multilevel latent traits. The paper also sorts on the contained ideas, model structures and respective characteristics, and provides direction for developing cognitive diagnosis in the future.

Graphical abstract

Keywords

higher-order latent traits / polytomous responses / polytomous attributes / multilevel latent traits / cognitive diagnosis

References

[1] Leighton J P, Gierl M J. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge: Cambridge University Press, 2007

[2] Tu D B , Cai Y , Dai H Q , Qi S Q . A review on cognitive diagnostic models under modern test theory. Psychological Exploration, 2008, 28( 2): 64– 68

[3] Fischer G H . The linear logistic test model as an instrument in educational research. Acta Psychologica, 1973, 37( 6): 359– 374

[4] Tatsuoka K K . Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 1983, 20( 4): 345– 354

[5] DiBello L V, Stout W F, Roussos L A. Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In: Nichols P D, Chipman S F, Brennan R L, eds. Cognitively Diagnostic Assessment. Hillsdale: Erlbaum, 1995, 361– 389

[6] Hartz S M. A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. University of Illinois at Urbana-Champaign, Dissertation, 2002

[7] Schmid J , Leiman J M . The development of hierarchical factor solutions. Psychometrika, 1957, 22( 1): 53– 61

[8] Rupp A A, Templin J, Henson R A. Diagnostic Measurement: Theory, Methods, and Applications. New York: Guilford Press, 2010

[9] Sheng Y Y , Wikle C K . Bayesian multidimensional IRT models with a hierarchical structure. Educational and Psychological Measurement, 2008, 68( 3): 413– 430

[10] Rijmen F , Jeon M , Von Davier M , Rabe-Hesketh S . A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 2014, 39( 4): 235– 256

[11] Zhan P D , Yu Z H , Li F M , Wang L J . Using a multi-order cognitive diagnosis model to assess scientific literacy. Acta Psychologica Sinica, 2019, 51( 6): 734– 746

[12] Huang H Y , Wang W C , Chen P H , Su C M . Higher-order item response models for hierarchical latent traits. Applied Psychological Measurement, 2013, 37( 8): 619– 637

[13] Organisation for Economic Co-operation. Technical report of the survey of adult skills (PIAAC). Paris: OECD, 2013

[14] Huang H Y , Wang W C . Higher order testlet response models for hierarchical latent traits and testlet-based items. Educational and Psychological Measurement, 2013, 73( 3): 491– 511

[15] Huo Y , De La Torre J , Mun E Y , Kim S Y , Ray A E , Jiao Y , White H R . A hierarchical multi-unidimensional IRT approach for analyzing sparse, multi-group data for integrative data analysis. Psychometrika, 2015, 80( 3): 834– 855

[16] Huang H Y . A multilevel higher order item response theory model for measuring latent growth in longitudinal data. Applied Psychological Measurement, 2015, 39( 5): 362– 372

[17] Zhang X , Wang C , Tao J . Assessing item-level fit for higher order item response theory models. Applied Psychological Measurement, 2018, 42( 8): 644– 659

[18] Fu Z H , Zhang X , Tao J . Gibbs sampling using the data augmentation scheme for higher-order item response models. Physica A: Statistical Mechanics and its Applications, 2020, 541: 123696

[19] Tu D B. Advanced Cognitive Diagnosis. Beijing: Beijing Normal University Publishing House, 2019

[20] Samejima F . Estimation of latent ability using a response pattern of graded scores. Psychometrika, 1969, 34( 1): 1– 97

[21] Masters G N . A rasch model for partial credit scoring. Psychometrika, 1982, 47( 2): 149– 174

[22] Muraki E . A generalized partial credit model: application of an EM algorithm. Applied Psychological Measurement, 1992, 16( 2): 159– 176

[23] Tu D B , Cai Y , Dai H Q , Ding S L . A polytomous cognitive diagnosis model: P-DINA model. Acta Psychologica Sinica, 2010, 42( 10): 1011– 1020

[24] Chen J S , De La Torre J . Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 2018, 9: 1474

[25] Ma W C , De La Torre J . A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 2016, 69( 3): 253– 275

[26] De La Torre J . The generalized DINA model framework. Psychometrika, 2011, 76( 2): 179– 199

[27] Templin J, Henson R, Rupp A, Jang E, Ahmed M. Cognitive diagnosis models for nominal response data. See researchgate.net/profile/Robert-Henson/publication/228894528_Cognitive_diagnosis_models_for_nominal_response_data/links/0a85e5332fadc2ef60000000/Cognitive-diagnosis-models-for-nominal-response-data.pdf  website, 2008

[28] Ma W C . A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology, 2019, 72( 1): 61– 82

[29] Chen J S , De La Torre J . A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 2013, 37( 6): 419– 437

[30] Tjoe H , De La Torre J . Designing cognitively-based proportional reasoning problems as an application of modern psychological measurement models. Journal of Mathematics Education, 2013, 6( 2): 17– 26

[31] Cai Y , Tu D B . Extension of cognitive diagnosis models based on the polytomous attributes framework and their Q-matrices designs. Acta Psychologica Sinica, 2015, 47( 10): 1300– 1308

[32] Zhao S Y, Chang W, Wang L J, Zhan P D. A polytomous extension of reparametrized polytomous attributes DINA. CNKI (in Chinese), 2019

[33] Rost J . Rasch models in latent classes: an integration of two approaches to item analysis. Applied Psychological Measurement, 1990, 14( 3): 271– 282

[34] Cho S J , Cohen A S . A multilevel mixture IRT model with an application to DIF. Journal of Educational and Behavioral Statistics, 2010, 35( 3): 336– 370

[35] Meij A M M D , Kelderman H , Van Der Flier H . Fitting a mixture item response theory model to personality questionnaire data: characterizing latent classes and investigating possibilities for improving prediction. Applied Psychological Measurement, 2008, 32( 8): 611– 631

[36] Tay L , Newman D A , Vermunt J K . Using mixed-measurement item response theory with covariates (MM-IRT-C) to ascertain observed and unobserved measurement equivalence. Organizational Research Methods, 2011, 14( 1): 147– 176

[37] Kim E S , Joo S H , Lee P , Wang Y , Stark S . Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 2016, 23( 6): 870– 887

[38] Wang X , Tan G H , Wang X , Zhang M Q , Luo C . The mixture item response theory models and its application traces. Advances in Psychological Science, 2014, 22( 3): 540– 548

[39] Cheng S, Liu Q, Chen E H, Huang Z, Huang Z Y, Chen Y Y, Ma H P, Hu G P. DIRT: deep learning enhanced item response theory for cognitive diagnosis. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019, 2397-2400

[40] Yao L H , Schwarz R D . A multidimensional partial credit model with associated item and test statistics: an application to mixed-format tests. Applied Psychological Measurement, 2006, 30( 6): 469– 492

[41] Huang Z Y, Liu Q, Chen E H, Zhao H K, Gao M Y, Wei S, Su Y, Hu G P. Question difficulty prediction for READING problems in standard tests. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 2017, 1352−1359

[42] Shu Z , Henson R , Willse J . Using neural network analysis to define methods of DINA model estimation for small sample sizes. Journal of Classification, 2013, 30( 2): 173– 194

[43] Lamb R L , Annetta L , Vallett D B , Sadler T D . Cognitive diagnostic like approaches using neural-network analysis of serious educational videogames. Computers & Education, 2014, 70: 92– 104

[44] Wang F, Liu Q, Chen E H, Huang Z Y, Chen Y Y, Yin Y, Huang Z, Wang S J. Neural cognitive diagnosis for intelligent education systems. In: Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. 2019, 6153−6161

[45] Liu Q , Wu R Z , Chen E H , Xu G D , Su Y , Chen Z G , Hu G P . Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology, 2018, 9( 4): 48

[46] Boyle E A , Hainey T , Connolly T M , Gray G , Earp J , Ott M , Lim T , Ninaus M , Ribeiro C , Pereira J . An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 2016, 94: 178– 192

[47] Andrich D , Hagquist C . Real and artificial differential item functioning. Journal of Educational and Behavioral Statistics, 2012, 37( 3): 387– 416

[48] Yu X F, Cheng Y, Chang H H. Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): a comprehensive review. In: Von Davier M, Lee Y S, eds. Handbook of Diagnostic Classification Models. Cham: Springer, 2019, 307– 331

[49] Liu H Y , Zhang T C , Wu P W , Yu G . A review of knowledge tracking. Journal of East China Normal University: Natural Science, 2019, ( 5): 1– 15

RIGHTS & PERMISSIONS

Higher Education Press