Specificity of multiple performance validity tests in a mixed neurological patient sample from Indonesia

Abstract

Performance validity testing (PVT) is crucial in contemporary neuropsychological assessment. This study aimed to validate several PVTs— - the Reliable Digit Span (RDS), the Longest Digit Forward-1 Trial, the Longest Digit Forward-2 Trials (LDF-2), the Tes Memori Jangka Pendek Indonesia (TMJPI), and the Non-Verbal Medical Symptom Validity Test - in an Indonesian mixed neurological sample. We recruited 141 patients through convenience sampling, divided into three groups: Neurocognitive disorder due to possible neurodegenerative disease (ND; n = 49), post-stroke (n = 42), and mixed etiology (n = 47). Data were collected prospectively. The PVT cut-off scores were adjusted when specificity rates fell below 0.90. Intercorrelations between Mini-Mental State Examination (MMSE) scores, demographic variables, and PVT scores were computed. The cut-off scores were modified for each group due to unacceptable specificity rates, with the most substantial adjustment made for the TMJPI cut-off score in the ND group. Three variables (MMSE, education, and age) were significantly correlated with PVT scores, with the exception of RDS and LDF-2. The PVTs were also significantly intercorrelated. We conclude that the previously developed Indonesian PVTs can be validly applied in neurological patients. However, clinicians should exercise caution when selecting PVTs and consider the demographic backgrounds of patients to minimize false-positive results.

Keywords

Neuropsychological tests / Malingering / Cognitive dysfunction / Diagnosis / Motivation / Sensitivity and specificity

Funding

The first author is an awardee of the “Beasiswa Pendidikan Pascasarjana Luar Negeri 2019” (No: T/993/D3.2/ KD.02.01/2019) from the Directorate General of Higher Education, Research, and Technology, Ministry of Education, Culture, Research, and Technology, Republic of Indonesia.

Conflict of interest

The authors declare that they have no competing interests.

References

[1] Rudman N, Oyebode JR, Jones CA, Bentham P. An investigation into the validity of effort tests in a working age dementia population. Aging Ment Health. 2011; 15(1):47-57. doi: 10.1080/13607863.2010.508770

[2] Bush SS, Ruff RM, Tröster AI, et al. Symptom validity assessment: Practice issues and medical necessity NAN Policy and Planning Committee. Arch Clin Neuropsychol. 2005; 20(4):419-426. doi: 10.1016/j.acn.2005.02.002

[3] Heilbronner RL, Sweet JJ, Morgan JE, Larrabee GJ, Millis SR. American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. Clin Neuropsychol. 2009; 23(7):1093-1129. doi: 10.1080/13854040903155063

[4] Larrabee GJ. Performance validity and symptom validity in neuropsychological assessment. J Int Neuropsychol Soc. 2012; 18(4):625-631. doi: 10.1017/S1355617712000240

[5] Schroeder RW, Martin PK. Validity assessment in clinical settings:How it differs from forensic settings and why it is important. In: Schroeder RW, Martin PK, editors. Validity Assessment in Clinical Neuropsychological Practice: Evaluating and Managing Noncredible Performance. New York, USA: The Guilford Press; 2022. p. 3-10.

[6] Suchy Y. Introduction to special issue: Current trends in empirical examinations of performance and symptom validity. Clin Neuropsychol. 2019; 33(8):1349-1353. doi: 10.1080/13854046.2019.1672334

[7] Schroeder RW, Martin PK. Validity assessment within the memory disorders/dementia clinic. In: Schroeder RW, Martin PK, editors. Validity Assessment in Clinical Neuropsychological Practice: Evaluating and Managing Noncredible Performance. New York: The Guilford Press; 2022. p. 254-279.

[8] Martin PK, Schroeder RW. Base rates of invalid test performance across clinical non-forensic contexts and settings. Arch Clin Neuropsychol. 2020; 35(6):717-725. doi: 10.1093/arclin/acaa017

[9] Martin PK, Schroeder RW, Olsen DH. Performance validity in the dementia clinic: Specificity of validity tests when used individually and in aggregate across levels of cognitive impairment severity. Clin Neuropsychol. 2022; 36(1):165-188. doi: 10.1080/13854046.2020.1778790

[10] Ownsworth T, Fleming JM, Hardwick S. Symptom reporting and associations with compensation status, self-awareness, causal attributions, and emotional wellbeing following traumatic brain injury. Brain Impair. 2006; 7(2):95-106. doi: 10.1375/brim.7.2.95

[11] Nauta IM, Bertens D, van Dam M, et al. Performance validity in outpatients with multiple sclerosis and cognitive complaints. Mult Scler J. 2022; 28(4):642-653. doi: 10.1177/13524585211025780

[12] Loring DW, Goldstein FC, Chen C, et al. False-positive error rates for reliable digit span and auditory verbal learning test performance validity measures in amnestic mild cognitive impairment and early Alzheimer disease. Arch Clin Neuropsychol. 2016; 31(4):313-331. doi: 10.1093/arclin/acw014

[13] Lippa SM. Performance validity testing in neuropsychology: A clinical guide, critical review, and update on a rapidly evolving literature. Clin Neuropsychol. 2018; 32(3):391-421. doi: 10.1080/13854046.2017.1406146

[14] Roor JJ, Peters MJV, Dandachi-FitzGerald B, Ponds RWH. Performance validity test failure in the clinical population : A systematic review and meta-nalysis of prevalence rates. Neuropsychol Rev. 2023;34:299-319. doi: 10.1007/s11065-023-09582-7

[15] Larrabee GJ. Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. Clin Neuropsychol. 2008; 22(4):666-679. doi: 10.1080/13854040701494987

[16] McWhirter L, Ritchie CW, Stone J, Carson A. Performance validity test failure in clinical populations-a systematic review. J Neurol Neurosurg Psychiatry. 2020;91:945-952. doi: 10.1136/jnnp-2020-323776

[17] Greher MR, Wodushek TR. Performance validity testing in neuropsychology: Scientific basis and clinical application-A brief review. J Psychiatr Pract. 2017; 23(2):134-140. doi: 10.1097/PRA.0000000000000218

[18] Wodushek TR, Domen CH. Comparing two models of performance validity assessment in patients with Parkinson’s disease who are candidates for deep brain stimulation surgery. Appl Neuropsychol Adult. 2020; 27(1):9-21. doi: 10.1080/23279095.2018.1473251

[19] Singhal A, Green P, Ashaye K, Shankar K, Gill D. High specificity of the medical symptom validity test in patients with very severe memory impairment. Arch Clin Neuropsychol. 2009; 24(8):721-728. doi: 10.1093/arclin/acp074

[20] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: American Psychiatric Association; 1994.

[21] Dean AC, Victor TL, Boone KB, Philpott LM, Hess RA. Dementia and effort test performance. Clin Neuropsychol. 2009; 23(1):133-152. doi: 10.1080/13854040701819050

[22] Adhiatma W, Hendriks MPH, Halim MS, Kessels RPC. The validation of multiple performance validity tests (PVTs) for Indonesia. Psychol Neurosci. 2025. doi: 10.1037/pne0000340

[23] Dandachi-Fitzgerald B, Ponds RWH, Peters MJV, Merckelbach H. Cognitive underperformance and symptom over-reporting in a mixed psychiatric sample. Clin Neuropsychol. 2011; 25(5):812-828. doi: 10.1080/13854046.2011.583280

[24] Boone KB, Victor TL, Wen J, Razani J, Pontón M. The association between neuropsychological scores and ethnicity, language, and acculturation variables in a large patient population. Arch Clin Neuropsychol. 2007; 22(3):355-365. doi: 10.1016/j.acn.2007.01.010

[25] Evers A, McCormick CM, Hawley LR, et al. Testing practices and attitudes toward tests and testing: An international survey. Int J Test. 2017; 17(2):158-190. doi: 10.1080/15305058.2016.1216434

[26] Green P. Green’s Non-Verbal Medical Symptom Validity Test (NV-MSVT): User’s Manual for Microsoft Windows. Edmonton: Green’s Publishing Inc.; 2008.

[27] Schagen S, Schmand B, De Sterke S, Lindeboom J. Amsterdam short-term memory test: A new procedure for the detection of feigned memory deficits. J Clin Exp Neuropsychol. 1997; 19(1):43-51. doi: 10.1080/01688639708403835

[28] Schmand B, Lindeboom J. Amsterdam Short-Term Memory Test. Manual. Leiden, The Netherlands: PITS; 2005.

[29] Suwartono C, Halim MS, Hidajat LL, Hendriks MPH, Kessels RPC. Development and reliability of the Indonesian wechsler adult intelligence scale-fourth edition (WAIS-IV). Psychology. 2014; 5(14):1611-1619. doi: 10.4236/psych.2014.514171

[30] Jasinski LJ, Berry DTR, Shandera AL, Clark JA. Use of the Wechsler adult intelligence scale digit span subtest for malingering detection: A meta-analytic review. J Clin Exp Neuropsychol. 2011; 33(3):300-314. doi: 10.1080/13803395.2010.516743

[31] Martin PK, Schroeder RW, Odland AP. Neuropsychologists’ validity testing beliefs and practices: A survey of North American professionals. Clin Neuropsychol. 2015; 29(6):741-776. doi: 10.1080/13854046.2015.1087597

[32] Greiffenstein MF, Baker WJ, Gola T. Validation of malingered amnesia measures with a large clinical sample. Psychol Assess. 1994; 6(3):218-224. doi: 10.1037/1040-3590.6.3.218

[33] Babikian T, Boone KB, Lu P, Arnold G. Sensitivity and specificity of various digit span scores in the detection of suspect effort. Clin Neuropsychol. 2006; 20(1):145-159. doi: 10.1080/13854040590947362

[34] Schroeder RW, Twumasi-Ankrah P, Baade LE, Marshall PS. Reliable digit span: A systematic review and cross-validation study. Assessment. 2012; 19(1):21-30. doi: 10.1177/1073191111428764

[35] Alzheimer’s Indonesia. Tentang Alzheimer Indonesia. Alzheimer’s Indonesia; 2023. Availble from: https://alzi.or.id/tentang-alzheimer-indonesia [Last accessed on 2023 Feb 02].

[36] Anam. Lembar Kerja, Pedoman Pemeriksaan, Dan Penilaian Tes Penapisan Mini Mental Status Examination, Clock Drawing Test, Short Blessed Test, MOCA, Dan AD-8 Versi Bahasa Indonesia. Bandung: Kliniki Memori RS Hasan Sadikin, Fakultas Kedokteran Universitas Padjadjaran; 2013.

[37] Buderer NMF. Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity. Acad Emerg Med. 1996; 3(9):895-900. doi: 10.1111/j.1553-2712.1996.tb03538.x

[38] Field A. Discovering Statistics Using SPSS. 3rd ed. London: SAGE Publications Ltd.; 2009.

[39] Cohen J. A power primer. Psychol Bull. 1992; 112(1):155-159. doi: 10.1037//0033-2909.112.1.155

[40] Tomczak M, Tomczak E. The need to report effect size estimates revisited. An overview of some recommended measures of effect size. Trends Sport Sci. 2014; 1(21):19-25.

[41] Kochhann R, Varela JS, de Macedo Lisboa CS, Chaves MLF. The mini mental state examination: Review of cutoff points adjusted for schooling in a large Southern Brazilian sample. Dement Neuropsychol. 2010; 4(1):35-41. doi: 10.1590/S1980-57642010DN40100006

[42] Nijdam-Jones A, Rosenfeld B. Cross-cultural feigning assessment: A systematic review of feigning instruments used with linguistically, ethnically, and culturally diverse samples. Psychol Assess. 2017; 29(11):1321-1336. doi: 10.1037/pas0000438

[43] Sweet JJ, Heilbronner RL, Morgan JE, et al. American Academy of Clinical Neuropsychology (AACN) 2021 consensus statement on validity assessment: Update of the 2009 AACN consensus conference statement on neuropsychological assessment of effort, response bias, and malingering. Clin Neuropsychol. 2021; 35(6):1053-1106. doi: 10.1080/13854046.2021.1896036

[44] Gravetter FJ, Wallnau LB. Statistics for the Behavioral Sciences. 10th ed. Belmont, CA: Wadsworth, Cengage Learning; 2017.

[45] JASP Team. JASP (Version 0.17.1); 2023. Available from: https://jasp-stats.org [Last accessed on 2023 Sep 28].

[46] Henry M, Merten T, Wolf SA, Harth S. Nonverbal Medical Symptom Validity Test performance of elderly healthy adults and clinical neurology patients. J Clin Exp Neuropsychol. 2010; 32(1):19-27. doi: 10.1080/13803390902791653

[47] Sherman EMS, Slick DJ, Iverson GL. Multidimensional malingering criteria for neuropsychological assessment: A 20-year update of the malingered neuropsychological dysfunction criteria. Arch Clin Neuropsychol. 2020; 35(6):735-764. doi: 10.1093/arclin/acaa019

[48] Critchfield E, Soble JR, Marceaux JC, et al. Cognitive impairment does not cause invalid performance: Analyzing performance patterns among cognitively unimpaired, impaired, and noncredible participants across six performance validity tests. Clin Neuropsychol. 2019; 33(6):1083-1101. doi: 10.1080/13854046.2018.1508615

[49] Bodner T, Merten T, Benke T. Performance validity measures in clinical patients with aphasia. J Clin Exp Neuropsychol. 2019; 41(5):476-483. doi: 10.1080/13803395.2019.1579783

[50] Merten T, Bossink L, Schmand B. On the limits of effort testing: Symptom validity tests and severity of neurocognitive symptoms in nonlitigant patients. J Clin Exp Neuropsychol. 2007; 29(3):308-318. doi: 10.1080/13803390600693607

[51] Kiewel NA, Wisdom NM, Bradshaw MR, Pastorek NJ, Strutt AM. A retrospective review of Digit span-related effort indicators in probable Alzheimer’s disease patients. Clin Neuropsychol. 2012; 26(6):965-974. doi: 10.1080/13854046.2012.694478

[52] Zenisek R, Millis SR, Banks SJ, Miller JB. Prevalence of below-criterion Reliable Digit Span scores in a clinical sample of older adults. Arch Clin Neuropsychol. 2016; 31(5):426-433. doi: 10.1093/arclin/acw025

[53] Kanser RJ, Rapport LJ, Hanks RA, Patrick SD. Utility of WAIS-IV Digit Span indices as measures of performance validity in moderate to severe traumatic brain injury. Clin Neuropsychol. 2022; 36(7):1950-1963. doi: 10.1080/13854046.2021.1921277

[54] Webber TA, Soble JR. Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. Clin Neuropsychol. 2018; 32(4):657-670. doi: 10.1080/13854046.2017.1415374

[55] Maiman M, Del Bene VA, MacAllister WS, et al. Reliable digit span: Does it adequately measure suboptimal effort in an adult epilepsy population? Arch Clin Neuropsychol. 2018; 34(2):259-267. doi: 10.1093/arclin/acy027

[56] Ruchinskas R. Wechsler adult intelligence scale-4th edition digit span performance in subjective cognitive complaints, amnestic mild cognitive impairment, and probable dementia of the Alzheimer type. Clin Neuropsychol. 2019; 33(8):1436-1444. doi: 10.1080/13854046.2019.1585574

[57] Van der Werf SP, Prins JB, Jongen PJH, Van der Meer JWM, Bleijenberg G. Abnormal neuropsychological findings are not necessarily a sign of cerebral impairment: A matched comparison between chronic fatigue syndrome and multiple sclerosis. Neuropsychiatry Neuropsychol Behav Neurol. 2000; 13(3):199-203.

[58] Rensen YCM, Vos SH, Aaftink IM, Kessels RPC. Performance validity testing in individuals with alcohol-related cognitive impairments. Psychol Neurosci. 2024; 17(2):131-141. doi: 10.1037/pne0000335

[59] Martín-Aragoneses MT, Mejuto G, del Río D, Fernandes SM, Rodrigues PFS, López-Higes R. Task demands and sentence reading comprehension among healthy older adults: The complementary roles of cognitive reserve and working memory. Brain Sci. 2023; 13(3):428. doi: 10.3390/brainsci13030428

[60] Tyson BT, Baker S, Greenacre M, et al. Differentiating epilepsy from psychogenic nonepileptic seizures using neuropsychological test data. Epilepsy Behav. 2018; 87:39-45. doi: 10.1016/j.yebeh.2018.08.010

[61] Nelson NW, Boone K, Dueck A, Wagener L, Lu P, Grills C. Relationships between eight measures of suspect effort. Clin Neuropsychol. 2003; 17(2):263-272. doi: 10.1076/clin.17.2.263.16511

[62] Priherdityo E. Indonesia “Lupa” Catat Jumlah Penderita Demensia. CNN Indonesia; 2016. Available from: https://www.cnnindonesia.com/gaya-hidup/20160924070505-255-160792/indonesia-lupa-catat-jumlah-penderita-demensia [Last accessed on 2023 Jul 20].

[63] Nieuwenhuis-Mark RE. The death knoll for the MMSE: Has it outlived its purpose? J Geriatr Psychiatry Neurol. 2010; 23(3):151-157. doi: 10.1177/0891988710363714

[64] McGuire C, Crawford S, Evans JJ. Effort testing in dementia assessment: A systematic review. Arch Clin Neuropsychol. 2019;34:114-131. doi: 10.1093/arclin/acy012

[65] Dunning DL, Westgate B, Adlam ALR. A meta-analysis of working memory impairments in survivors of moderate-to-severe traumatic brain injury. Neuropsychology. 2016; 30(7):811-819. doi: 10.1037/neu0000285