Computerized Adaptive Test
13369835
Development of an abbreviated form of the Penn Line Orientation Test using large samples and computerized adaptive test simulation.
Visuospatial processing is a commonly assessed neurocognitive domain with deficits linked to dysfunction in right posterior regions of the brain. With the growth of large scale clinical research studies, there is an increased need for efficient and scalable assessments of neurocognition, including visuospatial processing. The purpose of the current study was to use a novel method that combines item response theory (IRT) and computerized adaptive testing (CAT) approaches to create an abbreviated form of the computerized Penn Line Orientation Test (PLOT) The 24 item PLOT was administered to 8,498 youths (aged 8 21 years) as part of the Philadelphia Neurodevelopmental Cohort study and, by Web based data collection, in an independent sample of 4,593 adults from Great Britain as part of a TV documentary. IRT based CAT simulations were used to select the best PLOT items for an abbreviated form by performing separate simulations in each group and choosing only items that were selected as useful (i.e. high item discrimination and in the appropriate difficulty range) in at least 1 of the simulations. Fifteen items were chosen for the final, short form of the PLOT, indicating substantial agreement among the models in how they evaluated each item's usefulness. Moreover, this abbreviated version performed comparably to the full version in tests of sensitivity to age and sex effects. This abbreviated version of the PLOT cuts administration time by 50% without detectable loss of information, which points to its feasibility for large scale clinical and genomic studies.
Psychological assessment
Tyler M Moore J Cobb Scott Steven P Reise Allison M Port Chad Jackson Kosha Ruparel Adam P Savitt Raquel E Gur Ruben C Gur
42047171
Development of a Computerized Adaptive Test for Assessing Activities of Daily Living in Outpatients With Stroke
Background An efficient, reliable, and valid measure for assessing activities of daily living (ADL) function is useful to improve the efficiency of patient management and outcome measurement. Objective The purpose of this study was to construct a computerized adaptive testing (CAT) system for measuring ADL function in outpatients with stroke. Design Two cohort studies were conducted at 6 hospitals in Taiwan. Methods A candidate item bank (44 items) was developed, and 643 outpatients were interviewed. An item response theory model was fitted to the data and estimated the item parameters (eg, difficulty and discrimination) for developing the ADL CAT. Another sample of 51 outpatients was interviewed to examine the concurrent validity and efficiency of the CAT. The ADL CAT, as the outcome measure, and the Barthel index (BI) and Frenchay Activities index (FAI) were administered on the second group of participants. Results Ten items did not satisfy the model's expectations and were deleted. Thirty four items were included in the final item bank. Two stopping rules (ie, reliability coefficient .9 and maximum test length of 7 items) were set for the CAT. The participants' ADL scores had an average reliability of .93. The CAT scores were highly associated with those of the full 34 items (Pearson r=.98) The scores of the CAT were closely correlated with those of the combined BI and FAI (r=.82) The time required to complete the CAT was about one fifth of the time used to administer both the BI and FAI. Limitations The participants were outpatients living in the community. Further studies are needed to cross validate the results. Conclusions The results demonstrated that the ADL CAT is quick to administer, reliable, and valid in outpatients with stroke.
Physical Therapy
I P Hsueh Jyun-Hong Chen Chun-Hou Wang Wen-Hsuan Hou Ching-Lin Hsieh
8511768
The EORTC emotional functioning computerized adaptive test: phases I III of a cross cultural item bank development
The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Group is currently developing computerized adaptive testing measures for the Quality of Life Questionnaire Core 30 (QLQ C30) scales. The work presented here describes the development of an EORTC item bank for emotional functioning (EF) which is one of the core domains of the QLQ C30.
Psycho oncology
Eva M Gamper Mogens Groenvold Morten Aa Petersen Teresa Young A Costantini Neil K Aaronson Johannes M Giesinger Verena Meraner Georg Kemmler Bernhard Holzner
24706252
Psychometric evaluation of the EORTC computerized adaptive test (CAT) fatigue item pool
PurposeFatigue is one of the most common symptoms associated with cancer and its treatment. To obtain a more precise and flexible measure of fatigue, the EORTC Quality of Life Group has developed a computerized adaptive test (CAT) measure of fatigue. This is part of an ongoing project developing a CAT version of the widely used EORTC QLQ C30 questionnaire.MethodsBased on the literature search and evaluations by experts and patients, 41 new fatigue items were developed (in addition to the three QLQ C30 fatigue items) Psychometric properties of the items, including evaluations of dimensionality, fit to item response theory (IRT) model, and differential item functioning (DIF) were assessed in an international sample of cancer patients.ResultsResponses were obtained from 1,321 cancer patients coming from eight countries. Factor analysis showed that 37 of the items could be included in a unidimensional model (RMSEA 0.098, TLI 0.995, CFI 0.920) Of the 37 items, two were deleted because of poor fit to the IRT model forming the basis for the CAT, and one because of DIF between cancer sites.ConclusionsWe have established a 34 item fatigue bank allowing for more precise and flexible measurement of fatigue, while still being backward compatible with the QLQ C30 fatigue scale.
Quality of Life Research
Morten Aa Petersen Johannes M Giesinger Bernhard Holzner Juan Ignacio Arraras Thierry Conroy Eva M Gamper Madeleine T King Irma M Verdonck-de Leeuw Teresa Young Mogens Groenvold
25591149
Items and dimensions for the construction of a multidimensional computerized adaptive test to measure fatigue in patients with rheumatoid arthritis.
OBJECTIVES Development of an item pool to construct a future computerized adaptive test (CAT) for fatigue in rheumatoid arthritis (RA) The item pool was based on the patients' perspective and examined for face and content validity previously. This study assessed the fit of the items with seven predefined dimensions and examined the item pool's dimensionality structure in statistical terms. STUDY DESIGN AND SETTING A total of 551 patients with RA participated in this study. Several steps were conducted to come from an explorative item pool to a psychometrically sound item bank. The item response theory (IRT) analysis using the generalized partial credit model was conducted for each of the seven predefined dimensions. Poorly fitting items were removed. Finally, the best possible multidimensional IRT (MIRT) model for the data was identified. RESULTS In IRT analysis, 49 items showed insufficient item characteristics. Items with a discriminative ability below 0.60 and/or model misfit effect sizes greater than 0.10 were removed. Factor analysis on the 196 remaining items revealed three dimensions, namely severity, impact, and variability of fatigue. The dimensions were further confirmed in MIRT model analysis. CONCLUSION This study provided an initially calibrated item bank and showed which dimensions and items can be used for the development of a multidimensional CAT for fatigue in RA.
Journal of clinical epidemiology
Stephanie Nikolaus Christina Bode Erik Taal Johanna C M Oostveen Cees A W Glas Mart Afj van de Laar
182947183
Development of a computerized adaptive screening tool for overall psychopathology "p"
A substantial body of work supports the existence of a general psychopathology factor "p" Psychometrically, this is important because it implies that there is a psychological phenomenon (overall psychopathology) that can be measured and potentially used in clinical research or treatment. The present study aimed to construct, calibrate, and begin to validate a computerized adaptive (CAT) screener for "p" In a large community sample (N 4544; age 11 21) we modeled 114 clinical items using a bifactor multidimensional item response theory (MIRT) model and constructed a fully functional (and public) CAT for assessing "p" called the Overall mental illness (OMI) screener. In a random, non overlapping sample (N 1019) with extended phenotyping (neuroimaging) from the same community cohort, adaptive versions of the OMI screener (10 20 and 40 item) were simulated and compared to the full 114 item test in their ability to predict demographic characteristics, common mental disorders, and brain parameters. The OMI screener performed almost as well as the full test, despite being only a small fraction of the length. For prediction of 13 mental disorders, the mid length (20 item) adaptive version showed mean area under the receiver operating characteristic curve of 0.76, compared to 0.79 for the full version. For prediction of brain parameters, mean absolute standardized relationship was 0.06 for the 20 item adaptive version, compared to 0.07 for the full form. This brief, public tool may facilitate the rapid and accurate measurement of overall psychopathology in large scale studies and in clinical practice.
Journal of psychiatric research
Monica E Calkins Theodore Daniel Satterthwaite David R Roalf Adon F G Rosen Ruben C Gur Raquel E Gur
19730205
Development and Simulation Testing of a Computerized Adaptive Version of the Philadelphia Naming Test.
Purpose The purpose of this study was to develop a computerized adaptive test (CAT) version of the Philadelphia Naming Test (PNT; Roach, Schwartz, Martin, Grewal, Brecher, 1996) to reduce test length while maximizing measurement precision. This article is a direct extension of a companion article (Fergadiotis, Kellough, Hula, 2015) in which we fitted the PNT to a 1 parameter logistic item response theory model and examined the validity and precision of the resulting item parameter and ability score estimates. Method Using archival data collected from participants with aphasia, we simulated two PNT CAT versions and two previously published static PNT short forms, and compared the resulting ability score estimates to estimates obtained from the full 175 item PNT. We used a jackknife procedure to maintain independence of the samples used for item estimation and CAT simulation. Results The PNT CAT recovered full PNT scores with equal or better accuracy than the static short forms. Measurement precision was also greater for the PNT CAT than the static short forms, though comparison of adaptive and static nonoverlapping alternate forms showed minimal differences between the two approaches. Conclusion These results suggest that CAT assessment of naming in aphasia has the potential to reduce test burden while maximizing the accuracy and precision of score estimates.
Journal of speech, language, and hearing research JSLHR
William D Hula Stacey Kellough Gerasimos Fergadiotis
46981110
High Efficiency Response Distribution Based Item Selection Algorithms for Short Length Cognitive Diagnostic Computerized Adaptive Testing
Cognitive diagnostic computerized adaptive testing (CD CAT) purports to obtain useful diagnostic information with great efficiency brought by CAT technology. Most of the existing CD CAT item selection algorithms are evaluated when test length is fixed and relatively long, but some applications of CD CAT, such as in interim assessment, require to obtain the cognitive pattern with a short test. The mutual information (MI) algorithm proposed by Wang is the first endeavor to accommodate this need. To reduce the computational burden, Wang provided a simplified scheme, but at the price of scale/sign change in the original index. As a result, it is very difficult to combine it with some popular constraint management methods. The current study proposes two high efficiency algorithms, posterior weighted cognitive diagnostic model (CDM) discrimination index (PWCDI) and posterior weighted attribute level CDM discrimination index (PWACDI) by modifying the CDM discrimination index. They can be considered as an extension of the Kullback Leibler (KL) and posterior weighted KL (PWKL) methods. A pre calculation strategy has also been developed to address the computational issue. Simulation studies indicate that the newly developed methods can produce results comparable with or better than the MI and PWKL in both short and long tests. The other major advantage is that the computational issue has been addressed more elegantly than MI. PWCDI and PWACDI can run as fast as PWKL. More importantly, they do not suffer from the problem of scale/sign change as MI and, thus, can be used with constraint management methods together in a straightforward manner.
Applied psychological measurement
Chanjin Zheng Hua-hua Chang
47009453
Is a Computerized Adaptive Test More Motivating Than a Fixed Item Test?
Computer adaptive tests provide important measurement advantages over traditional fixed item tests, but research on the psychological reactions of test takers to adaptive tests is lacking. In particular, it has been suggested that test taker engagement, and possibly test performance as a consequence, could benefit from the control that adaptive tests have on the number of test items examinees answer correctly. However, previous research on this issue found little support for this possibility. This study expands on previous research by examining this issue in the context of a mathematical ability assessment and by considering the possible effect of immediate feedback of response correctness on test engagement, test anxiety, time on task, and test performance. Middle school students completed a mathematics assessment under one of three test type conditions (fixed, adaptive, or easier adaptive) and either with or without immediate feedback about the correctness of responses. Results showed little evidence for test type effects. The easier adaptive test resulted in higher engagement and lower anxiety than either the adaptive or fixed item tests; however, no significant differences in performance were found across test types, although performance was significantly higher across all test types when students received immediate feedback. In addition, these effects were not related to ability level, as measured by the state assessment achievement levels. The possibility that test experiences in adaptive tests may not in practice be significantly different than in fixed item tests is raised and discussed to explain the results of this and previous studies.
Applied psychological measurement
Guangming Ling Yigal Attali Bridgid M Finn Elizabeth Stone
31156244
A computerized adaptive test for enterprising personality assessment in youth.
Backgound Assessing specific personality traits has shown better predictive power of enterprising personality than have broad personality traits. Hitherto, there have been no instruments that evaluate the combination of specific personality traits of enterprising personality in an adaptive format. So, the aim was to develop a Computerized Adaptive Test (CAT) to assess enterprising personality in young people. METHODS A pool of 161 items was developed and applied to two sets of participants (n 1 357 students, M age 17.89; SD age 3.26; n 2 2,693 students; M age 16.52, SD age 1.38) using a stratified sampling method. RESULTS 107 items that assess achievement motivation, risk taking, innovativeness, autonomy, self efficacy, stress tolerance, internal locus of control, and optimism were selected. The assumption of unidimensionality was tested. The CAT demonstrated high precision for a wide range of q, using a mean of 10 items and demonstrating a relatively low Standard Error (0.378) CONCLUSIONS A brief, valid, and precise instrument was obtained with relevant implications for educational and entrepreneurial contexts.
Psicothema
Ignacio Pedrosa Javier Suarez-Alvarez Eduardo Garcia-Cueto Jose Muniz