IRT
7929303
An item response theory analysis of self report measures of adult attachment.
Self report measures of adult attachment are typically scored in ways (e.g. averaging or summing items) that can lead to erroneous inferences about important theoretical issues, such as the degree of continuity in attachment security and the differential stability of insecure attachment patterns. To determine whether existing attachment scales suffer from scaling problems, the authors conducted an item response theory (IRT) analysis of 4 commonly used self report inventories: Experiences in Close Relationships scales (K. A. Brennan, C. L. Clark, P. R. Shaver, 1998) Adult Attachment Scales (N. L. Collins S. J. Read, 1990) Relationship Styles Questionnaire (D. W. Griffin K. Bartholomew, 1994) and J. Simpson's (1990) attachment scales. Data from 1,085 individuals were analyzed using F. Samejima's (1969) graded response model. The authors' findings indicate that commonly used attachment scales can be improved in a number of important ways. Accordingly, the authors show how IRT techniques can be used to develop new attachment scales with desirable psychometric properties.
Journal of personality and social psychology
Robert Chris Fraley Niels G Waller Kelly A Brennan
14336138
IRT Based Internal Measures of Differential Functioning of Items and Tests
Internal measures of differential functioning of items and tests (DHFIT) based on item response theory (IRT) are proposed. Within the DFIT context, the new differential test functioning (DTF) index leads to two new measures of differential item functioning (DIF) with the following properties: (1) The compensatory DIF (CDIF) indexes for all items in a test sum to the DTF index for that test and, unlike current DIF procedures, the CDIF index for an item does not assume that the other items in the test are unbi ased (2) the noncompensatory DIF (NCDIF) index, which assumes that the other items in the test are unbiased, is comparable to some of the IRT based DIP indexes; and (3) COIF and NCDIF, as well as DTF, are equally valid for polytomous and multidimensional IRT models. Monte carlo study results, comparing these indexes with Lord's kh2 test, the signed area measure, and the unsigned area measure, demonstrate that the DFIT framework is accu rate in assessing DTF, COIF, and NCDIF.
Nambury S Roju Wim J van der Linden Paul F Fleer
8800171
Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement
BackgroundHealth outcomes researchers are increasingly applying Item Response Theory (IRT) methods to questionnaire development, evaluation, and refinement efforts.ObjectiveTo provide a brief overview of IRT, to review some of the critical issues associated with IRT applications, and to demonstrate the basic features of IRT with an example.MethodsExample data come from 6,504 adolescent respondents in the National Longitudinal Study of Adolescent Health public use data set who completed to the 19 item Feelings Scale for depression. The sample was split into a development and validation sample. Scale items were calibrated in the development sample with the Graded Response Model and the results were used to construct a 10 item short form. The short form was evaluated in the validation sample by examining the correspondence between IRT scores from the short form and the original, and by comparing the proportion of respondents identified as depressed according to the original and short form observed cut scores.ResultsThe 19 items varied in their discrimination (slope parameter range: .86 2.66) and item location parameters reflected a considerable range of depression .72 3.39) However, the item set is most discriminating at higher levels of depression. In the validation sample IRT scores generated from the short and long forms were correlated at .96 and the average difference in these scores was .01. In addition, nearly 90% of the sample was classified identically as at risk or not at risk for depression using observed score cut points from the short and long forms.ConclusionsWhen used appropriately, IRT can be a powerful tool for questionnaire development, evaluation, and refinement, resulting in precise, valid, and relatively brief instruments that minimize response burden.
Quality of Life Research
Maria Edelen Bryce B Reeve
145778544
An Application of Item Response Time: The Effort Moderated IRT Model
The validity of inferences based on achievement test scores is dependent on the amount of effort that examinees put forth while taking the test. With low stakes tests, for which this problem is particularly prevalent, there is a consequent need for psychometric models that can take into account differing levels of examinee effort. This article introduces the effort moderated IRT model, which incorporates item response time into proficiency estimation and item parameter estimation. In two studies of the effort moderated model when rapid guessing (i.e. reflecting low examinee effort) was present, one based on real data and the other on simulated data, the effort moderated model performed better than the standard 3PL model. Specifically, it was found that the effort moderated model (a) showed better model fit, (b) yielded more accurate item parameter estimates, (c) more accurately estimated test information, and (d) yielded proficiency estimates with higher convergent validity.
Steven L Wise Christine E DeMars
2587408
Mokken scale analysis of mental health and well being questionnaire item responses: a non parametric IRT method in empirical research for applied health researchers
BackgroundMokken scaling techniques are a useful tool for researchers who wish to construct unidimensional tests or use questionnaires that comprise multiple binary or polytomous items. The stochastic cumulative scaling model offered by this approach is ideally suited when the intention is to score an underlying latent trait by simple addition of the item response values. In our experience, the Mokken model appears to be less well known than for example the (related) Rasch model, but is seeing increasing use in contemporary clinical research and public health. Mokken's method is a generalisation of Guttman scaling that can assist in the determination of the dimensionality of tests or scales, and enables consideration of reliability, without reliance on Cronbach's alpha. This paper provides a practical guide to the application and interpretation of this non parametric item response theory method in empirical research with health and well being questionnaires.MethodsScalability of data from 1) a cross sectional health survey (the Scottish Health Education Population Survey) and 2) a general population birth cohort study (the National Child Development Study) illustrate the method and modeling steps for dichotomous and polytomous items respectively. The questionnaire data analyzed comprise responses to the 12 item General Health Questionnaire, under the binary recoding recommended for screening applications, and the ordinal/polytomous responses to the Warwick Edinburgh Mental Well being Scale.Results and conclusionsAfter an initial analysis example in which we select items by phrasing (six positive versus six negatively worded items) we show that all items from the 12 item General Health Questionnaire (GHQ 12) when binary scored were scalable according to the double monotonicity model, in two short scales comprising six items each (Bech's "well being" and "distress" clinical scales) An illustration of ordinal item analysis confirmed that all 14 positively worded items of the Warwick Edinburgh Mental Well being Scale (WEMWBS) met criteria for the monotone homogeneity model but four items violated double monotonicity with respect to a single underlying dimension.Software availability and commands used to specify unidimensionality and reliability analysis and graphical displays for diagnosing monotone homogeneity and double monotonicity are discussed, with an emphasis on current implementations in freeware.
BMC Medical Research Methodology
Jan Stochl Peter B Jones Tim J Croudace
56252490
The development and testing of a new version of the cognitive reflection test applying item response theory (irt)
The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two parameter logistic model) and at developing a new version of the scale (the CRT long) which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT long, we report the first investigation into the development of cognitive reflection. Copyright (c) 2015 John Wiley Sons, Ltd. [ABSTRACT FROM AUTHOR]
Caterina Primi Kinga Morsanyi Francesca Chiesi Maria Benedetta Donati Jayne Hamilton
120817561
Testing Measurement Invariance: A Comparison of Multiple Group Categorical CFA and IRT
This study investigated two major approaches in testing measurement invariance for ordinal measures: multiple group categorical confirmatory factor analysis (MCCFA) and item response theory (IRT) Unlike the ordinary linear factor analysis, MCCFA can appropriately model the ordered categorical measures with a threshold structure. A simulation study under various conditions was conducted for the comparison of MCCFA and IRT with respect to the power to detect the lack of invariance across groups. Both MCCFA and IRT showed reasonable power to identify the noninvariant item when differential item functioning (DIF) was large. The false positive rates were relatively high in both methods, however. The adjustment of critical values improved the performance of MCCFA by reducing false positive rates substantially and yet yielding adequate power. Alternative model fit indexes of MCCFA were also examined and they were found to be reliable to detect DIF, in general.
Eun Sook Kim Myeongsun Yoon
141311098
Modeling Nonignorable Missing Data with Item Response Theory (IRT) Research Report. ETS RR 10 11.
Large scale educational surveys are low stakes assessments of educational outcomes conducted using nationally representative samples. In these surveys, students do not receive individual scores, and the outcome of the assessment is inconsequential for respondents. The low stakes nature of these surveys, as well as variations in average performance across countries and other factors such as different testing traditions, are contributing factors to the amount of omitted responses in these assessments. While underlying reasons for omissions are not completely understood, common practice in international assessments is to employ a deterministic treatment of omissions, either as missing observations or as responses that are considered wrong. Both approaches appear problematic. In this project, we analyzed the effects of treating omitted responses either as missing or as wrong, as is done in the majority of international studies, and compared these data treatment solutions to model based approaches to treating omitted responses. The two types of model based approaches used in this study are: (a) extensions of multidimensional item response theory (IRT) with an additional dimension based on response indicator variables defined and calibrated together with the set of items containing the observed responses and (b) multidimensional, multiple group IRT models with a grouping variable representing the within country stratification of respondents by the amount of omitted responses. These two model based approaches were compared on the basis of simulated data and data from about 250,000 students from 30 Organisation for Economic Co operation and Development (OECD) Member countries participating in an international large scale assessment.
Norman Rose Matthias von Davier Xueli Xu
146320226
The concept of life satisfaction across cultures: An IRT analysis
Abstract The present study examined measurement equivalence of the Satisfaction with Life Scale between American and Chinese samples using multigroup Structural Equation Modeling (SEM) Multiple indicator multiple cause model (MIMIC) and Item Response Theory (IRT) Whereas SEM and MIMIC identified only one biased item across cultures, the IRT analysis revealed that four of the five items had differential item functioning. According to IRT, Chinese whose latent life satisfaction scores were quite high did not endorse items such as "So far I have gotten the important things I want in life" and "If I could live my life over, I would change almost nothing." The IRT analysis also showed that even when the unbiased items were weighted more heavily than the biased items, the latent mean life satisfaction score of Chinese was substantially lower than that of Americans. The differences among SEM, MIMIC, and IRT are discussed.
Shigehiro Oishi
143397584
Modeling Item Position Effects Within an IRT Framework
Changing the order of items between alternate test forms to prevent copying and to enhance test security is a common practice in achievement testing. However, these changes in item order may affect item and test characteristics. Several procedures have been proposed for studying these item order effects. The present study explores the use of descriptive and explanatory models from item response theory for detecting and modeling these effects in a one step procedure. The framework also allows for consideration of the impact of individual differences in position effect on item difficulty. A simulation was conducted to investigate the impact of a position effect on parameter recovery in a Rasch model. As an illustration, the framework was applied to a listening comprehension test for French as a foreign language and to data from the PISA 2006 assessment. In achievement testing, administering the same set of items in different orders is a common strategy to prevent copying and to enhance test security. These item order manipulations across alternate test forms, however, may not be without consequence. After the early work of Mollenkopf (1950) it repeatedly has been shown that changes in the placement of items may have unintended effects on test and item characteristics (Leary Dorans, 1985) Traditionally, two kinds of item position effects have been discerned (Kingston Dorans, 1984) a practice or a learning effect occurs when the items become easier in later positions, and a fatigue effect occurs when items become more difficult if placed towards the end of the test. Recent empirical studies on the effect of item position include Hohensinn et al. (2008) Meyers, Miller, and Way (2009) Moses, Yang and Wilson (2007) Pommerich and Harris (2003) and Schweizer, Schreiner and Gold (2009) In the present article, item position effects will be studied within De Boeck and Wilson's (2004) framework of descriptive and explanatory item response models. It will be argued that modeling item position effects across alternate test forms can be considered as a special case of differential item functioning (DIF) Apart from the DIF approach, the linear logistic test model of Fischer (1973) and its random weights extension (Rijmen De Boeck, 2002) will be used to investigate the effect of item position on individual item parameters and to model the trend of item position effects across items. A new feature of the approach is that individual differences in the effects of item position on difficulty can be taken into account. In the following pages we first will present a brief overview of current approaches to studying the impact of item position on test scores and item characteristics. We then present the proposed item response theory (IRT) framework used for modeling item position effects. After demonstrating the impact of a position effect on parameter recovery with simulated data, the framework is applied to a listening
Dries Debeer Rianne Janssen