Abstract
Sixteen indices of interobserver agreement and six methods for estimating coefficients of interobserver reliability were critiqued. The agreement statistics were found to be imprecise, limited psychometrically, and relatively inflexible in terms of the diverse categorical and quantitative data sets typically encountered in mental retardation research. Five of the reliability statistics produced precise estimates of agreement, yet possessed similar limitations. Only the intraclass correlation--generalizability theory approach seemed to offer the precision, comprehensiveness, and flexibility required to deal with the complexity of reliability assessment. A basic generalizability model was described and illustrated with group and single-subject research data.
Keywords
Generalizability theoryIntraclass correlationCategorical variableReliability (semiconductor)PsychologyStatisticsInter-rater reliabilityFlexibility (engineering)GeneralizationPsychometricsClinical psychologyMathematicsDevelopmental psychologyRating scale
Affiliated Institutions
Related Publications
Interrater reliability: the kappa statistic.
The kappa statistic is frequently used to test interrater reliability. The importance of rater reliability lies in the fact that it represents the extent to which the data colle...
Intraclass correlations: Uses in assessing rater reliability.
Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intracla...
Interobserver Variability in the Assessment of Neurologic History and Examination in the Stroke Data Bank
Interobserver reliability in obtaining neurologic histories and examinations was investigated among neurologists collaborating in the Stroke Data Bank (SDB). Seventeen in-hospit...
The Measurement of Observer Agreement for Categorical Data
This paper presents a general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies. The procedure essentially invo...
Measuring Agreement between Two Judges on the Presence or Absence of a Trait
At least a dozen indexes have been proposed for measuring agreement between two judges on a categorical scale. Using the binary (positive-negative) case as a model, this paper p...
Publication Info
- Year
- 1979
- Type
- article
- Volume
- 83
- Issue
- 5
- Pages
- 460-72
- Citations
- 160
- Access
- Closed
External Links
Citation Metrics
160
OpenAlex
Cite This
Ronald A. Berk
(1979).
Generalizability of behavioral observations: a clarification of interobserver agreement and interobserver reliability..
PubMed
, 83
(5)
, 460-72.