Simple moment estimates of the kappa-coefficient and its variance

Abstract

Estimating equations are used to develop simple non-iterative estimates of the kappa-coefficient that can be used when there are more than two random raters and/or unbalanced data (each subject is not judged by every rater). We show that there is a simple way to estimate the variance of any estimate of the kappa-coefficient that is a solution to an estimating equation. Two non-iterative estimates that are shown to be solutions to estimating equations are Fleiss's estimate and Schouten's estimate. Also, assuming that the underlying data are beta-binomial, we compare the asymptotic relative efficiency of the non-iterative estimators Of kappa relative to the iterative maximum likelihood estimator (MLE) of kappa from the beta-binomial distribution. Fleiss's estimator was found to have high efficiency. Finally, simulations are used to compare the finite sample performance of these estimators as well as the MLE from the beta-binomial distribution. In the simulations, the Newton-Raphson algorithm for the MLE from the beta-binomial model did not always converge in small samples, which also supports the use of a non-iterative estimate in small samples. The estimators are also compared by using a psychiatric data set given by Fleiss.