Psychometric Validation of Continuous Rating Scales from Complex Data

Abstract

Rating scales are frequently used for the primary outcome measurement in psychopharmacological trials. When using such scales in research or in clinical practice, information on their psychometric properties should be available. These properties are generally investigated when a scale is being developed, however, the reliability of a scale is not a fixed characteristic of the instrument, but is rather population dependent. More heterogeneous populations give rise to more reliable measurements. Furthermore, reliability can also depend on other external factors like, for instance, the skills or the level of training of the raters. It is therefore useful to evaluate the reliability of certain rating scale, whenever this scale is applied. However, many approaches for estimating reliability are based on very restrictive modelling frameworks. A common feature in present-day psychopharmacological trials is the presence of repeated measurements. The modelling frameworks used in CTT or G-theory will frequently be inappropriate to study reliability in this scenario. In the present work we have tried to extend the concept of reliability to this more general setting. A psychiatric symptom scale will be useful only if it can discriminate among different patients, essentially those who have a mental illness from those that do not, or those patients who are in a more advanced stage of a disease from those who are in a more primary stage, or those patients who have made progress from those who have not, or did so to a lesser extent. This discriminating capability will be possible only if the scale’s values vary more between subjects than what they vary within the same subject. This relation between the within and between-subject variability is what we try to determine when we study the reliability of the scale. The reliability of a scale is therefore the capacity of the scale to discriminate between different subjects or different groups of subjects. The appraisal of reliability has certainly been among the most central issues in psychometrics during the past century. Despite the fact that they are all targeting the same concept, measures used to quantify reliability have largely depended on the data structure. This lack of a unifying approach has resulted in a myriad of measures, which sometimes lead to different conclusions and varying interpretations. Hitherto, the two main contexts for the appraisal of reliability, i.e., the cross-sectional and longitudinal scenario (single-administration and multiple-administration) have been studied using different approaches. In the present work we have introduced a general definition of reliability based on a simple set of properties. This definition can be equally applied in the cross-sectional and longitudinal setting. The definition lead to a whole family of reliability measures, the family. All the members of this family are built upon the same basic elements: the roots of the equation q(λ) = |Σ − λV | = 0; where Σ expresses the within-subject variability and V the total variability, and therefore V − Σ the between-subject variability.