Inter-rater Reliability - Correlation Coefficients

Correlation Coefficients

Main articles: Pearson product-moment correlation coefficient, Spearman's rank correlation coefficient

Either Pearson's or Spearman's can be used to measure pairwise correlation among raters using a scale that is ordered. Pearson assumes the rating scale is continuous; Spearman assumes only that it is ordinal. If more than two raters are observed, an average level of agreement for the group can be calculated as the mean of the (or ) values from each possible pair of raters.

Both the Pearson and Spearman coefficients consider only relative position. For example, (1, 2, 1, 3) is considered perfectly correlated with (2, 3, 2, 4).