Equations
Let N be the total number of subjects, let n be the number of ratings per subject, and let k be the number of categories into which assignments are made. The subjects are indexed by i = 1, ... N and the categories are indexed by j = 1, ... k. Let nij represent the number of raters who assigned the i-th subject to the j-th category.
First calculate pj, the proportion of all assignments which were to the j-th category:
(2)
Now calculate, the extent to which raters agree for the i-th subject (i.e., compute how many rater--rater pairs are in agreement, relative to the number of all possible rater--rater pairs):
(3)
Now compute, the mean of the 's, and which go into the formula for :
(4)
(5)
Read more about this topic: Fleiss' Kappa