The Philosophy of Inter-rater Agreement
There are several operational definitions of "inter-rater reliability" in use by Examination Boards, reflecting different viewpoints about what is reliable agreement between raters.
There are three operational definitions of agreement:
1. Reliable raters agree with the "official" rating of a performance.
2. Reliable raters agree with each other about the exact ratings to be awarded.
3. Reliable raters agree about which performance is better and which is worse.
These combine with two operational definitions of behavior:
A. Reliable raters are automatons, behaving like "rating machines". This category includes rating of essays by computer . This behavior can be evaluated by Generalizability theory.
B. Reliable raters behave like independent witnesses. They demonstrate their independence by disagreeing slightly. This behavior can be evaluated by the Rasch model.
Read more about this topic: Inter-rater Reliability
Famous quotes containing the words philosophy and/or agreement:
“Englishmen are babes in philosophy and so prefer faction-fighting to the labour of its unfamiliar thought.”
—William Butler Yeats (1865–1939)
“The doctrine of those who have denied that certainty could be attained at all, has some agreement with my way of proceeding at the first setting out; but they end in being infinitely separated and opposed. For the holders of that doctrine assert simply that nothing can be known; I also assert that not much can be known in nature by the way which is now in use. But then they go on to destroy the authority of the senses and understanding; whereas I proceed to devise helps for the same.”
—Francis Bacon (1560–1626)