Joint Probability of Agreement
The joint-probability of agreement is probably the most simple and least robust measure. It is the number of times each rating (e.g. 1, 2, ... 5) is assigned by each rater divided by the total number of ratings. It assumes that the data are entirely nominal. It does not take into account that agreement may happen solely based on chance. Some question, though, whether there is a need to 'correct' for chance agreement; and suggest that, in any case, any such adjustment should be based on an explicit model of how chance and error affect raters' decisions.
When the number of categories being used is small (e.g. 2 or 3), the likelihood for 2 raters to agree by pure chance increases dramatically. This is because both raters must confine themselves to the limited number of options available, which impacts the overall agreement rate, and not necessarily their propensity for "intrinsic" agreement (is considered "intrinsic" agreement, an agreement not due to chance). Therefore, the joint probability of agreement will remain high even in the absence of any "intrinsic" agreement among raters. A useful inter-rater reliability coefficient is expected to (1) be close to 0, when there is no "intrinsic" agreement, and (2) to increase as the "intrinsic" agreement rate improves. Most chance-corrected agreement coefficients achieve the first objective. However, the second objective is not achieved by many known chance-corrected measures.
Read more about this topic: Inter-rater Reliability
Famous quotes containing the words joint, probability and/or agreement:
“No Government can be long secure without a formidable Opposition. It reduces their supporters to that tractable number which can be managed by the joint influences of fruition and hope. It offers vengeance to the discontented, and distinction to the ambitious; and employs the energies of aspiring spirits, who otherwise may prove traitors in a division or assassins in a debate.”
—Benjamin Disraeli (1804–1881)
“Liberty is a blessing so inestimable, that, wherever there appears any probability of recovering it, a nation may willingly run many hazards, and ought not even to repine at the greatest effusion of blood or dissipation of treasure.”
—David Hume (1711–1776)
“The doctrine of those who have denied that certainty could be attained at all, has some agreement with my way of proceeding at the first setting out; but they end in being infinitely separated and opposed. For the holders of that doctrine assert simply that nothing can be known; I also assert that not much can be known in nature by the way which is now in use. But then they go on to destroy the authority of the senses and understanding; whereas I proceed to devise helps for the same.”
—Francis Bacon (1560–1626)