Inter-rater Reliability - Joint Probability of Agreement

Joint Probability of Agreement

The joint-probability of agreement is probably the most simple and least robust measure. It is the number of times each rating (e.g. 1, 2, ... 5) is assigned by each rater divided by the total number of ratings. It assumes that the data are entirely nominal. It does not take into account that agreement may happen solely based on chance. Some question, though, whether there is a need to 'correct' for chance agreement; and suggest that, in any case, any such adjustment should be based on an explicit model of how chance and error affect raters' decisions.

When the number of categories being used is small (e.g. 2 or 3), the likelihood for 2 raters to agree by pure chance increases dramatically. This is because both raters must confine themselves to the limited number of options available, which impacts the overall agreement rate, and not necessarily their propensity for "intrinsic" agreement (is considered "intrinsic" agreement, an agreement not due to chance). Therefore, the joint probability of agreement will remain high even in the absence of any "intrinsic" agreement among raters. A useful inter-rater reliability coefficient is expected to (1) be close to 0, when there is no "intrinsic" agreement, and (2) to increase as the "intrinsic" agreement rate improves. Most chance-corrected agreement coefficients achieve the first objective. However, the second objective is not achieved by many known chance-corrected measures.

Read more about this topic:  Inter-rater Reliability

Famous quotes containing the words joint, probability and/or agreement:

    Your letter of excuses has arrived. I receive the letter but do not admit the excuses except in courtesy, as when a man treads on your toes and begs your pardon—the pardon is granted, but the joint aches, especially if there is a corn upon it.
    George Gordon Noel Byron (1788–1824)

    The source of Pyrrhonism comes from failing to distinguish between a demonstration, a proof and a probability. A demonstration supposes that the contradictory idea is impossible; a proof of fact is where all the reasons lead to belief, without there being any pretext for doubt; a probability is where the reasons for belief are stronger than those for doubting.
    Andrew Michael Ramsay (1686–1743)

    That which corrodes the souls of the persecuted is the monstrous inner agreement with the prevailing prejudice against them.
    Eric Hoffer (1902–1983)