When should I use Kappa instead of percent agreement?

Always use Kappa when reporting inter-rater reliability in research. Percent agreement inflates reliability by ignoring chance.

What is a good Kappa value?

Values above 0.60 indicate substantial agreement. For high-stakes decisions, aim for 0.80 or higher.

Cohen's Kappa is designed for two raters. For multiple raters, use Fleiss' Kappa instead.

Calculate Cohen's Kappa to measure the agreement between two raters beyond what would be expected by chance.

Both Raters Agree Yes

Both Raters Agree No

Rater 1 Yes, Rater 2 No

Rater 1 No, Rater 2 Yes

Cohen's Kappa

0.400

Observed Agreement70.0 %

Chance Agreement50.0 %

Total Observations100

Cohen's Kappa vs Rater 1 Yes, Rater 2 No

Cohen's Kappa measures inter-rater agreement while correcting for chance. It is more robust than simple percent agreement.

Kappa = (P_observed - P_expected) / (1 - P_expected)

< 0: Less than chance agreement

0.01-0.20: Slight agreement

0.21-0.40: Fair agreement

0.41-0.60: Moderate agreement

0.61-0.80: Substantial agreement

0.81-1.00: Almost perfect agreement

Two graders evaluate 100 essays: both say pass (40), both say fail (30), only Rater 1 passes (10), only Rater 2 passes (20).