Inter-Rater Reliability Calculator Formula

Understand the math behind the inter-rater reliability calculator. Each variable explained with a worked example.

Use the Inter-Rater Reliability Calculator

Formulas Used

Cohen's Kappa

kappa = (p_observed - p_expected) / (1 - p_expected)

Observed Agreement

observed_agreement = p_observed * 100

Chance Agreement

chance_agreement = p_expected * 100

Total Observations

total_observations = n_total

Variables

VariableDescriptionDefault
agree_both_yesBoth Raters Agree Yes40
agree_both_noBoth Raters Agree No30
rater1_yes_rater2_noRater 1 Yes, Rater 2 No10
rater1_no_rater2_yesRater 1 No, Rater 2 Yes20
n_totalDerived value= agree_both_yes + agree_both_no + rater1_yes_rater2_no + rater1_no_rater2_yescalculated
p_observedDerived value= (agree_both_yes + agree_both_no) / n_totalcalculated
p_yes_r1Derived value= (agree_both_yes + rater1_yes_rater2_no) / n_totalcalculated
p_yes_r2Derived value= (agree_both_yes + rater1_no_rater2_yes) / n_totalcalculated
p_no_r1Derived value= 1 - p_yes_r1calculated
p_no_r2Derived value= 1 - p_yes_r2calculated
p_expectedDerived value= p_yes_r1 * p_yes_r2 + p_no_r1 * p_no_r2calculated

How It Works

How Cohen's Kappa Works

Cohen's Kappa measures inter-rater agreement while correcting for chance. It is more robust than simple percent agreement.

Formula

Kappa = (P_observed - P_expected) / (1 - P_expected)

Interpretation

  • < 0: Less than chance agreement
  • 0.01-0.20: Slight agreement
  • 0.21-0.40: Fair agreement
  • 0.41-0.60: Moderate agreement
  • 0.61-0.80: Substantial agreement
  • 0.81-1.00: Almost perfect agreement

    • Worked Example

    Two graders evaluate 100 essays: both say pass (40), both say fail (30), only Rater 1 passes (10), only Rater 2 passes (20).

    agree_both_yes = 40agree_both_no = 30rater1_yes_rater2_no = 10rater1_no_rater2_yes = 20
    1. 01Total: 40 + 30 + 10 + 20 = 100
    2. 02P_observed = (40 + 30) / 100 = 0.70
    3. 03P(R1 yes) = 50/100 = 0.50, P(R2 yes) = 60/100 = 0.60
    4. 04P_expected = 0.50 x 0.60 + 0.50 x 0.40 = 0.30 + 0.20 = 0.50
    5. 05Kappa = (0.70 - 0.50) / (1 - 0.50) = 0.20 / 0.50 = 0.400

    Ready to run the numbers?

    Open Inter-Rater Reliability Calculator