Weighted Kappa Model Overview

Weighted Kappa, often called weighted Cohen’s kappa, evaluates agreement between two ratings measured on the same ordered scale. It improves on ordinary Cohen’s kappa by recognising that a one-category disagreement is less serious than a two- or three-category disagreement. The present analysis compares G1 and G3 grade classifications using four ordered bands: low, medium, high and very high.

The complete 4 × 4 table contains 649 paired classifications. The diagonal contains 463 exact matches. Of the 186 disagreements, 185 are only one category apart, 1 is two categories apart and 0 are three categories apart. That concentration near the diagonal is the central reason quadratic weighted kappa (0.765160) is higher than linear weighted kappa (0.647398).

Exact Research Question

How strongly do G1 and G3 ordinal grade bands agree beyond the agreement expected from their different marginal distributions, and how does the result change when disagreements are penalised linearly rather than quadratically?

Quick Answer: Weighted Kappa Result

The primary quadratic weighted kappa is 0.765160. Under quadratic disagreement weights, observed disagreement is 0.032357, compared with 0.137785 expected from the row and column margins. The calculation therefore removes 76.52% of the weighted disagreement expected by chance.

Table of Contents

1. What weighted kappa measures
2. Research question and hypotheses
3. When to use weighted kappa
4. Linear, quadratic and custom weights
5. Variables and ordinal coding
6. Formula and manual calculation
7. Observed, expected and residual tables
8. Complete weighted kappa results
9. Assumptions and diagnostic checks
10. Comparison with related methods
11. Calculator workflow
12. Python chart interpretations
13. Paired R chart interpretations
14. Python, R, SPSS, SAS and Excel
15. Expandable code
16. APA interpretation and reporting
17. Common mistakes
18. Practice questions
19. Reports and worked files
20. Related guides
21. Frequently asked questions
22. Conclusion

What Is Weighted Kappa?

Weighted Kappa is a chance-corrected agreement coefficient for two ratings recorded on the same ordinal scale. It starts with a cross-classification table, assigns each cell a disagreement weight, and compares the observed weighted disagreement with the disagreement expected if the two marginal distributions were independent.

Ordinary kappa treats low-versus-medium and low-versus-very-high disagreements as equally wrong. Weighted kappa does not. In this four-band example, a low-to-medium disagreement receives a linear penalty of 1/3 or a quadratic penalty of 1/9, whereas low-to-very-high receives the maximum penalty of 1 under both systems.

Why Weighted Kappa Is Not a Reliability Percentage

The value 0.7652 is not “76.52% of ratings are correct.” It means the observed quadratic disagreement is 76.52% lower than the disagreement expected from the marginal distributions. Raw exact agreement is 71.34%, while the weighted agreement score is 96.76%. These quantities answer different questions and should be reported separately.

Why Marginal Distributions Matter

G1 has 157 low, 340 medium, 128 high and 24 very-high classifications. G3 has 100, 355, 148 and 46, respectively. Because the G3 distribution is shifted upward, raw agreement alone would not fully describe the agreement process. Weighted kappa retains the paired table and corrects for the agreement expected from these margins.

Research Question, Hypotheses and Data Design

Research Question

Do G1 and G3 place students in the same or nearby ordinal grade bands more often than expected from their separate grade-band distributions?

Prespecified Analysis Decisions

The grade cut points, category order, pairing rule and weighting scheme should be selected before inspecting the kappa result. Quadratic weights are primary here because a two-band classification error is intended to count four times as much as a one-band error, while a three-band error receives the maximum penalty.

When to Use Weighted Kappa

Decision Flow

For two nominal ratings, use the ordinary kappa statistic. For more than two raters, consider Fleiss’ kappa. For paired binary change rather than agreement, use McNemar’s test; for paired multicategory marginal change, use the McNemar–Bowker test.

Linear, Quadratic and Custom Weighted Kappa

The numerical value of weighted kappa depends on the weight matrix. Weighting is part of the estimand, not a decorative software option.

Linear Disagreement Weights

Quadratic Disagreement Weights

Why Quadratic Kappa Is Higher Here

There are 185 adjacent disagreements, only 1 two-step disagreement and no three-step disagreement. Quadratic weights sharply discount adjacent errors: a one-step error contributes only 1/9 of a maximum error instead of 1/3. Because almost all mismatches are adjacent, quadratic observed disagreement falls to 0.032357, compared with linear observed disagreement of 0.096045.

Variables, Ordinal Coding and Data Dictionary

Marginal Grade-Band Distributions

Scale-Score Summary

Category Retention and Movement from Each G1 Band

The very-high band has the highest exact retention at 91.67%, although it contains only 24 cases. The low band has the lowest exact retention at 56.05%; all 69 low-band mismatches move one step upward to medium. Medium is the largest row and retains 267 of 340 students, while 61 move upward and 12 move downward.

Where Each G3 Band Came From

Column percentages reveal a different perspective from row retention. Of the 100 students classified low at G3, 88.00% were already low at G1 and 12.00% came from medium. Among the 46 very-high G3 classifications, 47.83% were already very high, 50.00% moved up from high, and 2.17% moved two bands from medium.

Weighted Kappa Formula and Manual Calculation

This article uses disagreement weights. With cell proportion p_ij, row marginal p_i+, column marginal p_+j and disagreement weight w_ij, weighted kappa is:

Linear Calculation

Quadratic Calculation

Manual Check from Disagreement Distances

For the linear system, the 185 one-step disagreements contribute 185 × (1/3) weighted count units and the one two-step disagreement contributes 1 × (2/3). Dividing by 649 gives D_o,L = 0.096045. For quadratic weights, the same cells contribute 185 × (1/9) plus 1 × (4/9), producing D_o,Q = 0.032357.

Cell-by-Cell Disagreement Contributions

The contribution table separates frequency from severity. Low→medium is the largest quadratic contribution because it occurs 69 times. Medium→very-high spans two categories and receives a four-times-larger quadratic weight than an adjacent error, but it occurs once and therefore contributes only 0.000685. This is why a weighted analysis must multiply cell frequency by the selected penalty rather than ranking errors by distance alone.

Expected Disagreement Is Built from the Margins

Expected cell probability is p_i+p_+j. For example, the expected low-G1/medium-G3 probability is (157/649) × (355/649) = 0.132324. Under quadratic weights, multiplying that probability by 1/9 contributes 0.014703 to expected disagreement. Repeating this for all 16 cells gives D_e,Q = 0.137785100753.

Chance correction therefore does not assume equal category frequencies. It uses the actual G1 and G3 margins. This matters because medium dominates both distributions and G3 is shifted toward higher categories.

Observed Agreement, Expected Counts and Diagnostic Tables

Observed G1-by-G3 Agreement Matrix

The diagonal holds 463 exact matches. Blue cells above the diagonal represent movement to a higher G3 band; amber cells below the diagonal represent movement to a lower G3 band.

Row Percentages

Exact retention is 56.05% for low, 78.53% for medium, 67.19% for high and 91.67% for very-high G1 classifications.

Expected Counts Under Independent Margins

Expected diagonal counts total 241.060, corresponding to 37.14% chance exact agreement. Observed diagonal agreement is 463, substantially above that expectation.

Pearson Residuals for the Unweighted Table

Large positive diagonal residuals show that exact matches occur far more often than expected under independence. Negative residuals in distant off-diagonal cells show that severe disagreements are rarer than expected.

Disagreement Distance Distribution

Quadratic Observed Disagreement Contributions

Observed Versus Expected Exact Agreement by Band

Every diagonal cell exceeds its chance expectation. The strongest proportional concentration is in the very-high cell: 22 observed versus 1.701 expected. The low cell has 88 observed versus 24.191 expected. These large diagonal excesses are why both unweighted and weighted kappa are well above zero.

Marginal Shift from G1 to G3

The margins show a redistribution rather than identical prevalence. Low declines by 57 cases, while high rises by 20 and very high rises by 22. Medium rises by 15. Weighted kappa remains high because most individual transitions are exact or adjacent, but the margin table demonstrates that agreement does not imply an unchanged grade distribution.

Complete Weighted Kappa Results

Primary Weighted Kappa Results

Agreement Decomposition

Bootstrap Precision Check

The bootstrap is a supplementary reproducibility check generated directly from the verified 4 × 4 empirical table with a fixed seed. It is not presented as an output from the linked software reports. Both intervals remain well above zero and preserve the ordering quadratic κ > linear κ.

Directional Transition Detail

Where the Quadratic Disagreement Comes From

Upward adjacent transitions account for 80.42% of observed quadratic disagreement, while downward adjacent transitions account for 17.46%. The coefficient is therefore not only high; its remaining disagreement is directionally concentrated.

Three Complementary Conclusions from the Same Table

These conclusions are not contradictory. A student can remain close to the earlier band while still being more likely to move upward than downward. Reporting only weighted kappa would omit the third conclusion; reporting only the margins would omit the strong case-level pairing.

Weighted Kappa Assumptions and Diagnostic Checks

Weighted kappa requires a defensible paired ordinal design. The coefficient can be computed mechanically even when the design is inappropriate, so each condition should be checked explicitly.

Prevalence and Marginal Effects

Kappa can change when category prevalence or rater margins change, even with similar raw agreement. Here medium classifications dominate both margins, while G3 contains fewer low and more high/very-high classifications. Reporting the full margins, exact agreement and direction counts protects against overinterpreting one coefficient.

Sparse-Cell Interpretation

Seven of the 16 cells are zero. In this dataset, zero distant-error cells are substantively favourable because they indicate no low-to-high, low-to-very-high, high-to-low, very-high-to-low or very-high-to-medium movements. They also mean that uncertainty for rare error types cannot be learned directly from many observed cases. The supplementary bootstrap interval therefore resamples the full empirical table rather than applying a normal approximation to individual sparse cells.

Independence and Temporal Pairing

The pair is formed within student, so G1 and G3 are intentionally dependent. The independence assumption applies across students: one student’s grade-band pair should not be duplicated or nested in a way that makes pairs correlated. If students are clustered in classrooms or schools and inference must generalise beyond the observed sample, a multilevel ordinal model may be needed in addition to descriptive weighted kappa.

Weighted Kappa Versus Related Agreement and Change Methods

Weighted Kappa Versus Ordinary Kappa

Ordinary κ is 0.5441; linear weighted κ is 0.6474; quadratic weighted κ is 0.7652. The increase is not evidence that ordinary kappa is wrong. It reflects a different loss function: ordinary kappa treats all 186 disagreements equally, while weighted kappa recognises that 185 of them are only one category apart.

Weighted Kappa Versus Marginal-Homogeneity Tests

Agreement and change are distinct. The 153 upward and 33 downward transitions suggest that G3 tends to be higher than G1. A McNemar–Bowker test would test symmetry of the paired square table, while weighted kappa quantifies closeness. A complete study can report both when both questions matter.

Weighted Kappa Versus Correlation

Correlation is unaffected by certain systematic shifts and does not require identical values. Weighted kappa rewards equality and near equality on a common scale. Use Spearman rank correlation for monotonic ordering and weighted kappa for category agreement.

What Different Analyses Would Say About This Exact Matrix

Weighted Kappa Calculator: Step-by-Step Workflow

A weighted kappa calculator needs the full square count matrix and the intended weight system. Entering only the diagonal percentage is insufficient.

Manual Calculator Audit

A trustworthy calculator should reproduce κ_L = 0.647398407 and κ_Q = 0.765159855. If the result is instead .544, the calculator is reporting ordinary unweighted kappa.

Python Weighted Kappa Chart Interpretations

These six Python figures are interpreted with the exact matrix values. Each chart supports a different part of the agreement argument: counts, weights, contributions, sensitivity and final reporting.

1. Agreement Matrix Python

Exact values

The matrix contains 463 diagonal agreements, 153 cells above the diagonal and 33 below. The largest off-diagonal cells are low→medium (69) and medium→high (60).

Statistical meaning

The dense diagonal and near-diagonal concentration visually support a strong weighted kappa. Exact agreement is 71.34%, and 99.85% of all pairs are no more than one category apart.

Next analytical step: Use the matrix before interpreting any single coefficient; it reveals both severity and direction.

2. Linear Disagreement Weights Python

Exact values

Linear penalties are 0, 1/3, 2/3 and 1 for distances 0 through 3.

Statistical meaning

Applying these weights produces observed disagreement 0.096045, expected disagreement 0.272390 and κ = 0.647398.

Next analytical step: Check that category order is identical on both axes before applying the matrix.

3. Quadratic Disagreement Weights Python

Exact values

Quadratic penalties are 0, 1/9, 4/9 and 1.

Statistical meaning

Because 185 of 186 mismatches are adjacent, quadratic observed disagreement falls to 0.032357; κ rises to 0.765160.

Next analytical step: Use quadratic weights only when larger classification errors genuinely deserve disproportionate penalty.

4. Weighted Contributions Python

Exact values

The largest quadratic contributions are low→medium (0.011813) and medium→high (0.010272).

Statistical meaning

The single medium→very-high case contributes 0.000685; despite spanning two bands, it remains a small part of total observed disagreement because it occurs once.

Next analytical step: Investigate cells that are both frequent and heavily weighted; frequency alone does not determine contribution.

5. Weighting Comparison Python

Exact values

Unweighted κ = 0.5441, linear κ = 0.6474, and quadratic κ = 0.7652.

Statistical meaning

The 0.1178 gap between quadratic and linear results quantifies sensitivity to how adjacent errors are discounted.

Next analytical step: Report the weighting definition with the coefficient so readers know which disagreement loss function was used.

6. Result Summary Python

Exact values

The summary combines N = 649, exact agreement 71.34%, linear κ 0.6474 and quadratic κ 0.7652.

Statistical meaning

Observed quadratic agreement is 96.76% compared with 86.22% expected under the margins.

Next analytical step: Pair the summary with the full matrix and direction counts; the coefficient alone does not reveal the upward G3 shift.

R Weighted Kappa Charts in Paired Rows

The R figures are displayed in paired rows, matching the fresh permanent sample. Explanations remain separate so each visual has its own numerical interpretation.

R figure pair 1: Agreement Matrix and Linear Weights

Exact values

Agreement Matrix

The R agreement matrix reproduces all 16 counts and both margins. Diagonal agreement is 463/649.

The diagonal is strongest in the medium cell (267), while the most common movement is low→medium (69).

Cross-check against the verified matrix and the reported coefficient.

Exact values

Linear Weights

The linear matrix assigns 0.3333 to adjacent, 0.6667 to two-step and 1.0000 to maximum disagreements.

The resulting κ is 0.647398; observed weighted agreement is 90.40%.

Confirm that the displayed weight convention matches the formula.

R figure pair 2: Quadratic Weights and Weighted Contributions

Exact values

Quadratic Weights

The quadratic matrix assigns 0.1111, 0.4444 and 1.0000 to one-, two- and three-step errors.

The resulting κ is 0.765160; the higher value reflects the near absence of distant disagreements.

Cross-check against the verified matrix and the reported coefficient.

Exact values

Weighted Contributions

Observed quadratic contributions sum to 0.032357.

Low→medium and medium→high together contribute 0.022085, or 68.25% of observed quadratic disagreement.

Confirm that the displayed weight convention matches the formula.

R figure pair 3: Weighting Comparison and Result Summary

Exact values

Weighting Comparison

Linear κ = 0.647398; quadratic κ = 0.765160.

Quadratic κ exceeds linear κ by 0.117761 and unweighted κ by 0.221109.

Cross-check against the verified matrix and the reported coefficient.

Exact values

Result Summary

The summary reports the verified coefficients and disagreement terms.

The preferred conclusion is strong ordinal agreement with a separate upward-shift finding: 153 G3 increases versus 33 decreases.

Confirm that the displayed weight convention matches the formula.

Weighted Kappa in Python, R, SPSS, SAS and Excel

Cross-Software Reconciliation Checklist

If software values differ, check category ordering first, then whether the program uses agreement weights or disagreement weights, then whether missing values or zero-count categories were dropped. A reversed category on one axis can produce a radically different coefficient even when the same counts are present.

Expandable Weighted Kappa Code

import numpy as np

O = np.array([
    [88, 69,  0,  0],
    [12,267, 60,  1],
    [ 0, 19, 86, 23],
    [ 0,  0,  2, 22]
], dtype=float)

N = O.sum()
P = O / N
row = P.sum(axis=1)
col = P.sum(axis=0)
E = np.outer(row, col)

idx = np.arange(4)
distance = np.abs(idx[:, None] - idx[None, :])
W_linear = distance / 3
W_quadratic = (distance / 3) ** 2

def weighted_kappa(W):
    observed_disagreement = np.sum(W * P)
    expected_disagreement = np.sum(W * E)
    return 1 - observed_disagreement / expected_disagreement

print(weighted_kappa(W_linear))     # 0.647398407289
print(weighted_kappa(W_quadratic))  # 0.765159855031

from sklearn.metrics import cohen_kappa_score

g1 = []
g3 = []
for i in range(4):
    for j in range(4):
        g1.extend([i + 1] * int(O[i, j]))
        g3.extend([j + 1] * int(O[i, j]))

print(cohen_kappa_score(g1, g3, weights="linear"))
print(cohen_kappa_score(g1, g3, weights="quadratic"))

O <- matrix(c(
  88,69,0,0,
  12,267,60,1,
  0,19,86,23,
  0,0,2,22
), nrow=4, byrow=TRUE)

P <- O / sum(O)
E <- outer(rowSums(P), colSums(P))
d <- abs(outer(1:4, 1:4, "-"))
W.linear <- d / 3
W.quadratic <- (d / 3)^2

wk <- function(W) {
  Do <- sum(W * P)
  De <- sum(W * E)
  1 - Do / De
}

wk(W.linear)
wk(W.quadratic)

CROSSTABS
 /TABLES=grade1 BY grade3
 /FORMAT=AVALUE TABLES
 /STATISTICS=KAPPA
 /CELLS=COUNT ROW COLUMN
 /COUNT ROUND CELL.

=SUMPRODUCT(Observed_Proportions, Quadratic_Weights)

=SUMPRODUCT(Expected_Proportions, Quadratic_Weights)
=1-(Observed_Quadratic_Disagreement/Expected_Quadratic_Disagreement)

How to Interpret and Report Weighted Kappa

Report the weighting scheme, category structure, sample size, coefficient and enough table information to understand disagreement severity. Avoid labels such as “good” or “substantial” without the underlying numbers.

Data-Focused Interpretation

Reporting Hierarchy

How to Discuss the Upward Shift Without Undermining Agreement

A rigorous discussion should state that weighted agreement is strong and that G3 tends to be higher. The agreement coefficient summarises closeness after chance correction; the direction counts summarise change. The recommended wording is: “Ratings were usually identical or adjacent, but disagreements were asymmetric, with 153 upward and 33 downward transitions.” This avoids the misleading implication that high kappa means no systematic movement.

Interpretive Labels Are Secondary

Conventional labels such as moderate, substantial or strong vary across disciplines and should never replace the numerical evidence. In this example, the most defensible interpretation comes from the coefficient together with 71.34% exact agreement, 99.85% within-one-category agreement, and the complete absence of maximum-distance errors.

Common Weighted Kappa Mistakes and Corrections

Weighted Kappa Practice Questions

Weighted Kappa Reports and Worked Files

Related Salar Cafe Guides

Weighted Kappa Frequently Asked Questions

Weighted Kappa Conclusion

Weighted Kappa provides a precise summary of agreement when categories are ordered and disagreement severity matters. In this G1-versus-G3 analysis, 463 of 649 classifications match exactly, 185 differ by one category, one differs by two categories and none differ by three. Linear weighted κ is 0.647398, while quadratic weighted κ is 0.765160.

The quadratic result is higher because almost every error is adjacent. That finding should not hide the transition pattern: 153 classifications move upward from G1 to G3 and 33 move downward. The most complete conclusion is therefore strong distance-sensitive agreement with a clear net upward shift in G3 classifications.