What Is Cohen’s Kappa?

Cohen’s Kappa, usually written as κ, is a chance-corrected coefficient of agreement for two raters, two observers, two diagnostic methods, two machine-learning classifiers or two classification occasions that assign the same cases to the same set of categorical outcomes. It belongs to the broader family of categorical-data methods discussed in cross-tabulation, frequency and relative frequency tables and categorical data analysis in Python.

Raw percent agreement answers a simple question: what proportion of paired classifications match exactly? Cohen’s Kappa asks a more demanding question: how much of that observed agreement remains after subtracting the agreement that would arise from the two marginal category distributions? This distinction matters whenever one or more categories are common, because high raw agreement can occur even when the raters or methods add little information beyond category prevalence.

What does Cohen’s Kappa measure?

Cohen’s Kappa measures the proportion of the maximum possible agreement beyond chance that was actually achieved. A kappa of 1 means perfect agreement. A kappa of 0 means observed agreement is exactly what the marginal distributions predict under chance independence. A negative value means observed agreement is worse than chance expectation. Because the coefficient is based on a two-way table, it should always be interpreted beside the agreement matrix, marginal proportions and category-specific patterns.

Worked example in this guide

The current analysis classifies 649 students into four grade bands twice: once from G1 and once from G2. The band rules are 1 = score below 10, 2 = score 10–13, 3 = score 14–16 and 4 = score 17–19. The two classification occasions are compared as nominal categories. Although the bands have a natural order, the verified primary analysis is unweighted Cohen’s Kappa, so every off-diagonal disagreement is treated equally. Weighted kappa is discussed later as a separate ordinal extension.

The null states that agreement beyond chance is zero. The alternative states that agreement differs from chance. For hypothesis wording and decision rules, see null and alternative hypothesis, p-value and type I and type II error.

Quick Answer: Cohen’s Kappa Result

The verified Excel, Python, R and SPSS outputs report the same nominal four-category agreement result.

Table of Contents

1. Research question, design and hypotheses
2. When to use Cohen’s Kappa
3. Applications of Cohen’s Kappa
4. Cohen’s Kappa formula and hand calculation
5. Variables and grade-band coding
6. Observed agreement matrix
7. Complete Cohen’s Kappa results
8. Cohen’s Kappa interpretation
9. Kappa versus percent agreement
10. Weighted kappa, Fleiss kappa and alternatives
11. Prevalence, bias and negative kappa
12. Cohen’s Kappa calculator steps
13. Python chart interpretations
14. R chart pairs and explanations
15. Python, R, SPSS, Excel, SAS and Stata
16. Expandable software code
17. Assumptions and limitations
18. APA reporting
19. Worked practice questions
20. Reports and Excel download
21. Related Salar Cafe guides
22. Frequently asked questions
23. Conclusion

Research Question, Design and Hypotheses

Research question

How strongly do G1 and G2 agree when both scores are converted into the same four grade-band categories, after correcting raw agreement for chance agreement implied by the marginal category frequencies?

Null and alternative hypotheses

Why this is an agreement question, not a correlation question

Correlation evaluates association or rank/linear co-movement; agreement asks whether paired classifications are the same. Two methods can correlate highly while disagreeing systematically. Conversely, categories can agree frequently without a meaningful linear relationship. See correlation versus regression, Pearson correlation and Spearman rank correlation for association-focused alternatives.

When to Use Cohen’s Kappa

Binary versus multicategory Cohen’s Kappa

Cohen’s Kappa works with two categories or more than two categories. With binary data, the table is 2 × 2 and terms such as positive agreement, negative agreement, sensitivity and specificity may also matter. With multicategory data, the table is square with one row and one column per category. The current 4 × 4 analysis demonstrates the multicategory case.

Cohen’s Kappa in machine learning

In machine learning, Cohen’s Kappa can compare predicted classes with reference classes or compare two classifiers on the same observations. It is not identical to accuracy because it subtracts marginal chance agreement. However, kappa still depends on class prevalence and predicted marginals, so it should be interpreted with the confusion matrix and class-specific performance rather than used as the only model metric.

Applications of Cohen’s Kappa

Cohen’s Kappa in communication and content analysis

Communication researchers often use Cohen’s Kappa to quantify inter-coder reliability. Two coders independently classify the same messages, documents or media units into a common category system. The coefficient is meaningful only when the coding manual defines mutually exclusive categories and the coders evaluate the same units independently. A training phase can be used to refine definitions, but the final reliability sample should be coded without consultation.

Cohen’s Kappa in healthcare

Healthcare applications include agreement between radiologists, diagnostic tests, triage systems or categorical clinical ratings. High stakes require more than a generic label such as “substantial.” Report category prevalence, false-negative and false-positive patterns, confidence intervals and any clinically important disagreement. Agreement between two methods does not establish that either method is valid against a gold standard.

Cohen’s Kappa in machine learning versus accuracy and AUC

Accuracy is raw agreement between predictions and reference labels. Cohen’s Kappa adjusts that agreement for the prediction and reference marginals. AUC answers a different question: how well a scoring model ranks positive cases above negative cases across thresholds. Kappa is therefore a threshold- and category-dependent agreement metric, whereas AUC is a discrimination metric. Neither should replace the confusion matrix, class prevalence and application-specific error costs.

Cohen’s Kappa in education

In the present example, G1 and G2 are not two human raters; they are two classification occasions applied to the same students. Cohen’s Kappa is still applicable because the mathematical requirement is two paired categorical classifications using the same category set. This broader framing is useful when studying stability, repeatability or method agreement rather than only inter-rater reliability.

Cohen’s Kappa Formula and Hand Calculation

The formula compares observed exact agreement with expected chance agreement:

Step 1: Add the diagonal agreements

Step 2: Calculate marginal proportions

For G1, the row totals are 157, 340, 128 and 24. For G2, the column totals are 145, 352, 117 and 35. Divide each total by 649 to obtain marginal proportions.

Step 3: Sum expected chance-agreement components

Equivalently, the expected diagonal counts are approximately 35.077, 184.407, 23.076 and 1.294, totalling 243.854 expected chance agreements.

Step 4: Substitute into the kappa formula

Step 5: Test whether kappa differs from zero

The verified output gives z = 23.1776 and a two-sided p-value of 7.6555 × 10⁻¹¹⁹. The implied standard error is approximately 0.02632, producing a normal-approximation 95% confidence interval of about 0.5584 to 0.6616. For interval concepts, see standard error, confidence interval formula and margin of error and confidence level.

Variables and Grade-Band Coding

Why identical categories are required

Both classification occasions must use the same category definitions. If G1 used different cut points from G2, diagonal cells would no longer represent the same substantive category. Category harmonization is therefore a design requirement, not a cosmetic data-cleaning step. See categorical and quantitative variables and descriptive statistics for coding foundations.

Nominal treatment despite ordinal bands

The four grade bands are ordered, but the primary workbook deliberately treats them as nominal. Therefore, a disagreement from Band 1 to Band 2 receives the same penalty as a disagreement from Band 1 to Band 4. This is appropriate for an unweighted kappa demonstration and provides a conservative, transparent exact-match analysis. If severity of disagreement matters, weighted kappa should be prespecified and reported separately.

Observed Cohen’s Kappa Agreement Matrix

Diagonal and off-diagonal meaning

The diagonal cells 107, 274, 88 and 22 are exact agreements. The 158 off-diagonal cases are disagreements. Most disagreements are adjacent: 50 move from G1 Band 1 to G2 Band 2, 38 move from G1 Band 2 to G2 Band 1, 27 move from Band 2 to Band 3, 28 move from Band 3 to Band 2, 12 move from Band 3 to Band 4 and 2 move from Band 4 to Band 3. Only one case moves from Band 2 to Band 4, and no cases make the largest possible Band 1↔Band 4 jump.

Why adjacent disagreement matters

Unweighted Cohen’s Kappa correctly counts every off-diagonal cell as disagreement, but it does not distinguish small and large errors. The concentration of disagreements near the diagonal indicates that a weighted analysis could produce a higher coefficient because one-band differences would receive less penalty than distant differences. That does not invalidate the unweighted result; it answers a different question.

Complete Cohen’s Kappa Results

Statistical interpretation

The p-value is far below .05, .01 and .001, so the null of zero chance-corrected agreement is rejected. Statistical significance is unsurprising with 649 paired cases and κ above .60. The more useful questions are how large the agreement is, where disagreements occur and whether unweighted exact agreement matches the practical purpose.

Practical interpretation

The G1 and G2 band classifications match exactly for roughly three of every four students. After removing the 37.57% chance agreement predicted by the marginal category frequencies, the coefficient is .610. This indicates useful stability between classification occasions but leaves meaningful room for change or disagreement. Because many disagreements are only one band apart, users should not describe the two classifications as unreliable solely because 24.35% are off the diagonal.

Cohen’s Kappa Interpretation

There is no universal interpretation scale that overrides context. Common descriptive labels can help readers orient themselves, but they are conventions rather than laws. A coefficient acceptable for exploratory coding may be too low for a high-stakes diagnosis, legal decision or safety-critical classification.

The worked κ = 0.6100 lies almost exactly at the boundary between commonly labelled moderate and substantial agreement. It is better to report the numeric estimate, confidence interval, raw agreement and context than to argue over a boundary label. For broader magnitude reasoning, compare effect size, Cramer’s V, phi coefficient and contingency coefficient.

What is a good Cohen’s Kappa value?

A good value is one that meets the consequences and reliability requirements of the application. In a low-risk exploratory study, κ around .60 may be useful. In a clinical, safety or certification decision, κ around .60 may be insufficient. The minimum acceptable value should be defined before analysis whenever possible, supported by subject-matter standards and accompanied by a confidence interval.

Do not confuse Cohen’s Kappa with Cohen’s d

Cohen’s Kappa is an agreement coefficient for categorical classifications. Cohen’s d is a standardized mean difference for quantitative outcomes. They answer different questions, use different formulas and are not interchangeable. For quantitative group comparisons, see two-sample t test and effect size.

Cohen’s Kappa Versus Percent Agreement

Why 75.65% agreement becomes κ = .610

The raw agreement of 75.65% looks high, but the marginal distributions are concentrated in Band 2. Even unrelated classifications with the same marginals would agree about 37.57% of the time. Kappa subtracts this expected component and scales the remaining agreement against the maximum possible agreement beyond chance.

When percent agreement still matters

Kappa should not replace percent agreement. Raw agreement is directly interpretable and helps reveal whether a statistically respectable kappa is built from a practically acceptable match rate. Report both. For binary outcomes, positive and negative agreement may also be more informative than a single overall percentage.

Weighted Kappa, Fleiss Kappa and Other Alternatives

Why weighted kappa is relevant here

The grade bands are ordinal, and almost all disagreements are adjacent. A one-band disagreement is arguably less serious than a three-band disagreement. Weighted kappa can incorporate that reasoning through linear or quadratic weights. However, the weight system must be justified before viewing results because changing weights changes the coefficient.

Why the primary result remains unweighted

The uploaded workbook explicitly defines the categories as nominal for this analysis. The public post therefore preserves the verified unweighted κ = .6100 rather than replacing it with a different statistic. Weighted kappa belongs in a distinct sensitivity or extension section, not as an undocumented substitution.

Kappa versus ICC

If G1 and G2 are analysed as original quantitative scores rather than grade bands, the intraclass correlation coefficient guide is more directly relevant. Kappa is appropriate for the derived categorical classifications; ICC is appropriate for quantitative measurement agreement under a chosen ICC model.

Prevalence, Bias and Negative Cohen’s Kappa

The prevalence effect

When one category dominates, raw agreement may be high because both classifiers frequently choose that category. Expected chance agreement also becomes high, which can lower kappa. This is sometimes called a kappa prevalence paradox. It is not a computational mistake; it is a consequence of the marginal-based chance model.

The bias effect

If the two classifiers use categories at different rates, the marginal profiles differ. Depending on the table, marginal bias can increase or decrease kappa. In the worked example, the G1 and G2 marginal profiles are similar but not identical: Band 1 is 24.19% versus 22.34%, Band 2 is 52.39% versus 54.24%, Band 3 is 19.72% versus 18.03% and Band 4 is 3.70% versus 5.39%.

Can Cohen’s Kappa be negative?

Yes. κ becomes negative when observed agreement is less than expected agreement. A negative coefficient can result from systematic reversal, inconsistent category definitions, data-pairing errors or genuinely antagonistic classification. Before interpreting a negative result substantively, audit coding, case alignment and category order.

Can Cohen’s Kappa exceed 1?

No. The theoretical upper limit is 1. The lower limit can be below −1 only under malformed calculation; valid kappa values are bounded above by 1 and have a table-dependent minimum that is often greater than −1. Software returning a value outside the feasible range signals an implementation or data problem.

Cohen’s Kappa Calculator: Step-by-Step Workflow

A reliable Cohen’s Kappa calculator must accept the full square agreement table or paired category columns, not only the total number of agreements. Chance agreement cannot be computed without marginal category totals.

Calculator inputs for the worked example

Calculator validation targets

• Diagonal agreement count should equal 491.
• Observed agreement should equal 0.7565485.
• Expected agreement should equal 0.3757375.
• Unweighted Cohen’s Kappa should equal 0.6100175.
• The result should not change if both raters’ category order is rearranged identically.
• The result may change if only one rater’s category order or coding is changed.

Excel calculator logic

In Excel, use the sum of diagonal cells divided by the grand total for P_o. Multiply each row proportion by its corresponding column proportion and sum those products for P_e. Then apply the kappa formula. The supplied worked workbook already includes these calculations and formula checks.

Six Python Cohen’s Kappa Chart Interpretations

The Python workflow visualizes the agreement matrix, expected chance counts, category agreement, marginal distributions, agreement components and final result. These charts support the numeric analysis rather than replacing it.

Chart 1: Agreement MatrixPython

Pattern

The largest cell is the Band 2 exact agreement, and most off-diagonal counts lie immediately beside the diagonal.

Key Values

Exact matches are 107, 274, 88 and 22; total exact agreement is 491. The largest disagreements are 50 for Band 1→2 and 38 for Band 2→1.

Interpretation

Agreement is strong overall, while disagreements are mostly small shifts between neighbouring grade bands.

Why It Matters

The matrix reveals where kappa comes from and shows why an ordinal weighted extension could be informative.

Next step: Confirm the category definitions and read every diagonal and major off-diagonal cell before interpreting κ.

Chart 2: Chance-Expected Agreement CountsPython

Pattern

Band 2 contributes most of the chance agreement because it is the dominant category for both classifications.

Key Values

Expected counts are about 35.08, 184.41, 23.08 and 1.29, totalling 243.85 expected agreements.

Interpretation

Observed diagonal counts exceed chance counts in every category, especially Bands 1, 2 and 3.

Why It Matters

Kappa depends on expected agreement, so dominant categories can strongly affect the coefficient even when raw agreement is high.

Next step: Report P_e and marginal distributions beside P_o.

Chart 3: Category-Specific Exact AgreementPython

Pattern

Band 2 contributes the largest number of exact matches because more than half of observations are classified in that band.

Key Values

Diagonal counts are 107, 274, 88 and 22; their shares of the full sample are 16.49%, 42.22%, 13.56% and 3.39%.

Interpretation

A large category can dominate total agreement, so category-specific interpretation prevents the overall coefficient from hiding uneven performance.

Why It Matters

Users may need separate agreement diagnostics for rare and common categories, particularly when rare categories are important.

Next step: Compare diagonal counts with row and column prevalence rather than comparing counts alone.

Chart 4: Marginal DistributionsPython

Pattern

The two marginal profiles are similar, with Band 2 dominant and Band 4 rare, but G2 uses Band 4 somewhat more often.

Key Values

G1 proportions are 24.19%, 52.39%, 19.72% and 3.70%; G2 proportions are 22.34%, 54.24%, 18.03% and 5.39%.

Interpretation

Similar marginals support a straightforward agreement interpretation, while small differences explain directional off-diagonal flows.

Why It Matters

Marginal distributions determine P_e and can create prevalence or bias effects in kappa.

Next step: Inspect whether one classifier systematically favours particular categories.

Chart 5: Observed and Expected Agreement ComponentsPython

Pattern

Observed agreement is roughly double expected chance agreement, leaving a substantial positive gap.

Key Values

P_o = 0.75655, P_e = 0.37574, P_o − P_e = 0.38081 and 1 − P_e = 0.62426.

Interpretation

Kappa scales the 0.38081 observed advantage by the 0.62426 maximum available advantage.

Why It Matters

This chart makes the formula intuitive and explains why κ is lower than raw percent agreement.

Next step: Use the component chart when teaching or auditing a calculator result.

Chart 6: Cohen’s Kappa Result SummaryPython

Pattern

The coefficient is positive and far from zero, with a highly significant test result.

Key Values

κ = 0.6100, observed agreement = 75.65%, expected agreement = 37.57%, z = 23.18 and p ≈ 7.66 × 10⁻119.

Interpretation

The two classifications show useful agreement beyond chance but are not perfectly interchangeable.

Why It Matters

A compact result chart is useful for presentation, but the matrix and marginal charts remain necessary for diagnosis.

Next step: Pair the summary with an APA sentence and the exact 4 × 4 table.

R Cohen’s Kappa Chart Pairs

The R workflow reproduces the same agreement structure in three paired rows. Each pair combines a visual diagnostic with a dedicated explanation so the reader can move from counts to marginals and then to the final coefficient.

R pair 1: Agreement matrix and expected agreement

Pattern

Exact agreement dominates the diagonal

The matrix contains 491 diagonal cases and 158 disagreements. The largest diagonal cell is Band 2 with 274 matches, reflecting both genuine stability and high prevalence.

Matrix check: 107 + 274 + 88 + 22 = 491.

Meaning

Chance expectation is concentrated in Band 2

Band 2 alone contributes about 184.41 expected agreements because both marginals assign more than half the cases to that category. Expected agreement across all categories is 243.85 cases or 37.57%.

Kappa corrects the observed diagonal for this marginal concentration.

R pair 2: Diagonal agreement and marginal profiles

Pattern

Agreement counts reflect prevalence

Band 2 produces the most exact matches, while Band 4 produces the fewest. This does not mean Band 4 is necessarily less reliable; it is also much rarer.

Diagonal sample shares: 16.49%, 42.22%, 13.56% and 3.39%.

Meaning

Marginals are similar but not identical

G2 assigns slightly fewer cases to Bands 1 and 3 and slightly more to Bands 2 and 4. These shifts contribute to directional disagreement and the chance-agreement calculation.

Largest marginal difference is Band 4: 3.70% for G1 versus 5.39% for G2.

R pair 3: Observed versus expected agreement and result summary

Pattern

Observed agreement substantially exceeds chance

The 75.65% observed agreement is more than twice the 37.57% expected agreement. The difference of 38.08 percentage points supplies the numerator of kappa.

Formula check: (.75655 − .37574)/(1 − .37574) = .61002.

Meaning

The final decision is decisive

The result summary reports κ = .6100 and an extremely small p-value. Agreement is clearly above chance, but the coefficient remains below 1 because 158 pairs disagree.

APA core: κ = .61, 95% CI [.56, .66], p < .001.

Cohen’s Kappa in Python, R, SPSS, Excel, SAS and Stata

Expandable Cohen’s Kappa Code

import pandas as pd
from sklearn.metrics import cohen_kappa_score, confusion_matrix

# df contains original G1 and G2 scores.
bins = [-float("inf"), 9, 13, 16, float("inf")]
labels = [1, 2, 3, 4]
df["G1_band"] = pd.cut(df["G1"], bins=bins, labels=labels)
df["G2_band"] = pd.cut(df["G2"], bins=bins, labels=labels)

matrix = confusion_matrix(df["G1_band"], df["G2_band"], labels=labels)
kappa = cohen_kappa_score(df["G1_band"], df["G2_band"])
print(matrix)
print(kappa)

breaks <- c(-Inf, 9, 13, 16, Inf)
labels <- c("1", "2", "3", "4")
df$G1_band <- cut(df$G1, breaks = breaks, labels = labels)
df$G2_band <- cut(df$G2, breaks = breaks, labels = labels)

tab <- table(df$G1_band, df$G2_band)
# Use a validated agreement package/function for unweighted kappa.
print(tab)

* Derive common grade bands.
RECODE G1 G2 (Lowest thru 9=1) (10 thru 13=2) (14 thru 16=3) (17 thru Highest=4)
 INTO G1_band G2_band.

CROSSTABS
 /TABLES=G1_band BY G2_band
 /STATISTICS=KAPPA
 /CELLS=COUNT ROW COLUMN.

Observed agreement:
=SUM(diagonal_cells)/grand_total

Expected agreement:
=SUMPRODUCT(row_totals/grand_total,column_totals/grand_total)

Cohen's kappa:
=(observed_agreement-expected_agreement)/(1-expected_agreement)

Cohen’s Kappa Assumptions and Limitations

Independence of cases

The two ratings within a student are deliberately dependent, but different students should be independent. If students are clustered within schools, classrooms or raters, standard asymptotic uncertainty may understate dependence. The agreement coefficient may still be descriptively useful, but inference may need cluster-aware resampling or modelling.

Missing classifications

Complete-case kappa uses only cases with both classifications observed. If missingness is related to category or disagreement, complete-case results can be biased. Report the number of excluded pairs and compare marginal distributions before and after exclusion. For sampling concerns, see sampling methods and sampling bias.

Do not infer validity from agreement

Two classifiers can agree with each other and both be wrong relative to a gold standard. Cohen’s Kappa evaluates reliability or concordance, not criterion validity. When a reference standard exists, examine accuracy, sensitivity, specificity and class-specific errors in addition to inter-method agreement.

APA Reporting for Cohen’s Kappa

Methods wording

What to include in a complete report

• Names and roles of the two classifications.
• Category definitions and whether they are nominal or ordered.
• Sample size and missing-pair handling.
• Raw agreement and expected agreement.
• Kappa estimate, confidence interval and p-value.
• Weighting method if weighted kappa is used.
• Major disagreement patterns and practical implications.

For general statistical reporting, see hypothesis testing, p-value, significance level and test statistic and confidence intervals.

Cohen’s Kappa Worked Practice Questions

Practice 1: Binary diagnostic agreement

Two diagnostic methods classify 200 cases as positive or negative. They agree on 150 cases. Method A labels 80 positive, and Method B labels 70 positive. To calculate kappa, first build the complete 2 × 2 table or obtain the four cell counts. The total agreement count alone is insufficient because expected agreement depends on both marginal positive and negative totals.

Practice 2: Perfect agreement

If every case lies on the diagonal, P_o = 1. Provided P_e is less than 1, κ = (1 − P_e)/(1 − P_e) = 1. Perfect kappa means every paired classification matches.

Practice 3: Chance-level agreement

If P_o = P_e = .40, κ = 0. Agreement is present in raw terms, but no agreement remains after chance correction.

Practice 4: Negative kappa

If P_o = .20 and P_e = .35, κ = (.20 − .35)/(1 − .35) = −.2308. The classifications agree less often than their marginals predict by chance.

Practice 5: Weighted versus unweighted decision

Two teachers classify essays as poor, fair, good or excellent. Because the categories are ordered and poor-versus-fair disagreement is less severe than poor-versus-excellent disagreement, weighted kappa is preferable when the distance structure is substantively justified.

Practice 6: Kappa versus chi-square

A significant chi-square association between two classifications does not prove good agreement. An association can be strong even when one classifier consistently shifts every category upward. Use kappa for exact agreement and chi-square for association.

Cohen’s Kappa Reports and Worked Excel Download

Related Salar Cafe Guides

Conclusion

Cohen’s Kappa is the appropriate chance-corrected agreement statistic when two classifications assign the same cases to the same categorical outcomes. In the worked analysis, G1 and G2 grade-band classifications agreed exactly for 491 of 649 students. Observed agreement was 75.65%, expected chance agreement was 37.57%, and unweighted Cohen’s Kappa was .6100, with an approximate 95% confidence interval from .5584 to .6616.

The result demonstrates useful agreement beyond chance while also showing why one coefficient is not enough. The agreement matrix reveals that most errors are adjacent-band shifts, the marginal profiles explain expected agreement, and the distinction between unweighted and weighted kappa determines how disagreement severity is treated. A complete report should therefore present Cohen’s Kappa with raw agreement, expected agreement, confidence intervals, category definitions and the full table.

Continue with cross-tabulation, chi-square test of independence, intraclass correlation coefficient and correlation assumptions to compare agreement with related categorical and quantitative methods.