Quick Answer: Holm Bonferroni Method Result

The worked example compares G3 final grade across four studytime groups. The sample contains 649 students. The one-way ANOVA was statistically significant, F(3, 645) = 15.876, p < .001, so pairwise follow-up tests were justified. The Holm Bonferroni Method was then applied to the six pairwise p-values to control family-wise error.

The group means increased from the lowest studytime group to the higher studytime groups. Group 1 had the lowest mean G3 score, M = 10.84. Group 2 had M = 12.09. Group 4 had M = 13.06. Group 3 had the highest mean, M = 13.23. After Holm adjustment, 4 pairwise comparisons remained significant out of 6 total comparisons.

What Is Holm Bonferroni Method?

Holm Bonferroni Method is a multiple comparison correction used when a researcher tests several hypotheses at the same time. Without adjustment, testing many pairwise p-values increases the chance of getting at least one false-positive result. Holm-Bonferroni controls this family-wise error problem while keeping more statistical power than the classic Bonferroni correction.

The method works by ordering p-values from smallest to largest. The smallest p-value is tested against the strictest threshold. If it passes, the next smallest p-value is tested against a slightly less strict threshold. This continues until a p-value fails. After the first failure, the remaining larger p-values are not rejected.

In post hoc analysis, Holm Bonferroni Method is commonly used after an ANOVA or after a planned set of pairwise t tests. It is flexible because it can adjust any list of p-values, whether they came from pairwise t tests, correlation tests, regression contrasts or other multiple testing situations.

Before using this method, review one-way ANOVA, p-values, null and alternative hypothesis, confidence intervals, and Type I and Type II error.

When to Use Holm Bonferroni Method

Use Holm Bonferroni Method when you have several p-values and you want to control the family-wise error rate. In this guide, the method is used after one-way ANOVA to adjust six pairwise comparisons among four studytime groups.

Holm Bonferroni Method Formula

Suppose there are m p-values in one family of tests. First sort the p-values from smallest to largest:

The Holm step-down threshold for the kth ordered p-value is:

The decision rule is:

For adjusted p-values, the Holm adjusted value is calculated in sorted order by multiplying each p-value by the number of remaining tests and applying a cumulative maximum:

Holm Step-Down Decisions in This Example

Null and Alternative Hypotheses for Holm Bonferroni Method

The Holm Bonferroni Method is applied to a family of individual hypotheses. Each pairwise comparison has a null and alternative hypothesis, and Holm adjusts the decision so the whole family of tests is protected.

Dataset and Variables Used

The worked example uses a student performance dataset. The dependent variable is G3 final grade. The grouping variable is studytime, coded into four weekly study-time groups. The analysis asks whether average final grade differs across studytime groups and which pairwise differences remain significant after Holm correction.

Before interpreting Holm-adjusted p-values, review the group means, distributions and variance context. Helpful related guides include descriptive statistics, box plot interpretation, Levene test, and ANOVA in SPSS.

SPSS Output Interpretation for Holm Bonferroni Method

The SPSS output confirms the ANOVA context and gives a Bonferroni comparison table for reference. SPSS one-way ANOVA menus do not always provide a direct Holm-Bonferroni post hoc option in the same way that Python and R can adjust p-values directly. Therefore, the correct workflow is to use SPSS for group descriptives, assumption context and ANOVA, then apply Holm adjustment to the pairwise p-values in Python, R or Excel.

SPSS Group Descriptives

SPSS Homogeneity of Variances

SPSS One-Way ANOVA Table

Holm-Adjusted Pairwise Results

Python Chart-by-Chart Interpretation

The Python charts show the full Holm Bonferroni workflow visually. They show group means, group distributions, unadjusted versus Holm-adjusted p-values, the step-down decision path, pairwise confidence intervals, a p-value heatmap and group standard deviation context.

Python Chart 1: Group Means with 95% Confidence Intervals

The group means chart shows that group 1 has the lowest mean G3 score, while groups 3 and 4 have the highest means. Group 2 sits between the lowest and highest categories. This visual pattern explains why the ANOVA was significant and why pairwise testing was useful.

The chart also shows that group 3 and group 4 have very close mean values. This explains why the Holm-adjusted comparison between groups 3 and 4 is not significant even though both groups are higher than group 1.

Python Chart 2: Group Distribution Boxplots

The boxplot shows the spread of G3 values inside each studytime group. Group 1 has the lowest central position and includes several very low scores. Groups 3 and 4 are centered higher, while group 2 is in the middle.

Holm-Bonferroni adjusts p-values, not distributions. The boxplot is therefore used as context. It helps readers understand why some pairs have strong evidence and why some high-studytime groups remain statistically similar.

Python Chart 3: Unadjusted vs Holm Adjusted p-values

This chart shows how Holm correction changes the pairwise p-values. The smallest p-values remain very small after adjustment, so the comparisons 1 vs 3, 1 vs 2, 1 vs 4, and 2 vs 3 remain significant.

The comparison 2 vs 4 becomes clearly non-significant after adjustment, and 3 vs 4 is already far from significance. This chart is useful because it shows both the original evidence and the family-wise-error-controlled result.

Python Chart 4: Holm Step-Down Decision Path

The step-down decision path shows the ordered p-values and the changing Holm thresholds. The first four p-values are smaller than their step-specific thresholds, so those four hypotheses are rejected. The fifth comparison, 2 vs 4, fails the Holm rule.

This chart is the core of the Holm Bonferroni Method. It shows why the method is not the same as applying one fixed Bonferroni threshold to every p-value. Holm starts strict and then becomes less strict as earlier tests are rejected.

Python Chart 5: Pairwise Mean Difference Confidence Intervals

The mean difference chart uses zero as the no-difference reference line. Comparisons whose intervals stay away from zero are the strongest pairwise differences. In this example, the major differences involve group 1 compared with groups 2, 3 and 4, plus group 2 compared with group 3.

The intervals for 2 vs 4 and 3 vs 4 are consistent with non-significant Holm-adjusted decisions. These comparisons do not show a clear enough difference after accounting for multiple testing.

Python Chart 6: Holm Adjusted p-value Heatmap

The heatmap gives a compact matrix view of Holm-adjusted pairwise decisions. It highlights that group 1 differs from groups 2, 3 and 4, and that group 2 differs from group 3.

The heatmap also makes the non-significant comparisons easy to see. Groups 2 and 4 do not differ clearly, and groups 3 and 4 are almost the same in mean G3 after adjustment.

Python Chart 7: Group Standard Deviation Context

The standard deviation chart shows that group spread differs somewhat across the studytime groups. Group 3 has the smallest standard deviation, while groups 1 and 2 have larger standard deviations.

This chart does not change the Holm adjustment itself, but it improves interpretation. Pairwise tests depend on mean differences relative to uncertainty, so standard deviation context helps explain why some mean differences are significant and others are not.

R Chart-by-Chart Validation

The R validation charts confirm the Python result. The same group mean pattern appears, the same adjusted p-value decisions are reached, and the same four comparisons remain significant after Holm correction.

R Chart 1: Group Means with 95% Confidence Intervals

The R group means chart validates the same mean order: group 1 is lowest, group 2 is higher, and groups 3 and 4 are highest. This agreement supports the interpretation that the group mean pattern is stable across software.

The chart also reinforces that group 3 and group 4 are close together. This explains the non-significant adjusted result for the 3 vs 4 comparison.

R Chart 2: Group Distribution Boxplots

The R boxplot confirms the distribution pattern seen in Python. Group 1 is lower, groups 3 and 4 are higher, and group 2 sits in between.

This visual validation matters because Holm-Bonferroni decisions should be interpreted alongside the actual data structure. The plot shows that the significant p-values correspond to visible mean separation.

R Chart 3: Unadjusted vs Holm Adjusted p-values

The R p-value chart confirms that the smallest four p-values remain below .05 after Holm adjustment. These are the comparisons involving group 1 and the comparison between group 2 and group 3.

The non-significant comparisons remain above the threshold after adjustment. This confirms that the result is not dependent on one software implementation.

R Chart 4: Holm Step-Down Decision Path

The R step-down chart validates the Holm procedure visually. The first four ordered p-values pass their Holm thresholds, while the fifth comparison does not. After the first failure, the remaining larger p-values are not rejected.

This chart is useful for teaching because it shows the exact reason the method is called a step-down procedure. It tests the strongest evidence first and stops rejecting when the evidence is no longer strong enough.

R Chart 5: Pairwise Mean Difference Confidence Intervals

The R interval chart confirms the same direction of differences. Group 1 is lower than groups 2, 3 and 4, and group 2 is lower than group 3.

The comparisons between groups 2 and 4 and between groups 3 and 4 do not show a clear adjusted difference. This supports the final post hoc interpretation.

R Chart 6: Holm Adjusted p-value Heatmap

The R heatmap summarizes the adjusted p-values in a matrix format. It confirms that group 1 is the main separated group and that group 2 differs from group 3.

The non-significant cells for 2 vs 4 and 3 vs 4 support a careful report: the higher studytime groups are not all statistically different from one another.

R Chart 7: Group Standard Deviation Context

The R standard deviation chart confirms the spread pattern across groups. Group 3 is relatively tighter, while groups 1 and 2 show larger spread.

This validates the Python context chart and supports the overall interpretation that mean differences should be read together with uncertainty, not as raw differences alone.

SPSS, R, Python and Excel Workflows for Holm Bonferroni Method

The Holm Bonferroni Method can be reproduced in SPSS, R, Python and Excel. SPSS is useful for ANOVA and pairwise comparison context. R and Python can directly produce Holm adjusted p-values. Excel is useful for teaching the step-down calculation manually.

SPSS Workflow

R Workflow

Python Workflow

Excel Workflow

Excel can reproduce the Holm Bonferroni Method manually. Put the comparison names in one column, unadjusted p-values in another column, sort p-values from smallest to largest, calculate the Holm threshold, and calculate Holm adjusted p-values.

Code Blocks for Holm Bonferroni Method

SPSS Syntax for ANOVA and Reference Pairwise Output

ONEWAY G3 BY studytime
  /STATISTICS DESCRIPTIVES HOMOGENEITY
  /POSTHOC = BONFERRONI ALPHA(0.05).

R Code

data <- read.csv("dataset.csv")
data$studytime <- factor(data$studytime)

# One-way ANOVA context

model <- aov(G3 ~ studytime, data = data)
summary(model)

# Holm-Bonferroni adjusted pairwise t tests

holm_result <- pairwise.t.test(
x = data$G3,
g = data$studytime,
p.adjust.method = "holm",
pool.sd = TRUE
)

print(holm_result)

Python Code

import pandas as pd
import itertools
from scipy import stats
import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats.multitest import multipletests

df = pd.read_csv("dataset.csv")
df["studytime"] = df["studytime"].astype("category")

# ANOVA context

model = smf.ols("G3 ~ C(studytime)", data=df).fit()
anova = sm.stats.anova_lm(model, typ=2)

mse = anova.loc["Residual", "sum_sq"] / anova.loc["Residual", "df"]
df_error = anova.loc["Residual", "df"]

summary = df.groupby("studytime")["G3"].agg(["count", "mean", "std", "var"])

rows = []
for g1, g2 in itertools.combinations(summary.index, 2):
n1 = summary.loc[g1, "count"]
n2 = summary.loc[g2, "count"]
m1 = summary.loc[g1, "mean"]
m2 = summary.loc[g2, "mean"]

```
diff = m1 - m2
se = (mse * (1/n1 + 1/n2)) ** 0.5
t_value = diff / se
p_unadjusted = 2 * stats.t.sf(abs(t_value), df_error)

rows.append([g1, g2, diff, se, t_value, p_unadjusted])
```

pairwise = pd.DataFrame(rows, columns=[
"group_1", "group_2", "mean_difference",
"standard_error", "t_value", "p_unadjusted"
])

reject, p_holm, _, _ = multipletests(
pairwise["p_unadjusted"],
alpha=0.05,
method="holm"
)

pairwise["p_holm_adjusted"] = p_holm
pairwise["holm_significant"] = reject
pairwise["decision"] = pairwise["holm_significant"].map(
{True: "Reject equal means", False: "Fail to reject equal means"}
)

print(anova)
print(summary)
print(pairwise.sort_values("p_unadjusted"))

Excel Formula Pattern

Step 1: Sort unadjusted p-values from smallest to largest.

Number of tests:
m = COUNT(p_value_range)

Holm threshold:
=0.05/(m-rank+1)

Raw Holm adjusted p-value:
=(m-rank+1)*p_value

Decision:
=IF(p_value<=holm_threshold,"Reject equal means","Fail to reject equal means")

Important:
After the first failure in the sorted list, all remaining larger p-values are treated as not significant.

APA Reporting Wording for Holm Bonferroni Method

A one-way ANOVA was conducted to compare G3 final grade across four studytime groups. The ANOVA was statistically significant, F(3, 645) = 15.876, p < .001, indicating that mean final grade differed across studytime groups. Levene's test based on the mean was not significant, p = .400, supporting the homogeneity of variance assumption for this example.

Pairwise comparisons were adjusted using the Holm Bonferroni Method. Holm-adjusted comparisons showed that group 1 had significantly lower G3 scores than group 2, group 3 and group 4. Group 2 also had significantly lower G3 scores than group 3. The comparisons between group 2 and group 4 and between group 3 and group 4 were not statistically significant after Holm adjustment.

Common Mistakes in Holm Bonferroni Method

Downloads and Resources

Use the following downloadable outputs to verify the Holm Bonferroni Method result and compare the SPSS, Python and R workflows.

Related Guides

Continue with these related Salar Cafe guides to understand ANOVA, p-values, multiple testing, assumptions and reporting.