Name: Open Statistical Calculator Verification Benchmark 2026
Creator: Salar Cafe
Published: 2026-07-26
License: https://creativecommons.org/licenses/by/4.0/

Open research dataset · Version 1.0.0

Open Statistical Calculator Verification Benchmark 2026

An open, versioned benchmark for checking statistical calculator accuracy, method transparency, boundary behavior, and invalid-input handling.

Download benchmark Browse 120 cases Cite OSCVB

120test cases

12method families

96numeric cases

24invalid-input cases

120/120dual-reference passes

Quick answer

What does OSCVB verify?

OSCVB tests whether a statistical calculator produces the correct result, discloses the method it uses, preserves enough precision, and refuses inputs for which the requested statistic is invalid or undefined.

Standard worked examples
Boundary and perfect-association cases
Large-offset, tiny-scale, sparse, and extreme-tail stress cases
Deliberately invalid inputs that must not return plausible-looking answers

Central research claim

Ordinary examples are not enough

A calculator is not verified merely because a familiar classroom example works. Reliable tools must also disclose hidden defaults, distinguish valid boundary values from undefined results, and handle unsafe inputs explicitly.

CC BY 4.0 dataMIT codeVersioned releaseSHA-256 hashes

Coverage

Twelve method families

Every family contains eight valid cases and two deliberately invalid cases.

Independent reference paths

Each valid case is calculated with a direct formula or exact enumeration and independently checked with NumPy, SciPy, or Statsmodels.

Tolerance-aware comparison

Results are compared using explicit numerical tolerances so harmless floating-point representation differences are not mistaken for statistical errors.

Unsafe input rejection

Invalid cases pass only when the tool refuses to calculate or clearly states that the requested result is undefined.

Open data

Interactive benchmark case browser

Search, filter, inspect exact inputs, and expand the dual-reference expected result.

Search cases Method family Case class

Case	Method	Class	Purpose and evidence
OSCVB-001typical	Descriptive Statistics	Standard	Basic repeated integer data View input and expected result Input JSON { "data": [ 1, 2, 2, 4, 5 ] } Reference result { "n": 5, "sum": 14, "mean": 2.8, "median": 2, "min": 1, "max": 5, "range": 4, "population_variance": 2.16, "population_sd": 1.469693845669907, "sample_variance": 2.7, "sample_sd": 1.6431676725154984, "modes": [ 2 ] } Numeric result generated independently by two reference paths.
OSCVB-002decimals	Descriptive Statistics	Standard	Decimal values View input and expected result Input JSON { "data": [ 1.1, 2.2, 3.3, 4.4 ] } Reference result { "n": 4, "sum": 11, "mean": 2.75, "median": 2.75, "min": 1.1, "max": 4.4, "range": 3.3000000000000003, "population_variance": 1.5125000000000002, "population_sd": 1.2298373876248845, "sample_variance": 2.016666666666667, "sample_sd": 1.4200938936093863, "modes": [ 1.1, 2.2, 3.3, 4.4 ] } Numeric result generated independently by two reference paths.
OSCVB-003negative mixed	Descriptive Statistics	Standard	Negative and positive values View input and expected result Input JSON { "data": [ -5, -1, 0, 2, 8 ] } Reference result { "n": 5, "sum": 4, "mean": 0.8, "median": 0, "min": -5, "max": 8, "range": 13, "population_variance": 18.160000000000004, "population_sd": 4.261455150532504, "sample_variance": 22.700000000000003, "sample_sd": 4.764451699828639, "modes": [ -5, -1, 0, 2, 8 ] } Numeric result generated independently by two reference paths.
OSCVB-004single value	Descriptive Statistics	Boundary	Boundary n=1; sample variance intentionally undefined View input and expected result Input JSON { "data": [ 7 ] } Reference result { "n": 1, "sum": 7, "mean": 7, "median": 7, "min": 7, "max": 7, "range": 0, "population_variance": 0, "population_sd": 0, "sample_variance": null, "sample_sd": null, "modes": [ 7 ] } Numeric result generated independently by two reference paths.
OSCVB-005multimodal	Descriptive Statistics	Standard	Multiple modes View input and expected result Input JSON { "data": [ 1, 1, 2, 2, 3 ] } Reference result { "n": 5, "sum": 9, "mean": 1.8, "median": 2, "min": 1, "max": 3, "range": 2, "population_variance": 0.56, "population_sd": 0.7483314773547883, "sample_variance": 0.7000000000000001, "sample_sd": 0.8366600265340756, "modes": [ 1, 2 ] } Numeric result generated independently by two reference paths.
OSCVB-006large offset	Descriptive Statistics	Stress	Large location with small spread View input and expected result Input JSON { "data": [ 1000000001, 1000000002, 1000000003 ] } Reference result { "n": 3, "sum": 3000000006, "mean": 1000000002, "median": 1000000002, "min": 1000000001, "max": 1000000003, "range": 2, "population_variance": 0.6666666666666666, "population_sd": 0.816496580927726, "sample_variance": 1, "sample_sd": 1, "modes": [ 1000000001, 1000000002, 1000000003 ] } Numeric result generated independently by two reference paths.
OSCVB-007tiny scale	Descriptive Statistics	Stress	Very small scale View input and expected result Input JSON { "data": [ 1.0e-10, 2.0e-10, 3.0e-10, 4.0e-10 ] } Reference result { "n": 4, "sum": 1.0e-9, "mean": 2.5e-10, "median": 2.5e-10, "min": 1.0e-10, "max": 4.0e-10, "range": 3.0e-10, "population_variance": 1.25e-20, "population_sd": 1.1180339887498948e-10, "sample_variance": 1.6666666666666665e-20, "sample_sd": 1.2909944487358055e-10, "modes": [ 1.0e-10, 2.0e-10, 3.0e-10, 4.0e-10 ] } Numeric result generated independently by two reference paths.
OSCVB-008zeros	Descriptive Statistics	Boundary	Zero-heavy data View input and expected result Input JSON { "data": [ 0, 0, 0, 1 ] } Reference result { "n": 4, "sum": 1, "mean": 0.25, "median": 0, "min": 0, "max": 1, "range": 1, "population_variance": 0.1875, "population_sd": 0.4330127018922193, "sample_variance": 0.25, "sample_sd": 0.5, "modes": [ 0 ] } Numeric result generated independently by two reference paths.
OSCVB-009invalid empty	Descriptive Statistics	Invalid	Empty data must be rejected View input and expected result Input JSON { "data": [] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-010invalid text	Descriptive Statistics	Invalid	Nonnumeric entry must be rejected View input and expected result Input JSON { "data": [ 1, "abc", 3 ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-011left zero	Standard Normal Probability	Boundary	Central probability View input and expected result Input JSON { "mode": "left", "z": 0 } Reference result { "probability": 0.5, "z": 0, "mode": "left" } Numeric result generated independently by two reference paths.
OSCVB-012right 196	Standard Normal Probability	Standard	Common 5% two-sided threshold tail View input and expected result Input JSON { "mode": "right", "z": 1.96 } Reference result { "probability": 0.024997895148220435, "z": 1.96, "mode": "right" } Numeric result generated independently by two reference paths.
OSCVB-013left negative	Standard Normal Probability	Standard	Negative z View input and expected result Input JSON { "mode": "left", "z": -1.645 } Reference result { "probability": 0.04998490553912138, "z": -1.645, "mode": "left" } Numeric result generated independently by two reference paths.
OSCVB-014between one	Standard Normal Probability	Boundary	Central interval View input and expected result Input JSON { "mode": "between", "lower": -1, "upper": 1 } Reference result { "probability": 0.6826894921370859, "lower": -1, "upper": 1, "mode": "between" } Numeric result generated independently by two reference paths.
OSCVB-015extreme right	Standard Normal Probability	Stress	Extreme upper tail View input and expected result Input JSON { "mode": "right", "z": 8 } Reference result { "probability": 6.220960574271819e-16, "z": 8, "mode": "right" } Numeric result generated independently by two reference paths.
OSCVB-016extreme left	Standard Normal Probability	Stress	Extreme lower tail View input and expected result Input JSON { "mode": "left", "z": -8 } Reference result { "probability": 6.106226635438361e-16, "z": -8, "mode": "left" } Numeric result generated independently by two reference paths.
OSCVB-017small z	Standard Normal Probability	Standard	Rounding-sensitive near zero View input and expected result Input JSON { "mode": "left", "z": 1.0e-6 } Reference result { "probability": 0.5000003989422804, "z": 1.0e-6, "mode": "left" } Numeric result generated independently by two reference paths.
OSCVB-018wide between	Standard Normal Probability	Standard	Near-total probability View input and expected result Input JSON { "mode": "between", "lower": -6, "upper": 6 } Reference result { "probability": 0.9999999980268246, "lower": -6, "upper": 6, "mode": "between" } Numeric result generated independently by two reference paths.
OSCVB-019invalid bounds	Standard Normal Probability	Invalid	Reversed bounds must be rejected View input and expected result Input JSON { "mode": "between", "lower": 2, "upper": -2 } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-020invalid text	Standard Normal Probability	Invalid	Nonnumeric z must be rejected View input and expected result Input JSON { "mode": "left", "z": "abc" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-021exact basic	Binomial Probability	Standard	Symmetric exact probability View input and expected result Input JSON { "n": 10, "p": 0.5, "mode": "exact", "x": 5 } Reference result { "n": 10, "p": 0.5, "mode": "exact", "probability": 0.24609375 } Numeric result generated independently by two reference paths.
OSCVB-022at most	Binomial Probability	Standard	Lower cumulative probability View input and expected result Input JSON { "n": 20, "p": 0.3, "mode": "at_most", "x": 4 } Reference result { "n": 20, "p": 0.3, "mode": "at_most", "probability": 0.23750777887760133 } Numeric result generated independently by two reference paths.
OSCVB-023at least	Binomial Probability	Standard	Upper cumulative probability View input and expected result Input JSON { "n": 20, "p": 0.3, "mode": "at_least", "x": 8 } Reference result { "n": 20, "p": 0.3, "mode": "at_least", "probability": 0.22772820258183932 } Numeric result generated independently by two reference paths.
OSCVB-024between	Binomial Probability	Standard	Inclusive interval View input and expected result Input JSON { "n": 30, "p": 0.6, "mode": "between", "lower": 15, "upper": 20 } Reference result { "n": 30, "p": 0.6, "mode": "between", "probability": 0.7266566714352386 } Numeric result generated independently by two reference paths.
OSCVB-025p zero	Binomial Probability	Boundary	Degenerate p=0 boundary View input and expected result Input JSON { "n": 8, "p": 0, "mode": "exact", "x": 0 } Reference result { "n": 8, "p": 0, "mode": "exact", "probability": 1 } Numeric result generated independently by two reference paths.
OSCVB-026p one	Binomial Probability	Boundary	Degenerate p=1 boundary View input and expected result Input JSON { "n": 8, "p": 1, "mode": "at_least", "x": 8 } Reference result { "n": 8, "p": 1, "mode": "at_least", "probability": 1 } Numeric result generated independently by two reference paths.
OSCVB-027large n	Binomial Probability	Stress	Large n, small p View input and expected result Input JSON { "n": 1000, "p": 0.01, "mode": "at_most", "x": 15 } Reference result { "n": 1000, "p": 0.01, "mode": "at_most", "probability": 0.9521294142420446 } Numeric result generated independently by two reference paths.
OSCVB-028rare event	Binomial Probability	Stress	Rare-event upper probability View input and expected result Input JSON { "n": 200, "p": 0.001, "mode": "at_least", "x": 2 } Reference result { "n": 200, "p": 0.001, "mode": "at_least", "probability": 0.017457510966081866 } Numeric result generated independently by two reference paths.
OSCVB-029invalid p	Binomial Probability	Invalid	p outside [0,1] View input and expected result Input JSON { "n": 10, "p": 1.2, "mode": "exact", "x": 5 } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-030invalid x	Binomial Probability	Invalid	x greater than n View input and expected result Input JSON { "n": 10, "p": 0.5, "mode": "exact", "x": 11 } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-031typical two sided	One-Sample t Test	Standard	Ordinary two-sided test View input and expected result Input JSON { "data": [ 12, 15, 14, 10, 13, 16 ], "mu0": 11, "alternative": "two-sided" } Reference result { "n": 6, "mean": 13.333333333333334, "sample_sd": 2.1602468994692865, "standard_error": 0.8819171036881969, "mu0": 11, "t": 2.645751311064591, "df": 5, "p_value": 0.04565912378141161, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-032greater	One-Sample t Test	Standard	One-sided greater View input and expected result Input JSON { "data": [ 5.1, 5.3, 5.2, 5.4, 5 ], "mu0": 5, "alternative": "greater" } Reference result { "n": 5, "mean": 5.2, "sample_sd": 0.1581138830084191, "standard_error": 0.07071067811865481, "mu0": 5, "t": 2.8284271247461903, "df": 4, "p_value": 0.023710327792159796, "alternative": "greater" } Numeric result generated independently by two reference paths.
OSCVB-033less	One-Sample t Test	Standard	One-sided less View input and expected result Input JSON { "data": [ 8, 7, 9, 6, 8, 7 ], "mu0": 9, "alternative": "less" } Reference result { "n": 6, "mean": 7.5, "sample_sd": 1.0488088481701516, "standard_error": 0.4281744192888377, "mu0": 9, "t": -3.5032452487268526, "df": 5, "p_value": 0.0086122748401707, "alternative": "less" } Numeric result generated independently by two reference paths.
OSCVB-034n two	One-Sample t Test	Boundary	Smallest valid sample View input and expected result Input JSON { "data": [ 1, 3 ], "mu0": 0, "alternative": "two-sided" } Reference result { "n": 2, "mean": 2, "sample_sd": 1.4142135623730951, "standard_error": 1, "mu0": 0, "t": 2, "df": 1, "p_value": 0.2951672353008665, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-035large offset	One-Sample t Test	Stress	Large location View input and expected result Input JSON { "data": [ 1000001, 1000002, 1000003, 1000004 ], "mu0": 1000000, "alternative": "two-sided" } Reference result { "n": 4, "mean": 1000002.5, "sample_sd": 1.2909944487358056, "standard_error": 0.6454972243679028, "mu0": 1000000, "t": 3.872983346207417, "df": 3, "p_value": 0.030466291662170977, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-036tiny difference	One-Sample t Test	Stress	Rounding-sensitive difference View input and expected result Input JSON { "data": [ 1.000001, 1.000002, 0.999999, 1 ], "mu0": 1, "alternative": "two-sided" } Reference result { "n": 4, "mean": 1.0000005, "sample_sd": 1.290994448758596e-6, "standard_error": 6.45497224379298e-7, "mu0": 1, "t": 0.7745966693360807, "df": 3, "p_value": 0.4950253460114202, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-037negative	One-Sample t Test	Standard	Negative-valued sample View input and expected result Input JSON { "data": [ -4, -2, -3, -5, -1 ], "mu0": -2, "alternative": "two-sided" } Reference result { "n": 5, "mean": -3, "sample_sd": 1.5811388300841898, "standard_error": 0.7071067811865476, "mu0": -2, "t": -1.414213562373095, "df": 4, "p_value": 0.2301996410804989, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-038large n	One-Sample t Test	Stress	Larger deterministic sample View input and expected result Input JSON { "data": [ 10, 10.01, 10.02, 10.03, 10.04, 10.05, 10.06, 10.07, 10.08, 10.09, 10.1, 10.11, 10.12, 10.13, 10.14, 10.15, 10.16, 10.17, 10.18, 10.19, 10.2, 10.21, 10.22, 10.23, 10.24, 10.25, 10.26, 10.27, 10.28, 10.29, 10.3, 10.31, 10.32, 10.33, 10.34, 10.35, 10.36, 10.37, 10.38, 10.39, 10.4, 10.41, 10.42, 10.43, 10.44, 10.45, 10.46, 10.47, 10.48, 10.49, 10.5, 10.51, 10.52, 10.53, 10.54, 10.55, 10.56, 10.57, 10.58, 10.59, 10.6, 10.61, 10.62, 10.63, 10.64, 10.65, 10.66, 10.67, 10.68, 10.69, 10.7, 10.71, 10.72, 10.73, 10.74, 10.75, 10.76, 10.77, 10.78, 10.79, 10.8, 10.81, 10.82, 10.83, 10.84, 10.85, 10.86, 10.87, 10.88, 10.89, 10.9, 10.91, 10.92, 10.93, 10.94, 10.95, 10.96, 10.97, 10.98, 10.99 ], "mu0": 10.4, "alternative": "two-sided" } Reference result { "n": 100, "mean": 10.495, "sample_sd": 0.2901149197588202, "standard_error": 0.02901149197588202, "mu0": 10.4, "t": 3.274564440842089, "df": 99, "p_value": 0.0014587353384579692, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-039invalid n	One-Sample t Test	Invalid	n<2 must be rejected View input and expected result Input JSON { "data": [ 5 ], "mu0": 5, "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-040invalid zero variance	One-Sample t Test	Invalid	Zero variance must be flagged View input and expected result Input JSON { "data": [ 3, 3, 3, 3 ], "mu0": 2, "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-041typical	Welch Independent t Test	Standard	Ordinary unequal-variance comparison View input and expected result Input JSON { "group1": [ 12, 14, 15, 13, 16 ], "group2": [ 10, 11, 9, 12, 10 ], "alternative": "two-sided" } Reference result { "n1": 5, "n2": 5, "mean1": 14, "mean2": 10.4, "variance1": 2.5, "variance2": 1.2999999999999998, "mean_difference": 3.5999999999999996, "standard_error": 0.8717797887081347, "t": 4.129483209670112, "df": 7.27455919395466, "p_value": 0.00405449862457706, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-042unequal variance	Welch Independent t Test	Standard	Strong variance inequality View input and expected result Input JSON { "group1": [ 1, 2, 3, 4, 20 ], "group2": [ 5, 5.2, 5.1, 4.9, 5 ], "alternative": "two-sided" } Reference result { "n1": 5, "n2": 5, "mean1": 6, "mean2": 5.04, "variance1": 62.5, "variance2": 0.01299999999999998, "mean_difference": 0.96, "standard_error": 3.5359015823407756, "t": 0.2715007693637439, "df": 4.001663999928009, "p_value": 0.7994368152496399, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-043greater	Welch Independent t Test	Standard	One-sided greater View input and expected result Input JSON { "group1": [ 8, 9, 10, 11, 12 ], "group2": [ 5, 6, 7, 8, 9 ], "alternative": "greater" } Reference result { "n1": 5, "n2": 5, "mean1": 10, "mean2": 7, "variance1": 2.5, "variance2": 2.5, "mean_difference": 3, "standard_error": 1, "t": 3, "df": 8, "p_value": 0.008535840616891326, "alternative": "greater", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-044less	Welch Independent t Test	Standard	One-sided less View input and expected result Input JSON { "group1": [ 1, 2, 3, 4 ], "group2": [ 3, 4, 5, 6, 7 ], "alternative": "less" } Reference result { "n1": 4, "n2": 5, "mean1": 2.5, "mean2": 5, "variance1": 1.6666666666666667, "variance2": 2.5, "mean_difference": -2.5, "standard_error": 0.9574271077563381, "t": -2.6111648393354674, "df": 6.980769230769232, "p_value": 0.017469391179982, "alternative": "less", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-045small samples	Welch Independent t Test	Standard	Two observations per group View input and expected result Input JSON { "group1": [ 1, 4 ], "group2": [ 2, 3 ], "alternative": "two-sided" } Reference result { "n1": 2, "n2": 2, "mean1": 2.5, "mean2": 2.5, "variance1": 4.5, "variance2": 0.5, "mean_difference": 0, "standard_error": 1.5811388300841898, "t": 0, "df": 1.2195121951219512, "p_value": 1, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-046large offset	Welch Independent t Test	Stress	Large location View input and expected result Input JSON { "group1": [ 1000001, 1000002, 1000004 ], "group2": [ 1000000, 1000001, 1000001 ], "alternative": "two-sided" } Reference result { "n1": 3, "n2": 3, "mean1": 1000002.3333333334, "mean2": 1000000.6666666666, "variance1": 2.333333333333333, "variance2": 0.33333333333333337, "mean_difference": 1.6666666667442769, "standard_error": 0.9428090415820634, "t": 1.7677669530486868, "df": 2.5600000000000005, "p_value": 0.19079766891564973, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-047negative	Welch Independent t Test	Standard	Negative groups View input and expected result Input JSON { "group1": [ -5, -4, -3, -2 ], "group2": [ -1, 0, 1, 2 ], "alternative": "two-sided" } Reference result { "n1": 4, "n2": 4, "mean1": -3.5, "mean2": 0.5, "variance1": 1.6666666666666667, "variance2": 1.6666666666666667, "mean_difference": -4, "standard_error": 0.9128709291752769, "t": -4.381780460041329, "df": 6, "p_value": 0.004659214943993934, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-048unbalanced	Welch Independent t Test	Standard	Unequal sample sizes View input and expected result Input JSON { "group1": [ 1, 2, 3, 4, 5, 6, 7, 8 ], "group2": [ 2, 4, 6 ], "alternative": "two-sided" } Reference result { "n1": 8, "n2": 3, "mean1": 4.5, "mean2": 4, "variance1": 6, "variance2": 4, "mean_difference": 0.5, "standard_error": 1.4433756729740643, "t": 0.34641016151377546, "df": 4.477993858751279, "p_value": 0.7447089893473855, "alternative": "two-sided", "variance_model": "Welch" } Numeric result generated independently by two reference paths.
OSCVB-049invalid n	Welch Independent t Test	Invalid	Group with n<2 View input and expected result Input JSON { "group1": [ 1 ], "group2": [ 2, 3 ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-050invalid both constant	Welch Independent t Test	Invalid	Both variances zero View input and expected result Input JSON { "group1": [ 3, 3, 3 ], "group2": [ 4, 4, 4 ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-051typical	Paired t Test	Standard	Matched before/after View input and expected result Input JSON { "before": [ 10, 12, 9, 14, 11 ], "after": [ 9, 11, 10, 12, 10 ], "alternative": "two-sided" } Reference result { "n_pairs": 5, "mean_difference_before_minus_after": 0.8, "sd_difference": 1.0954451150103321, "standard_error": 0.48989794855663554, "t": 1.6329931618554525, "df": 4, "p_value": 0.17780780835622123, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-052greater	Paired t Test	Standard	Positive before-minus-after View input and expected result Input JSON { "before": [ 8, 9, 10, 11, 12 ], "after": [ 7, 8, 9, 10, 10 ], "alternative": "greater" } Reference result { "n_pairs": 5, "mean_difference_before_minus_after": 1.2, "sd_difference": 0.4472135954999579, "standard_error": 0.19999999999999998, "t": 6, "df": 4, "p_value": 0.0019412685234802554, "alternative": "greater" } Numeric result generated independently by two reference paths.
OSCVB-053less	Paired t Test	Standard	Negative before-minus-after View input and expected result Input JSON { "before": [ 1, 2, 3, 4, 5 ], "after": [ 2, 3, 4, 5, 7 ], "alternative": "less" } Reference result { "n_pairs": 5, "mean_difference_before_minus_after": -1.2, "sd_difference": 0.4472135954999579, "standard_error": 0.19999999999999998, "t": -6, "df": 4, "p_value": 0.0019412685234802554, "alternative": "less" } Numeric result generated independently by two reference paths.
OSCVB-054small n	Paired t Test	Boundary	Two pairs View input and expected result Input JSON { "before": [ 1, 4 ], "after": [ 2, 2 ], "alternative": "two-sided" } Reference result { "n_pairs": 2, "mean_difference_before_minus_after": 0.5, "sd_difference": 2.1213203435596424, "standard_error": 1.4999999999999998, "t": 0.33333333333333337, "df": 1, "p_value": 0.7951672353008665, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-055large offset	Paired t Test	Stress	Large paired values View input and expected result Input JSON { "before": [ 1000002, 1000004, 1000005 ], "after": [ 1000001, 1000002, 1000004 ], "alternative": "two-sided" } Reference result { "n_pairs": 3, "mean_difference_before_minus_after": 1.3333333333333333, "sd_difference": 0.5773502691896257, "standard_error": 0.3333333333333333, "t": 4, "df": 2, "p_value": 0.05719095841793663, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-056tiny diff	Paired t Test	Stress	Tiny differences View input and expected result Input JSON { "before": [ 1.000001, 1.000003, 1.000002, 1 ], "after": [ 1, 1.000001, 1.000003, 0.999999 ], "alternative": "two-sided" } Reference result { "n_pairs": 4, "mean_difference_before_minus_after": 7.500000000215667e-7, "sd_difference": 1.258305739189154e-6, "standard_error": 6.29152869594577e-7, "t": 1.1920791214142648, "df": 3, "p_value": 0.31893179189391346, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-057mixed sign	Paired t Test	Standard	Mixed signed values View input and expected result Input JSON { "before": [ -2, -1, 0, 1, 2 ], "after": [ -1, -2, 1, 0, 3 ], "alternative": "two-sided" } Reference result { "n_pairs": 5, "mean_difference_before_minus_after": -0.2, "sd_difference": 1.0954451150103324, "standard_error": 0.48989794855663565, "t": -0.408248290463863, "df": 4, "p_value": 0.7040000000000001, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-058n ten	Paired t Test	Standard	Larger paired sample View input and expected result Input JSON { "before": [ 10, 11, 13, 12, 14, 15, 12, 16, 13, 17 ], "after": [ 9, 12, 12, 11, 13, 14, 13, 15, 12, 16 ], "alternative": "two-sided" } Reference result { "n_pairs": 10, "mean_difference_before_minus_after": 0.6, "sd_difference": 0.8432740427115679, "standard_error": 0.26666666666666666, "t": 2.25, "df": 9, "p_value": 0.05100326070695079, "alternative": "two-sided" } Numeric result generated independently by two reference paths.
OSCVB-059invalid lengths	Paired t Test	Invalid	Unequal vector lengths View input and expected result Input JSON { "before": [ 1, 2, 3 ], "after": [ 1, 2 ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-060invalid constant diff	Paired t Test	Invalid	Constant differences View input and expected result Input JSON { "before": [ 2, 3, 4, 5 ], "after": [ 1, 2, 3, 4 ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-061three groups	One-Way ANOVA	Standard	Three ordinary groups View input and expected result Input JSON { "groups": [ [ 1, 2, 3, 4 ], [ 2, 3, 4, 5 ], [ 5, 6, 7, 8 ] ] } Reference result { "groups": 3, "n_total": 12, "group_ns": [ 4, 4, 4 ], "group_means": [ 2.5, 3.5, 6.5 ], "grand_mean": 4.166666666666667, "ss_between": 34.666666666666664, "ss_within": 15, "df_between": 2, "df_within": 9, "ms_between": 17.333333333333332, "ms_within": 1.6666666666666667, "f": 10.399999999999999, "p_value": 0.004572125092826608 } Numeric result generated independently by two reference paths.
OSCVB-062four groups	One-Way ANOVA	Standard	Four groups View input and expected result Input JSON { "groups": [ [ 10, 11, 9 ], [ 12, 13, 11 ], [ 10, 10.5, 11 ], [ 15, 14, 16 ] ] } Reference result { "groups": 4, "n_total": 12, "group_ns": [ 3, 3, 3, 3 ], "group_means": [ 10, 12, 10.5, 15 ], "grand_mean": 11.875, "ss_between": 45.5625, "ss_within": 6.5, "df_between": 3, "df_within": 8, "ms_between": 15.1875, "ms_within": 0.8125, "f": 18.692307692307693, "p_value": 0.0005672527585095778 } Numeric result generated independently by two reference paths.
OSCVB-063equal means	One-Way ANOVA	Standard	Similar means View input and expected result Input JSON { "groups": [ [ 1, 2, 3 ], [ 2, 2, 2.5 ], [ 1.5, 2, 2.5 ] ] } Reference result { "groups": 3, "n_total": 9, "group_ns": [ 3, 3, 3 ], "group_means": [ 2, 2.1666666666666665, 2 ], "grand_mean": 2.0555555555555554, "ss_between": 0.055555555555555455, "ss_within": 2.6666666666666665, "df_between": 2, "df_within": 6, "ms_between": 0.027777777777777728, "ms_within": 0.4444444444444444, "f": 0.06249999999999989, "p_value": 0.9400164897279196 } Numeric result generated independently by two reference paths.
OSCVB-064unbalanced	One-Way ANOVA	Standard	Unequal group sizes View input and expected result Input JSON { "groups": [ [ 1, 2, 3, 4, 5 ], [ 2, 3, 4 ], [ 6, 7, 8, 9 ] ] } Reference result { "groups": 3, "n_total": 12, "group_ns": [ 5, 3, 4 ], "group_means": [ 3, 3, 7.5 ], "grand_mean": 4.5, "ss_between": 54, "ss_within": 17, "df_between": 2, "df_within": 9, "ms_between": 27, "ms_within": 1.8888888888888888, "f": 14.294117647058824, "p_value": 0.0016082648751593702 } Numeric result generated independently by two reference paths.
OSCVB-065large offset	One-Way ANOVA	Stress	Large location View input and expected result Input JSON { "groups": [ [ 1000001, 1000002, 1000003 ], [ 1000002, 1000003, 1000004 ], [ 1000005, 1000006, 1000007 ] ] } Reference result { "groups": 3, "n_total": 9, "group_ns": [ 3, 3, 3 ], "group_means": [ 1000002, 1000003, 1000006 ], "grand_mean": 1000003.6666666666, "ss_between": 26, "ss_within": 6, "df_between": 2, "df_within": 6, "ms_between": 13, "ms_within": 1, "f": 13, "p_value": 0.006591796875 } Numeric result generated independently by two reference paths.
OSCVB-066negative	One-Way ANOVA	Standard	Negative values View input and expected result Input JSON { "groups": [ [ -5, -4, -3 ], [ -3, -2, -1 ], [ 0, 1, 2 ] ] } Reference result { "groups": 3, "n_total": 9, "group_ns": [ 3, 3, 3 ], "group_means": [ -4, -2, 1 ], "grand_mean": -1.6666666666666667, "ss_between": 38, "ss_within": 6, "df_between": 2, "df_within": 6, "ms_between": 19, "ms_within": 1, "f": 19, "p_value": 0.002535687453042824 } Numeric result generated independently by two reference paths.
OSCVB-067tiny scale	One-Way ANOVA	Stress	Tiny scale View input and expected result Input JSON { "groups": [ [ 1.0e-8, 2.0e-8, 3.0e-8 ], [ 2.0e-8, 3.0e-8, 4.0e-8 ], [ 5.0e-8, 6.0e-8, 7.0e-8 ] ] } Reference result { "groups": 3, "n_total": 9, "group_ns": [ 3, 3, 3 ], "group_means": [ 2.0000000000000004e-8, 3.0e-8, 6.0e-8 ], "grand_mean": 3.666666666666667e-8, "ss_between": 2.599999999999999e-15, "ss_within": 6.000000000000001e-16, "df_between": 2, "df_within": 6, "ms_between": 1.2999999999999994e-15, "ms_within": 1.0000000000000001e-16, "f": 12.999999999999993, "p_value": 0.006591796875000009 } Numeric result generated independently by two reference paths.
OSCVB-068many values	One-Way ANOVA	Standard	Thirty observations View input and expected result Input JSON { "groups": [ [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ], [ 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 ], [ 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 ] ] } Reference result { "groups": 3, "n_total": 30, "group_ns": [ 10, 10, 10 ], "group_means": [ 5.5, 7.5, 12.5 ], "grand_mean": 8.5, "ss_between": 260, "ss_within": 247.5, "df_between": 2, "df_within": 27, "ms_between": 130, "ms_within": 9.166666666666666, "f": 14.181818181818183, "p_value": 6.164226884158116e-5 } Numeric result generated independently by two reference paths.
OSCVB-069invalid one group	One-Way ANOVA	Invalid	Only one group View input and expected result Input JSON { "groups": [ [ 1, 2, 3 ] ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-070invalid singleton	One-Way ANOVA	Invalid	Group with one observation View input and expected result Input JSON { "groups": [ [ 1, 2, 3 ], [ 4 ] ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-071positive	Pearson Correlation	Standard	Strong positive association View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5, 6 ], "y": [ 2, 4, 5, 8, 10, 11 ] } Reference result { "n": 6, "r": 0.9912398251570825, "r_squared": 0.9825563909774435, "p_value": 0.00011477486410378612 } Numeric result generated independently by two reference paths.
OSCVB-072negative	Pearson Correlation	Standard	Strong negative association View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5 ], "y": [ 10, 8, 6, 5, 2 ] } Reference result { "n": 5, "r": -0.9904434667711052, "r_squared": 0.9809782608695653, "p_value": 0.0011198526620164965 } Numeric result generated independently by two reference paths.
OSCVB-073perfect positive	Pearson Correlation	Standard	Perfect positive correlation View input and expected result Input JSON { "x": [ 1, 2, 3, 4 ], "y": [ 3, 6, 9, 12 ] } Reference result { "n": 4, "r": 1, "r_squared": 1, "p_value": 0 } Numeric result generated independently by two reference paths.
OSCVB-074perfect negative	Pearson Correlation	Standard	Perfect negative correlation View input and expected result Input JSON { "x": [ 1, 2, 3, 4 ], "y": [ 8, 6, 4, 2 ] } Reference result { "n": 4, "r": -1, "r_squared": 1, "p_value": 0 } Numeric result generated independently by two reference paths.
OSCVB-075near zero	Pearson Correlation	Boundary	Near-zero association View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5, 6 ], "y": [ 1, -1, 1, -1, 1, -1 ] } Reference result { "n": 6, "r": -0.29277002188455997, "r_squared": 0.08571428571428573, "p_value": 0.5733922538253555 } Numeric result generated independently by two reference paths.
OSCVB-076large offset	Pearson Correlation	Stress	Large location View input and expected result Input JSON { "x": [ 1000001, 1000002, 1000003, 1000004 ], "y": [ 2000005, 2000007, 2000008, 2000010 ] } Reference result { "n": 4, "r": 0.9922778767136677, "r_squared": 0.9846153846153848, "p_value": 0.007722123286332266 } Numeric result generated independently by two reference paths.
OSCVB-077tiny scale	Pearson Correlation	Stress	Tiny scale View input and expected result Input JSON { "x": [ 1.0e-9, 2.0e-9, 3.0e-9, 4.0e-9 ], "y": [ 4.0e-9, 1.0e-9, 3.0e-9, 2.0e-9 ] } Reference result { "n": 4, "r": -0.4, "r_squared": 0.16000000000000003, "p_value": 0.5999999999999999 } Numeric result generated independently by two reference paths.
OSCVB-078n three	Pearson Correlation	Boundary	Smallest allowed n in benchmark View input and expected result Input JSON { "x": [ 1, 2, 4 ], "y": [ 2, 1, 5 ] } Reference result { "n": 3, "r": 0.8386278693775348, "r_squared": 0.7032967032967035, "p_value": 0.366716573320923 } Numeric result generated independently by two reference paths.
OSCVB-079invalid lengths	Pearson Correlation	Invalid	Unequal vector lengths View input and expected result Input JSON { "x": [ 1, 2, 3 ], "y": [ 1, 2 ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-080invalid constant	Pearson Correlation	Invalid	Constant x View input and expected result Input JSON { "x": [ 1, 1, 1, 1 ], "y": [ 1, 2, 3, 4 ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-081positive	Simple Linear Regression	Standard	Positive fitted line View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5 ], "y": [ 2, 4, 5, 8, 10 ] } Reference result { "n": 5, "slope": 2, "intercept": -0.20000000000000018, "r": 0.9901475429766744, "r_squared": 0.9803921568627453, "sse": 0.7999999999999996, "df_residual": 3, "slope_standard_error": 0.16329931618554516, "p_value_slope": 0.0011722164447168855 } Numeric result generated independently by two reference paths.
OSCVB-082negative	Simple Linear Regression	Standard	Negative slope View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5 ], "y": [ 10, 8, 7, 4, 2 ] } Reference result { "n": 5, "slope": -2, "intercept": 12.2, "r": -0.9901475429766744, "r_squared": 0.9803921568627453, "sse": 0.7999999999999999, "df_residual": 3, "slope_standard_error": 0.1632993161855452, "p_value_slope": 0.0011722164447168862 } Numeric result generated independently by two reference paths.
OSCVB-083perfect	Simple Linear Regression	Standard	Perfect line View input and expected result Input JSON { "x": [ 1, 2, 3, 4 ], "y": [ 3, 5, 7, 9 ] } Reference result { "n": 4, "slope": 2, "intercept": 1, "r": 1, "r_squared": 1, "sse": 0, "df_residual": 2, "slope_standard_error": 0, "p_value_slope": 0 } Numeric result generated independently by two reference paths.
OSCVB-084with intercept	Simple Linear Regression	Standard	Nonzero intercept View input and expected result Input JSON { "x": [ 0, 1, 2, 3, 4 ], "y": [ 5, 6.2, 7.9, 9.1, 10.8 ] } Reference result { "n": 5, "slope": 1.4500000000000002, "intercept": 4.8999999999999995, "r": 0.9982211666911958, "r_squared": 0.9964454976303321, "sse": 0.07500000000000044, "df_residual": 3, "slope_standard_error": 0.05000000000000015, "p_value_slope": 9.003695987429144e-5 } Numeric result generated independently by two reference paths.
OSCVB-085large offset	Simple Linear Regression	Stress	Large x location View input and expected result Input JSON { "x": [ 1000001, 1000002, 1000003, 1000004 ], "y": [ 2000003, 2000005, 2000007, 2000009 ] } Reference result { "n": 4, "slope": 2, "intercept": 1, "r": 1, "r_squared": 1, "sse": 0, "df_residual": 2, "slope_standard_error": 0, "p_value_slope": 0 } Numeric result generated independently by two reference paths.
OSCVB-086tiny scale	Simple Linear Regression	Stress	Tiny scale View input and expected result Input JSON { "x": [ 1.0e-6, 2.0e-6, 3.0e-6, 4.0e-6 ], "y": [ 2.0e-6, 2.5e-6, 4.0e-6, 5.0e-6 ] } Reference result { "n": 4, "slope": 1.05, "intercept": 7.5e-7, "r": 0.9844951849708403, "r_squared": 0.9692307692307691, "sse": 1.7499999999999973e-13, "df_residual": 2, "slope_standard_error": 0.13228756555322943, "p_value_slope": 0.015504815029159586 } Numeric result generated independently by two reference paths.
OSCVB-087noisy	Simple Linear Regression	Standard	Moderately noisy line View input and expected result Input JSON { "x": [ 1, 2, 3, 4, 5, 6, 7, 8 ], "y": [ 1, 4, 2, 7, 5, 9, 8, 11 ] } Reference result { "n": 8, "slope": 1.2976190476190477, "intercept": 0.03571428571428559, "r": 0.9128133087414265, "r_squared": 0.8332281366154709, "sse": 14.154761904761905, "df_residual": 6, "slope_standard_error": 0.23700145589903351, "p_value_slope": 0.001550424173435104 } Numeric result generated independently by two reference paths.
OSCVB-088negative x	Simple Linear Regression	Standard	Negative and positive x View input and expected result Input JSON { "x": [ -4, -3, -2, -1, 0, 1 ], "y": [ 8, 6, 5, 3, 2, 0 ] } Reference result { "n": 6, "slope": -1.542857142857143, "intercept": 1.6857142857142855, "r": -0.9959100033104787, "r_squared": 0.9918367346938778, "sse": 0.34285714285714325, "df_residual": 4, "slope_standard_error": 0.06998542122237655, "p_value_slope": 2.5057900499010673e-5 } Numeric result generated independently by two reference paths.
OSCVB-089invalid lengths	Simple Linear Regression	Invalid	Unequal lengths View input and expected result Input JSON { "x": [ 1, 2, 3 ], "y": [ 1, 2 ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-090invalid constant x	Simple Linear Regression	Invalid	Constant predictor View input and expected result Input JSON { "x": [ 2, 2, 2, 2 ], "y": [ 1, 2, 3, 4 ] } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-091two by two no correction	Chi-Square Independence	Standard	2x2 Pearson chi-square View input and expected result Input JSON { "table": [ [ 20, 30 ], [ 10, 40 ] ], "correction": false } Reference result { "rows": 2, "columns": 2, "correction": 0, "chi_square": 4.761904761904762, "df": 1, "p_value": 0.029096331741252267, "expected": [ [ 15, 35 ], [ 15, 35 ] ], "minimum_expected": 15 } Numeric result generated independently by two reference paths.
OSCVB-092two by two yates	Chi-Square Independence	Standard	2x2 Yates correction View input and expected result Input JSON { "table": [ [ 20, 30 ], [ 10, 40 ] ], "correction": true } Reference result { "rows": 2, "columns": 2, "correction": 1, "chi_square": 3.8571428571428577, "df": 1, "p_value": 0.04953461343562649, "expected": [ [ 15, 35 ], [ 15, 35 ] ], "minimum_expected": 15 } Numeric result generated independently by two reference paths.
OSCVB-093two by three	Chi-Square Independence	Standard	2x3 table View input and expected result Input JSON { "table": [ [ 10, 20, 30 ], [ 20, 15, 25 ] ], "correction": false } Reference result { "rows": 2, "columns": 3, "correction": 0, "chi_square": 4.502164502164502, "df": 2, "p_value": 0.10528521784009057, "expected": [ [ 15, 17.5, 27.5 ], [ 15, 17.5, 27.5 ] ], "minimum_expected": 15 } Numeric result generated independently by two reference paths.
OSCVB-094three by three	Chi-Square Independence	Standard	3x3 table View input and expected result Input JSON { "table": [ [ 12, 8, 5 ], [ 7, 15, 8 ], [ 3, 9, 18 ] ], "correction": false } Reference result { "rows": 3, "columns": 3, "correction": 0, "chi_square": 16.82229349951124, "df": 4, "p_value": 0.002092800958773323, "expected": [ [ 6.470588235294118, 9.411764705882353, 9.117647058823529 ], [ 7.764705882352941, 11.294117647058824, 10.941176470588236 ], [ 7.764705882352941, 11.294117647058824, 10.941176470588236 ] ], "minimum_expected": 6.470588235294118 } Numeric result generated independently by two reference paths.
OSCVB-095balanced	Chi-Square Independence	Standard	Exact independence View input and expected result Input JSON { "table": [ [ 25, 25 ], [ 25, 25 ] ], "correction": false } Reference result { "rows": 2, "columns": 2, "correction": 0, "chi_square": 0, "df": 1, "p_value": 1, "expected": [ [ 25, 25 ], [ 25, 25 ] ], "minimum_expected": 25 } Numeric result generated independently by two reference paths.
OSCVB-096sparse	Chi-Square Independence	Standard	Small expected cells View input and expected result Input JSON { "table": [ [ 1, 9 ], [ 8, 2 ] ], "correction": false } Reference result { "rows": 2, "columns": 2, "correction": 0, "chi_square": 9.8989898989899, "df": 1, "p_value": 0.0016536951637003395, "expected": [ [ 4.5, 5.5 ], [ 4.5, 5.5 ] ], "minimum_expected": 4.5 } Numeric result generated independently by two reference paths.
OSCVB-097large counts	Chi-Square Independence	Stress	Large counts View input and expected result Input JSON { "table": [ [ 10000, 12000 ], [ 11000, 9000 ] ], "correction": false } Reference result { "rows": 2, "columns": 2, "correction": 0, "chi_square": 381.8181818181818, "df": 1, "p_value": 5.001692026127486e-85, "expected": [ [ 11000, 11000 ], [ 10000, 10000 ] ], "minimum_expected": 10000 } Numeric result generated independently by two reference paths.
OSCVB-098unbalanced margins	Chi-Square Independence	Standard	Highly unequal margins View input and expected result Input JSON { "table": [ [ 50, 1, 2 ], [ 3, 20, 4 ] ], "correction": false } Reference result { "rows": 2, "columns": 3, "correction": 0, "chi_square": 57.119651309754545, "df": 2, "p_value": 3.9502566601107176e-13, "expected": [ [ 35.1125, 13.9125, 3.975 ], [ 17.8875, 7.0875, 2.025 ] ], "minimum_expected": 2.025 } Numeric result generated independently by two reference paths.
OSCVB-099invalid negative	Chi-Square Independence	Invalid	Negative count View input and expected result Input JSON { "table": [ [ 1, -1 ], [ 2, 3 ] ], "correction": false } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-100invalid zero row	Chi-Square Independence	Invalid	Zero row total View input and expected result Input JSON { "table": [ [ 0, 0 ], [ 2, 3 ] ], "correction": false } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-101two sided	Fisher Exact 2×2	Standard	Classic small-table example View input and expected result Input JSON { "table": [ [ 1, 9 ], [ 11, 3 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": 0.030303030303030304, "p_value": 0.002759456185220083 } Numeric result generated independently by two reference paths.
OSCVB-102greater	Fisher Exact 2×2	Standard	One-sided greater View input and expected result Input JSON { "table": [ [ 8, 2 ], [ 1, 5 ] ], "alternative": "greater" } Reference result { "alternative": "greater", "odds_ratio": 20, "p_value": 0.024475524475524476 } Numeric result generated independently by two reference paths.
OSCVB-103less	Fisher Exact 2×2	Standard	One-sided less View input and expected result Input JSON { "table": [ [ 1, 5 ], [ 8, 2 ] ], "alternative": "less" } Reference result { "alternative": "less", "odds_ratio": 0.05, "p_value": 0.024475524475524476 } Numeric result generated independently by two reference paths.
OSCVB-104zero cell	Fisher Exact 2×2	Boundary	Zero cell View input and expected result Input JSON { "table": [ [ 0, 5 ], [ 4, 1 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": 0, "p_value": 0.047619047619047616 } Numeric result generated independently by two reference paths.
OSCVB-105balanced	Fisher Exact 2×2	Standard	Balanced table View input and expected result Input JSON { "table": [ [ 5, 5 ], [ 5, 5 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": 1, "p_value": 1 } Numeric result generated independently by two reference paths.
OSCVB-106large counts	Fisher Exact 2×2	Stress	Larger exact table View input and expected result Input JSON { "table": [ [ 100, 120 ], [ 90, 130 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": 1.2037037037037037, "p_value": 0.3864020631685246 } Numeric result generated independently by two reference paths.
OSCVB-107separation	Fisher Exact 2×2	Standard	Complete separation View input and expected result Input JSON { "table": [ [ 10, 0 ], [ 0, 10 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": "Infinity", "p_value": 1.082508822446903e-5 } Numeric result generated independently by two reference paths.
OSCVB-108small counts	Fisher Exact 2×2	Standard	Very small counts View input and expected result Input JSON { "table": [ [ 1, 1 ], [ 1, 2 ] ], "alternative": "two-sided" } Reference result { "alternative": "two-sided", "odds_ratio": 2, "p_value": 1 } Numeric result generated independently by two reference paths.
OSCVB-109invalid shape	Fisher Exact 2×2	Invalid	Not 2x2 View input and expected result Input JSON { "table": [ [ 1, 2, 3 ], [ 4, 5, 6 ] ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-110invalid negative	Fisher Exact 2×2	Invalid	Negative count View input and expected result Input JSON { "table": [ [ 1, -1 ], [ 2, 3 ] ], "alternative": "two-sided" } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-111t 95	Confidence Interval for a Mean	Standard	95% t interval View input and expected result Input JSON { "method": "t", "data": [ 12, 15, 14, 10, 13, 16 ], "confidence": 0.95 } Reference result { "method": "t", "confidence": 0.95, "n": 6, "mean": 13.333333333333334, "sample_sd": 2.1602468994692865, "standard_error": 0.8819171036881969, "critical_value": 2.5705818356363146, "lower": 11.066293246055467, "upper": 15.6003734206112 } Numeric result generated independently by two reference paths.
OSCVB-112t 90	Confidence Interval for a Mean	Standard	90% t interval View input and expected result Input JSON { "method": "t", "data": [ 5.1, 5.3, 5.2, 5.4, 5 ], "confidence": 0.9 } Reference result { "method": "t", "confidence": 0.9, "n": 5, "mean": 5.2, "sample_sd": 0.1581138830084191, "standard_error": 0.07071067811865481, "critical_value": 2.1318467863266495, "lower": 5.0492556680937675, "upper": 5.350744331906233 } Numeric result generated independently by two reference paths.
OSCVB-113t small n	Confidence Interval for a Mean	Boundary	n=2 t interval View input and expected result Input JSON { "method": "t", "data": [ 1, 4 ], "confidence": 0.95 } Reference result { "method": "t", "confidence": 0.95, "n": 2, "mean": 2.5, "sample_sd": 2.1213203435596424, "standard_error": 1.4999999999999998, "critical_value": 12.706204736174694, "lower": -16.55930710426204, "upper": 21.55930710426204 } Numeric result generated independently by two reference paths.
OSCVB-114t large offset	Confidence Interval for a Mean	Stress	Large location View input and expected result Input JSON { "method": "t", "data": [ 1000001, 1000002, 1000003, 1000004 ], "confidence": 0.99 } Reference result { "method": "t", "confidence": 0.99, "n": 4, "mean": 1000002.5, "sample_sd": 1.2909944487358056, "standard_error": 0.6454972243679028, "critical_value": 5.840909309733355, "lower": 999998.7297092528, "upper": 1000006.2702907472 } Numeric result generated independently by two reference paths.
OSCVB-115t tiny scale	Confidence Interval for a Mean	Stress	Tiny scale View input and expected result Input JSON { "method": "t", "data": [ 1.0e-8, 2.0e-8, 3.0e-8, 4.0e-8 ], "confidence": 0.95 } Reference result { "method": "t", "confidence": 0.95, "n": 4, "mean": 2.5000000000000002e-8, "sample_sd": 1.2909944487358056e-8, "standard_error": 6.454972243679028e-9, "critical_value": 3.1824463052837078, "lower": 4.457397432394794e-9, "upper": 4.554260256760521e-8 } Numeric result generated independently by two reference paths.
OSCVB-116z 95	Confidence Interval for a Mean	Standard	Known sigma 95% View input and expected result Input JSON { "method": "z", "mean": 50, "sigma": 10, "n": 100, "confidence": 0.95 } Reference result { "method": "z", "confidence": 0.95, "n": 100, "mean": 50, "sigma": 10, "standard_error": 1, "critical_value": 1.959963984540054, "lower": 48.04003601545995, "upper": 51.95996398454005 } Numeric result generated independently by two reference paths.
OSCVB-117z 99	Confidence Interval for a Mean	Standard	Known sigma 99% View input and expected result Input JSON { "method": "z", "mean": 3.5, "sigma": 0.8, "n": 64, "confidence": 0.99 } Reference result { "method": "z", "confidence": 0.99, "n": 64, "mean": 3.5, "sigma": 0.8, "standard_error": 0.1, "critical_value": 2.5758293035489004, "lower": 3.24241706964511, "upper": 3.75758293035489 } Numeric result generated independently by two reference paths.
OSCVB-118z large n	Confidence Interval for a Mean	Stress	Very large n View input and expected result Input JSON { "method": "z", "mean": 1000, "sigma": 250, "n": 1000000, "confidence": 0.95 } Reference result { "method": "z", "confidence": 0.95, "n": 1000000, "mean": 1000, "sigma": 250, "standard_error": 0.25, "critical_value": 1.959963984540054, "lower": 999.510009003865, "upper": 1000.489990996135 } Numeric result generated independently by two reference paths.
OSCVB-119invalid confidence	Confidence Interval for a Mean	Invalid	Invalid confidence View input and expected result Input JSON { "method": "z", "mean": 10, "sigma": 2, "n": 20, "confidence": 1.2 } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.
OSCVB-120invalid n	Confidence Interval for a Mean	Invalid	n<2 for t interval View input and expected result Input JSON { "method": "t", "data": [ 5 ], "confidence": 0.95 } Reference result Both reference engines must reject this invalid input. Both reference engines must reject the input.

Beyond the number

100-point transparency rubric

Numeric accuracy is essential, but a reliable educational calculator must also reveal its method and limitations.

Numeric accuracy30 points

Exact/near-exact output

All required numeric fields are within benchmark tolerance.

Method specification12 points

Variant disclosed

Names Welch/pooled, Yates/no correction, sample/population SD, tail, CI method, and other relevant choices.

Invalid input handling12 points

Unsafe inputs rejected

Rejects impossible probabilities, negative counts, mismatched vectors, insufficient n, and undefined variance cases.

Boundary behavior10 points

Handles zero, one, perfect association, and extreme tails

Returns mathematically defined boundary results and clearly flags undefined cases.

Transparency10 points

Formula and assumptions visible

Formula, variable definitions, assumptions, and calculation path are visible.

Precision8 points

Sufficient significant digits

Shows enough digits to reproduce decisions and allows copying full precision.

Interpretation8 points

Decision wording is statistically correct

Separates evidence, magnitude, and causation; reports correct tail and null.

Reproducibility5 points

Inputs and output can be retained

Shareable URL, export, or copyable input/output is available.

Accessibility3 points

Keyboard/mobile/readability

Usable by keyboard and mobile with readable labels.

Versioning2 points

Method/version/date identified

Calculator version or reviewed date is visible.

Evidence, not advertising

Published candidate scorecards

Only completed evaluations published by an administrator appear here.

No candidate scorecards have been published yet.

This is intentional: a calculator should not receive a verification label before its evidence has been reviewed.

Reusable release files

Download OSCVB-2026

The data and documentation are versioned so teachers, developers, and reviewers can reproduce the same evaluation.

ZIPComplete OSCVB-2026 ZIP release66.8 KB XLSXFormatted benchmark workbook45.3 KB CSV120 benchmark test vectors22.2 KB CSVVerified reference results72.7 KB CSVCandidate results template6.4 KB CSV100-point transparency rubric2.1 KB MDResearch methodology9.5 KB CFFCitation metadata779.0 B

Recommended citation

Cite the benchmark

Salar Cafe. (2026). Open Statistical Calculator Verification Benchmark 2026 (Version 1.0.0) [Data set and software benchmark]. https://onlineinternetcafe.com.

Verification statement

Use only after testing

A calculator may state that it was tested against OSCVB only when the candidate version, test date, cases attempted, failures, and method variants are publicly documented.

Tested against OSCVB-2026 v1.0.0 — see the published scorecard.

Questions

Benchmark FAQ

Does OSCVB prove that a calculator is always correct?

No. It verifies the included cases for the tested version. It does not cover every possible dataset, method, or future software change.

Can developers reuse the test vectors?

Yes. The benchmark data and documentation are released under CC BY 4.0, while the included code is released under the MIT License.

Why are invalid inputs included?

Because returning a polished but meaningless number can be more harmful than displaying an error. Safe refusal is part of statistical correctness.

How are corrections handled?

Published releases remain immutable. A corrected value requires a new patch version with the affected case IDs documented.