# Open Statistical Calculator Verification Benchmark 2026 (OSCVB-2026) **Version:** 1.0.0 **Release date:** 2026-07-26 **Publisher:** Salar Cafe / onlineinternetcafe.com **Data license:** Creative Commons Attribution 4.0 International **Code license:** MIT ## Abstract The Open Statistical Calculator Verification Benchmark 2026 is a reusable, openly licensed set of test vectors for evaluating browser-based statistical calculators and educational analysis tools. The release contains 120 cases across 12 method families. Of these, 96 cases require numeric results and 24 cases require safe rejection or an explicit undefined-result message. Each numeric reference result is produced through two computational paths: a direct formula or exact enumeration, and a recognized scientific-computing implementation. The benchmark is designed to expose errors that ordinary worked examples often miss, including hidden method defaults, incorrect tail probabilities, sample-versus-population divisor mistakes, continuity-correction ambiguity, unstable calculations at large offsets or tiny scales, and false numeric output for invalid inputs. ## Why this benchmark is different Many software comparisons ask whether Python, R, SPSS, and Excel reach similar conclusions on a small collection of ordinary datasets. That question is useful, but it does not provide a reusable conformance test for online calculators. OSCVB-2026 instead defines exact inputs, expected outputs, method variants, tolerance rules, and required error behavior. It can therefore be rerun whenever a calculator changes, a browser library is upgraded, or a new tool enters the market. The benchmark is not a popularity ranking. It does not score design taste, advertising, domain authority, or brand recognition. It measures whether a tool implements a named statistical task correctly and explains enough of its method for another person to reproduce the result. ## Scope Version 1.0.0 includes the following twelve method families: 1. Descriptive statistics. 2. Standard normal probabilities. 3. Binomial probabilities. 4. One-sample t tests. 5. Welch independent-samples t tests. 6. Paired-samples t tests. 7. One-way analysis of variance. 8. Pearson product-moment correlation. 9. Simple linear regression. 10. Pearson chi-square tests of independence. 11. Fisher's exact test for a 2 by 2 table. 12. Confidence intervals for a population mean. Each family has ten cases. Eight cases are numerically valid and two are deliberately invalid. Valid cases span ordinary examples, boundary conditions, rounding-sensitive values, large numeric offsets, tiny scales, sparse tables, perfect relationships, and highly unbalanced inputs. Invalid cases test whether the candidate prevents misleading output. ## Reference construction Every valid test case is evaluated through two reference paths. The first path uses explicit formulas, sums of squares, exact combinatorial enumeration, the error function, or hypergeometric probabilities. The second path uses NumPy, SciPy, or Statsmodels. A case receives a reference PASS only when both paths complete and their shared numeric fields agree within an absolute difference of 1e-9. Invalid cases receive a PASS only when both paths reject the input. This dual-path design does not prove that every reference value is infallible, but it substantially reduces the risk of a single implementation error becoming the published answer. It also makes method choices visible. For example, the independent-samples cases explicitly use Welch's unequal-variance t test, the descriptive cases report both sample and population variance, and the chi-square cases identify whether Yates' correction is applied. ## Comparison tolerances Binary floating-point arithmetic cannot represent most decimal fractions exactly. A credible benchmark must therefore avoid demanding character-for-character equality from values that are mathematically equivalent. Candidate outputs should be compared using both absolute and relative error. Recommended grading thresholds: - **Exact:** absolute error no greater than 1e-10, or relative error no greater than 1e-10. - **Accurate:** absolute error no greater than 5e-7, or relative error no greater than 5e-7. - **Acceptable display rounding:** the value displayed by the tool rounds to the correct value at its stated precision and does not change the inferential decision. - **Failure:** the method, sign, tail, statistic, interval, or inferential conclusion is wrong; the discrepancy exceeds tolerance; or invalid input produces an apparently valid answer. P-values close to zero should be retained in scientific notation or with enough digits to distinguish a small positive probability from exactly zero. A tool may display “p < .001” for interpretation, but the downloadable or expanded output should preserve more precision. ## Invalid and undefined inputs Statistical correctness includes knowing when not to calculate. The benchmark contains invalid vectors, impossible probabilities, negative counts, mismatched paired samples, insufficient sample sizes, constant variables, constant paired differences, and zero-margin contingency tables. A candidate passes these cases by refusing to compute, disabling the action, or returning an explicit explanation that the requested statistic is undefined. Returning `0`, `NaN`, a blank result card, or a plausible-looking p-value without explanation is not safe error handling. The distinction matters because users often assume that any number rendered by a calculator is meaningful. ## Transparency score The numeric benchmark is accompanied by a 100-point transparency rubric. Thirty points concern numeric accuracy. The remaining points concern disclosed method variants, invalid-input handling, boundary behavior, formulas and assumptions, displayed precision, interpretation, reproducibility, accessibility, and versioning. This weighting prevents a polished interface with hidden statistical choices from receiving the same evaluation as a transparent, reproducible tool. It also prevents a technically accurate black box from being described as an ideal educational calculator. ## Candidate testing protocol 1. Record the candidate name, exact URL, test date, tester, browser, and device. 2. Reset the candidate between cases so previous inputs cannot affect the result. 3. Enter the values exactly as supplied in `benchmark_test_vectors.csv`. 4. Record every numeric field that corresponds to the expected output. 5. Record the method variant shown by the candidate. 6. Copy or export the output where possible. 7. For invalid cases, record the exact warning or behavior. 8. Compare values against `reference_results.csv`. 9. Apply the transparency rubric independently of the numeric grade. 10. Preserve screenshots for failed, ambiguous, or method-mismatched cases. ## Interpretation of scores A high benchmark score means that the tested version produced correct results for the included methods and documented its choices. It does not mean that the tool is correct for every method it offers, suitable for clinical or legal decisions, or a substitute for a qualified statistician. Recommended labels: - **95–100:** Verified exemplary implementation. - **85–94:** Strong implementation with limited deficiencies. - **70–84:** Generally usable but important limitations remain. - **50–69:** Material accuracy or transparency weaknesses. - **Below 50:** Not recommended for unsupervised statistical use. Numeric accuracy should also be reported separately as cases passed out of cases attempted. A tool should not improve its percentage by refusing to support difficult cases; unsupported cases must remain visible. ## Reproducibility and versioning OSCVB releases are immutable. Corrections should create a new patch version and document the affected case IDs. A candidate evaluation must identify the candidate's visible version or, when no version exists, the test date and a preserved screenshot. The release manifest records the Python, NumPy, SciPy, and Statsmodels versions used to generate this package. The raw test vectors use JSON inside CSV cells so arrays, tables, alternatives, and method flags remain unambiguous. ## Limitations The first release concentrates on widely used univariate and bivariate procedures. It does not yet cover nonparametric rank tests, generalized linear models, survival analysis, mixed models, repeated-measures designs, multiple-comparison corrections, Bayesian methods, or complex survey estimators. These are planned expansion areas. The benchmark also does not automate browser interaction. Candidate results must initially be entered manually or collected by a site-specific test harness. This protects against brittle scraping and allows the tester to evaluate labels, warnings, assumptions, and interpretation that a numeric API alone cannot capture. ## Publication value An open benchmark is more linkable than a self-asserted claim that a calculator is accurate. Teachers can assign the test vectors, developers can use them as regression tests, librarians can evaluate recommended tools, and researchers can cite a stable methodology. Publishing the raw data, exact outputs, licenses, version history, and candidate score sheets gives external websites a concrete resource to reference. ## Future releases Planned expansions include Mann–Whitney, Wilcoxon signed-rank, Kruskal–Wallis, Friedman, sign, median, Levene/Brown–Forsythe, exact proportion tests, odds ratios, relative risks, logistic regression, Poisson regression, power, and sample-size procedures. Each addition should maintain the same dual-reference and invalid-input design.