Statistics and Experimental Reasoning
Statistics and experimental reasoning begin with the question being estimated and the process that produced the observations. Define the population, unit, comparison, and outcome before choosing a test. A carefully calculated number can still answer the wrong question.
Use statistics to describe what the evidence supports and what remains uncertain. Separate effect size, uncertainty, practical consequence, and causal interpretation. None becomes unnecessary because a result crosses a conventional significance threshold.
TL;DR
- State the estimand, population, unit, and measurement process.
- Distinguish random sampling from random assignment.
- Report effect estimates and uncertainty with their assumptions.
- Interpret p-values under the specified model, not as hypothesis probabilities.
- Preserve planned analyses, missing data, and alternative explanations.
Quick Example
This fictional example enumerates all twenty ways to assign six fixed observations into two equal groups. Under its declared exchangeability null and absolute-difference statistic, the two-sided tail proportion is 0.1.
The observed difference is three units, but the calculation does not establish that a real intervention caused it. The allocation model is hypothetical. Additional local arithmetic checked a symmetric t-based mean interval on another tiny fictional sample; no real sampling or experiment occurred.
Core Concepts
The American Statistical Association explains that a p-value describes incompatibility with a specified statistical model. It is not the probability that the hypothesis is true and does not measure effect size or practical importance. Complete reporting and reasoning remain necessary. ASA statement.
NIST's mean-interval guidance describes uncertainty across repeated applications of an interval procedure and presents a t-based interval under the relevant assumptions. A 95 percent confidence interval is not a 95 percent probability statement about the fixed parameter within one realized frequentist interval. NIST confidence limits.
An estimand is the quantity you intend to learn: perhaps a mean difference for a defined population over a specified period. An estimator is the rule used to calculate an answer from data. Measurement and design determine whether that rule is relevant.
Match the evidence to the question
This editorial design map organizes the analysis. It is not a universal study protocol or a substitute for qualified statistical review when consequences warrant it.
Separate sampling, assignment, and repeated observations
Random sampling concerns how units enter the sample. Random assignment concerns how an intervention is allocated among eligible units. The first can support population estimation under its design; the second can support causal comparisons under the experiment's assumptions. One does not automatically supply the other.
Identify the independent unit. Ten thousand events from twenty devices are not necessarily ten thousand independent observations. Repeated measures, shared environments, and clustered assignment can change the uncertainty calculation and the design needed for a valid comparison.
Document exclusions and missing outcomes. If one arm or subgroup loses more observations, the analyzed sample may no longer support the intended comparison without additional assumptions. A complete-case calculation is still a calculation, but its interpretation needs the missingness context.
Use representative time windows and measurement rules. A weekly process, rare failure, or seasonal population can be poorly represented by a convenient short interval. More decimal places do not repair absent coverage.
Interpret estimates and uncertainty together
Report the effect estimate in meaningful units and identify the comparison. A percentage-point difference and a relative percentage change are different quantities. Preserve the denominator and baseline so the reader can understand the scale.
Choose an interval or other uncertainty summary appropriate to the design and estimand. A confidence interval for a mean, a prediction interval for a future observation, and an interval for a model parameter answer different questions. Label them rather than treating all lower/upper bounds as interchangeable.
Do not interpret a wide interval as proof of no effect. It can indicate that the evidence is insufficiently precise to distinguish important possibilities. Likewise, a narrow interval can be precisely biased if the sampling or measurement process misses the target.
For the toy example, the 0.1 tail proportion does not erase the three-unit observed difference. It describes that statistic under a declared allocation model with only twenty possible assignments. Whether the difference matters or generalizes is outside the example.
Plan tests and comparisons before interpreting them
State the primary outcome, comparison, directionality, and analysis plan in advance where feasible. If many outcomes, subgroups, thresholds, or stopping points are explored, disclose that exploration and use an appropriate strategy for the resulting selection problem.
Separate exploratory findings from confirmatory evidence. Exploration is useful for discovering mechanisms and future questions. The problem is presenting a selected result as if it were the only prespecified test performed.
Avoid repeatedly checking a conventional fixed-sample test and stopping only when it becomes significant. If interim decisions are needed, use a design and analysis that account for them. The appropriate method depends on the actual study, not a universal dashboard rule.
Preserve null and inconclusive results in the evidence record. Selective reporting can make a collection of individually correct calculations tell a misleading story. Describe what was measured and analyzed, not just the result most convenient to publish.
Best Practices
Write a short analysis contract containing the question, population, unit, outcome, estimator, assumptions, uncertainty method, and decision context. Include what observation would change the conclusion.
Use reproducible calculations and meaningful sanity checks. Verify units, denominators, grouping, missing values, and assignment rules before interpreting a statistical test. A library's successful execution does not establish that the input design was appropriate.
Seek qualified statistical and domain review when the result drives consequential action. Keep the reasoning transparent enough for a reviewer to assess both the mathematical procedure and the assumptions connecting it to the question.
Common Mistakes
Bad: Describe p = 0.1 as a ten-percent probability that the null is true.
Correct: Explain the tail calculation under the specified model and statistic, without assigning a posterior probability it did not calculate.
Bad: Treat a non-significant result as proof of equivalence.
Correct: Assess uncertainty and the appropriate equivalence or decision design if that is the actual question.
Bad: Count correlated rows as independent experimental units.
Correct: Match analysis and uncertainty to sampling, assignment, clustering, and repeated measurement.
FAQ
Does a larger sample remove bias?
Not automatically. It can reduce some sampling variability while preserving selection, measurement, or design bias. Examine how observations were obtained.
Is a p-value useless?
No. It can be useful within a suitable design and complete analysis. It becomes misleading when treated as effect size, hypothesis probability, or the whole decision.
What would reverse a statistical conclusion?
A changed estimand, invalid independence assumption, important missingness, or evidence of measurement error can change interpretation even when the arithmetic remains correct. Record those conditions explicitly.
Related Topics
- Data Science and Machine Learning — Connect estimates to modeling choices.
- Causal Inference and Observational Data — Separate association from intervention.
- Responsible Machine Learning — Keep consequential use and evidence limits visible.