What a Small Experiment Cannot Prove

Recognize statistical power thresholds, curvature limitations, and higher-order interaction masking inherent in minimal screening designs.

2026-10-11
Charles Dent
4 min read
Design Limits4 min read
Published: 2026-10-11
What a Small Experiment Cannot Prove

Contents Overview

Article Sections

Limited Power

A small experiment has few runs and therefore limited ability to detect modest effects. Failing to find an effect is not evidence that the effect is absent.

Before running, consider what effect size would matter in practice and whether the planned number of runs and replicates can detect it.

Key Principle

The effect of one factor may depend on the level of another.


Limited Scope

Conclusions apply to the factor ranges, materials and conditions that were tested. A two-level design says nothing about behaviour outside the tested range or about curvature between the levels.

In fractional designs, aliased effects cannot be separated; an apparent main effect may partly reflect an interaction.


Treat Results as a Next Step

Use small experiments to narrow down questions, then confirm promising settings with follow-up runs. A confirmation run at the chosen settings is a simple and valuable check.

Report limitations alongside results so readers do not overextend the conclusions.


Key Takeaways

  • Absence of evidence is not evidence of absence.
  • Results hold only within the tested ranges and conditions.
  • Confirm conclusions with follow-up or confirmation runs.
Charles Dent

Charles Dent

Contributor

Educational article for FactorInteraction Notebook. It explains planning logic and does not report experimental results or statistical significance.

Editorial Method