Basics
If the design of an experiment is faulty, any method of interpretation which makes it out to be decisive must be faulty too.
–Fisher, 1935, Design of Experiments
Data is a resultant reality so any examination of the data to answer questions in a satisfying manner requires understanding of the origin of the data, i.e. how was the data collected and curated? If you don’t know how the data originated, how do you know what population the sample represents? In the absence of understanding the origin of data, your interpretation can be flawed – we’ve seen this with prediction algorithms trained on biased data, election outcome that was unexpected by many because the poll indicated otherwise, retracted articles due to fabricated data, and so on. You can dissect your data using myraids of different analysis but it’ll do no good if your data is rubbish. Knowing the method of data collection and data curation doesn’t guarantee your analysis will be better but it can raise your understanding of the limitation of your interpretation.
Experimental data, as the name suggests, is data collected from an experiment. Unlike other methods of data collection, experimental data originate from a controlled environment where specific factors in the data are within the control of the experimenter. In other words, experimental data are unique in that we have control over how data is created. How to control these factors is what entails as experimental design.
There are many articles and books dedicated to explaining the concepts of experimental designs. This includes the seminal book by Fisher (1935), Bailey (2008), etc. This chapter presents an overview of the statistical concepts that are most pertinent in designing an experiment.
Most experiments are comparative in nature, or more specifically, most experiments involve study of two or more experimental conditions in which the primary interest is to compare the outcome under the different conditions. You’ll therefore find that most experiments in this book are comparative experiments.
At the heart of each experiment, we are ultimately seeking confidence in any conclusion we are making from the analysis of the experimental data. What is perhaps not emphasised enough is that the experiments are human endeavours. Often there are multiple people involved in running experiments and a key challenge is to ensure each individual has sufficient understanding to play their role well. We touch more on this in Chapter 10.
All experiments have a cost whether that be financial, resources, time or other. We can consider the experimental cost as a function of the ability to redo the experiment again. For example, if your experimental resources are based on examination of fossils and the fossils are destroyed in the experimental process, then the cost of the experiment is infinite – you only have one chance to do the experiment – so it’s absolutely essential that you plan, design and execute the experiment well.
Doing an experiment well requires a good understanding of the subject matter so that the potential sources of variation can be controlled or accounted for in the experimental design. Prior to the collection of data, the statistical component of an experiment tends to be focussed on the design of the experiment, i.e. how the treatments are assigned to experimental units under practical constrains to maximise the statistical information of interest. Under this consideration, the statistical problem for experimental design is often reduced to either a randomisation or an optimisation problem and the experimental context may be stripped away in the generation of the design using computer software. We discuss this in Chapter 11, then describe a system that encourages higher order thinking of the experimental design, termed the grammar of experimental designs, in Chapter 7, implemented as the edibble R-package and present how to get started with the edibble system to construct experimental design in Chapter 14.
Before we get into the crux of the basics of the experimental design, consider the three scenarios below. Each scenario describes an experiment where technical details have been reduced so it doesn’t serve as a distraction for now. For each scenario, try to see if you can identify what are the basic components to build the design of the experiment.