The Genesis of Experimentation

👩🏻‍💻 Emi Tanaka @ Monash University

  • emi.tanaka@monash.edu
  • @statsgen
  • github.com/emitanaka
  • emitanaka.org



28th July 2022 Ihaka Lecture

Which vaccines are effective against COVID-19?

The Originator of an Experiment



  • What diet lowers insulin?
  • Is the lockout laws effective to reduce alcohol-fuelled violence?
  • Which brand of washing powder is most effective for cleaning clothes?
  • How much fertilizer should you use for optimal crop yield?



The “domain expert” drives the experimental objective and has the intricate knowledge about the subject area

Stick person images by OpenClipart-Vectors from Pixabay

The Designer of the Experiment

Let there be an experimental design!



The “statistican” creates the experimental design layout after taking into account the statistical and practical constraints.

The Executor of the Experiment



The “technician” carries out the experiment and collects the data.

The Digester of the Experiment



The “analyst” analyses the data after the data is collected.

The actors are purely illustrative

Roles may be fuzzy

In practice:

  • multiple people can take on each role,
  • one person can take on multiple roles, and/or
  • a person in the role may not specialise in that role.

Communications may be fuzzy

Produced by OmniGraffle 7.19.2\n2022-07-27 23:28:41 +0000 comm-model Layer 1 noise personB personA Domain expert’s knowledge & experience Statistician’s knowledge & experience Encode Decode Respond NOISE NOISE Message Message

What diet works best for lowering insulin?

Control
High-sugar
  • Process: Measure insulin level after 4 weeks of subjects under either the control or high-sugar diet
  • Design: Randomise treatments (control or high-sugar diet) to subjects

What diet works best for lowering insulin?

High-fat
Control
High-sugar
  • In practice, experiments can be modified in consultation
  • We need to take into account the experimental structure for appropriate randomisation and/or optimisation

Proposition

Communication is key to get the most out of the experimental data

Design of the (Internal) Domain Specific Language

  • Domain Specific Language (DSL) is a computer language specialised in a particular application domain. Source: Wikipedia
  • R is one of the most popular DSL for statistics. As suggested by the TIOBE index

Experimental design R-packages

  • An R-package is an internal DSL
  • There are over 100 R-packages for the design and analysis of experiments. (In contrast, say Python has only a handful of packages with limited designs of experiments.)
  • Widely used R-packages for constructing the design of experiments often are recipe-based functions with context stripped away.
    More details in Tanaka & Amaliah (2022) Current state and prospects of R-packages for the design of experiments. arxiv.org/abs/2206.07532

agricolae package

“Recipe”-based experimental designs

diets <- c("Control", "High-sugar", "High-fat")

# Completely Randomised Design
agricolae::design.crd(trt = diets, r = 10) 
# Randomised Complete Block Design
agricolae::design.rcbd(trt = diets, r = 10) 

exercise <- c("Yes", "No")

# Factorial design
agricolae::design.ab(trt = c(length(diets), length(exercise)), r = 10) 
# Split-plot design
agricolae::design.split(trt1 = diets, trt2 = exercise, r = 10) 

AlgDesign package

“Recipe”-based optimal designs

diets <- c("Control", "High-sugar", "High-fat")
exercise <- c("Yes", "No")
data <- expand.grid(trt1 = diets, trt2 = exercise, rep = 1:10)

# Optimal design with Federov's exchange algorithm
AlgDesign::optFederov(~ trt1 + trt2 + trt1:trt2, 
                      data = data,
                      nTrials = 40,
                      criterion = "D")

# Optimal design via Monte Carlo
AlgDesign::optMonteCarlo(~ trt1 + trt2 + trt1:trt2, 
                         data = data,
                         nTrials = 40,
                         criterion = "D")

But what are the units??

The Grammar of Experimental Designs

emitanaka.org/edibble-book (work-in-progress)

Definitions from computer science

  • Concrete syntax refers to the external representation of a language
  • Abstract syntax refers to the internal representation of a language

Proposition

Design of the abstract and concrete syntax can aid both intra- and inter-personal communication and this will in turn help the experimental design.

  • Intra-personal: users can clearly express, abstract and clarify the desired experimental design.
  • Inter-personal: others can read the users understanding and intention.

Hypothesis:

  1.   increasing plant diversity leads to increasing soil microbial biomass and enzyme activity,
  2.   higher temperature decreases soil microbial biomassand enzyme activity, and
  3.   higher plant diversity buffers effects of elevated temperature on soil microbialbiomass and enzyme activity.

🌾

MATERIALS AND METHODS

Experimental design

The present study took place in the BAC experiment site established at the Cedar Creek Ecosystem Science Reserve, Minnesota, USA. The site occurs on a glacial outwash plain with sandy soils. Mean temperature during the growing season (April–September) was 15.98°C in 2011 and 17.18°C in 2012. Precipitation during the growing season was 721 mm in 2011. The growing season in 2012 was considerably drier, with 545 mm rainfall.

Experimental plots (9Ă—9 m) were planted in 1994 and 1995 with different plant communities spanning a plant diversity gradient of one, four, and 16 species, which were randomly chosen from the species listed below (Tilman et al. 2001). The grassland prairie species belonged to one of five plant functional groups: C3 grasses (Agropyron smithii Tydb., Elymus canadensis L., Koeleria cristata (Ledeb.) Schult., Poa pratensis L.), C4 grasses (Andropogon gerardii Vitman., Panicum virgatum L., Schizachyrium scoparium (Michx.) Nash, Sorghas-trum nutans (L.) Nash), legumes (Amorpha canescens Pursh., Lespedeza capitata Michx., Lupinus perennis L., Petalostemum purpureum (Vent.) Rydb., Petalostemum villosum Spreng.), nonlegume forbs (Achillea millefolium L., Asclepias tuberosa L., Liatris aspera Michx., Monarda fistulosa L., Soldidago rigida L.), and woody species (Quercus ellipsoidalis E. J. Hill, Quercus macro-carpa Michx.). The individuals of those two woody species (Quercus spp.), which were small in size and rare because of low survival, were removed from all plots in which they occurred in 2010.

In addition to the manipulation of plant diversity,the plots were divided into three subplots (2.5Ă—3.0 m). Heat treatments were applied from March to November each year, beginning in 2009, using infrared lamps 1.8 m above ground emitting 600 W (which caused a 1.5°C increase in soil temperature for vegetation-freesoils) and 1200 W (which caused a 3°C increase; Valpine and Harte 2001, Kimball 2005, Whittingtonet al. 2013) to increase the surface soil temperature of each subplot (see Plate 1). To account for possible shading effects, metal flanges and frames were hungover control subplots. An average across all vegetated plots, temperature manipulations elevated soil temperature at 1 cm depth by 1.18°C in the low warming (+1.5°C) treatment and by 2.69°C in the high warming (+3°C) treatment, and at 10 cm depth temperature by 1.00°C in the low warming (+1.5°C) treatment and by 2.16°C in the high warming (+3°C) treatment.

Soil samples of three subplots in each of 27 experimental plots were taken; due to technical difficulties we could only analyze 66 samples out of 81 existing subplots (monoculture, 10 replicates in ambient +0°C treatment, eight replicates in +1.5°C treatment, nine replicates in +3°C treatment; four species mixture, six replicates in ambient +0°C treatment, six replicates in +1.5°C treatment, seven replicates in +3°C treatment; 16 species mixture, six replicates in ambient +0°C treatment, six replicates in +1.5°C treatment, eight replicates in +3°C treatment). The BAC plots are a representative subset of the plots in the biodiversity experiment E120 at Cedar Creek, which were assembled as random draws of a given number of species from the species pool (Zak et al. 2003). Given low heterogeneity of soil abiotic conditions at the start of the experiment, the experiment was not blocked.

🌾

MATERIALS AND METHODS

Experimental design (condensed version)

  • Experimental plots were planted with different plant communities spanning a plant diversity gradient of one, four, and 16 species, which were randomly chosen from the species listed (5 plant functional groups – 19 species in total)
  • Plots were divided into three subplots
  • Heat treatments were applied to subplots emitting 600 W which caused a 1.5°C increase in soil temperature for vegetation-free soils) and 1200 W (which caused a 3°C increase) (control with 0°C included)
  • Soil samples of three subplots in each of 27 experimental plots were taken
  • Given low heterogeneity of soil abiotic conditions at the start of the experiment, the experiment was not blocked.

This is in fact a split-plot design!

library(edibble)
des1 <- design("Steinauer et al. 2015") %>% 
  set_units(plot = 27,
            subplot = nested_in(plot, 3)) %>% 
  set_trts(
    nspecies = c(1, 4, 16),
    temperature = c("0°C", "1.5°C", "3°C")) %>% 
  allot_trts(   nspecies ~ plot,
             temperature ~ subplot) %>% 
  assign_trts("random") %>% 
  serve_table()
des1
# Steinauer et al. 2015 
# An edibble: 81 x 4
         plot    subplot nspecies temperature
   <unit(27)> <unit(81)> <trt(3)>    <trt(3)>
 1      plot1  subplot1        4        1.5°C
 2      plot1  subplot2        4        3°C  
 3      plot1  subplot3        4        0°C  
 4      plot2  subplot4        16       3°C  
 5      plot2  subplot5        16       1.5°C
 6      plot2  subplot6        16       0°C  
 7      plot3  subplot7        1        3°C  
 8      plot3  subplot8        1        1.5°C
 9      plot3  subplot9        1        0°C  
10      plot4  subplot10       1        1.5°C
# … with 71 more rows

Why edibble?

  • Semantics of the syntactic sugar are aligned with basic terms in experimental design (i.e. code can be used for communication that captures user intention)
  • Experimental structure are built up step-by-step:
    • Abstract syntax allows for an incomplete experimental structure
    • Concrete syntax makes it easy to modify or replace a single step instead of redoing whole structure
  • A number of downstream benefits…

Downstream Benefit #1 Easy-and-quick visualisation

## Only developmental now
# install.packages("remotes")
# remotes::install_github("emitanaka/deggust")
library(deggust)
autoplot(des1)

Downstream Benefit #2 Cutomise visualisation

library(deggust)
autoplot(des1, aspect_ratio = 3/2) +
  # the result is a ggplot object --
  # add ggplot functions on top 
  # as you like to customise the plot!
  theme(
    legend.position = "bottom",
    text = element_text(size = 18),
    plot.background = 
      element_rect(color = "black",
                   linetype = "dashed",
                   fill = "wheat"),
    plot.margin = margin(20, 20, 20, 20),
    plot.subtitle =
      element_text(margin = margin(0, 0, 10, 0))) +
  
  scale_fill_brewer(palette = "Reds")

Downstream Benefit #3 Set record

  • In edibble, records are intended variables, e.g. responses, that will be measured or observed
  • You can set expectations of the record (plausible values) and simulate records, censoring values (default as missing) outside of expectations, or export data with data validation
des1 %>% 
  set_rcrds(microbial_biomass = subplot) %>% 
  expect_rcrds(microbial_biomass >= 0) 
# Steinauer et al. 2015 
# An edibble: 81 x 5
         plot    subplot nspecies temperature microbial_biomass
   <unit(27)> <unit(81)> <trt(3)>    <trt(3)>            <rcrd>
 1      plot1  subplot1        4        1.5°C                 o
 2      plot1  subplot2        4        3°C                   o
 3      plot1  subplot3        4        0°C                   o
 4      plot2  subplot4        16       3°C                   o
 5      plot2  subplot5        16       1.5°C                 o
 6      plot2  subplot6        16       0°C                   o
 7      plot3  subplot7        1        3°C                   o
 8      plot3  subplot8        1        1.5°C                 o
 9      plot3  subplot9        1        0°C                   o
10      plot4  subplot10       1        1.5°C                 o
# … with 71 more rows

Specifying unbalanced designs

MATERIALS AND METHODS

Experimental design (condensed version)

  • Experimental plots were planted with different plant communities spanning a plant diversity gradient of one, four, and 16 species, which were randomly chosen from the species listed (5 plant functional groups – 19 species in total)
des2 <- design("Steinauer et al. 2015 Part II") %>% 
  set_units(plot = 27,
            plant = nested_in(plot, 1:9 ~ 1,
                                  10:18 ~ 4,
                                  19:27 ~ 16)) %>% 
  set_trts(species = 19) %>% 
  allot_trts(species ~ plant) %>% 
  assign_trts("random") %>% 
  serve_table()

options(deggust.nfill_max = 3)
autoplot(des2) + facet_wrap(~plot, nrow = 2)

Leveraging existing work

From computer science:

  1. Make your language consistent
  2. Compose existing language where possible
  3. Limit the number of language elements
  4. Adopt existing notations domain experts use
  5. Enable modularity

And more (see Karsai et al, 2009)

Leveraging existing work

Building on the shoulders of giants:

  • Original creators of R, Ross Ihaka and Robert Gentleman,
  • R Core Team,
  • Tidyverse Team and
  • the wider R community.


  • Experiments are human endeavours.
  • Our understanding is fuzzy but certain frameworks can shed clarity.
  • Remember the “garbage-in garbage-out” principle.