ETC5521: Exploratory Data Analysis

.info-box.w-50.bg-white[
These slides are viewed best by Chrome or Firefox and occasionally need to be refreshed if elements did not load properly. See <a href=lecture-12A.pdf>here for the PDF <i class="fas fa-file-pdf"></i></a>. 
]

<br>

---

# .monash-blue[ETC5521: Exploratory Data Analysis]

<br>

<h2 style="font-weight:900!important;">Extending beyond the data, what can and cannot be inferred more generally, given the data collection</h2>

.bottom_abs.width100[

Lecturer: *Emi Tanaka*

<i class="fas fa-envelope"></i>  ETC5521.Clayton-x@monash.edu

<i class="fas fa-calendar-alt"></i> Week 12 - Session 1

<br>

]

---

.f1[Today we are going to use surveys in visual inference as a way to think further about what can and cannot be inferred more generally given the data collection]

---

# Recap: Visual inference

.w-70[
* Last week we studied how to use a hypothesis testing framework to assess a feature of a plot by treating a **plot as a test statistic**.

]

.flex[
.w-30[
<img src="images/week12A/demo-data-plot-1.png" width="288" style="display: block; margin: auto;" />

]
.w-70[
<img src="images/week12A/lineup-demo-1.png" width="720" style="display: block; margin: auto;" />
]]

---

# Visual inference test notations

We will use the following notation for the remaining lecture:

* There are `\(\color{red}{n}\)` *independent* participants.
* We assume that each participant have the same detection probability `\((\color{red}{p})\)`, i.e. the probability of selecting the data plot in a lineup with `\(\color{red}{m}\)` plots.
* Let `\(\color{red}{X}\)` be the number of participants who detect the data plot out of `\(n\)` participants.
* We denote `\(\color{red}{x}\)` to be the observed value of `\(X\)`. 
* For visual inference test, we test the hypothesis: `\(\color{red}{H_0: p = 1/m}\)` vs. `\(\color{red}{H_1: p > 1/m}\)`. 
* Under `\(H_0\)`, `\(X \sim B(n, 1/m)\)`.
* Therefore, the visual inference p-value is `\(P(X \geq x) = 1 - P(X \leq x - 1) = \sum_{k = x}^{n}{n\choose k}\frac{(m - 1)^k}{m^n}\)`.
* Recall `pbinom(x, n, p)` is `\(P(X \leq x)\)`.

---

.flex[
.w-65[
<img src="images/week12A/lineup-demo-1.png" width="720" style="display: block; margin: auto;" />

]
.w-35.pa3[

.choice:after {
	content: '';
	position: absolute;
	bottom: 16px;
	left: 25px;
	width: 0;
	height: 0;
	border: 5px solid transparent;
	border-top-color: #00aabb;
	border-bottom: 0;
	margin-left: -20px;
	margin-bottom: -20px;
}
.viewer {
  display: inline-block;
  position: relative;
  margin-top: 20px;
  width: 20px;
}
.viewer-choice {
  display: inline-block;
  margin-left: -20px;
}
</style>

]]

<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;">  5 </th>
   <th style="text-align:right;">  6 </th>
   <th style="text-align:right;">  7 </th>
   <th style="text-align:right;">  8 </th>
   <th style="text-align:right;">  9 </th>
   <th style="text-align:right;">  10 </th>
   <th style="text-align:right;">  11 </th>
   <th style="text-align:right;">  12 </th>
   <th style="text-align:right;">  13 </th>
   <th style="text-align:right;">  14 </th>
   <th style="text-align:right;">  15 </th>
   <th style="text-align:right;">  16 </th>
   <th style="text-align:right;">  17 </th>
   <th style="text-align:right;">  18 </th>
   <th style="text-align:right;">  19 </th>
   <th style="text-align:right;">  20 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 11 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:right;"> 112 </td>
  </tr>
</tbody>
</table>
The data plot is Plot 13 and visual inference p-value is `\(P(X \geq 6)\)` = 0.491 where `\(X \sim B(112, 0.05)\)`.

---

# Data collection

Let's use last week's survey results to discuss about data collection issues

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[1] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>5</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>5</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>5</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>9</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>8</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>9</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>9</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>9</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div>
.f4[
* A person is indexed by the combination of the icon and the color 
* The above colored icons are ordered by the time of response with those that answered it earlier appearing first

]]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[2] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div>

]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[3] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>8</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>8</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>8</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>8</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-injured"></i></span></div></div>

]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[4] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>7</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-injured"></i></span></div></div>

]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[5] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>5</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-injured"></i></span></div></div>
]]

---

# .monash-blue[Lineup] .circle.monash-bg-blue.white[6] In which plot is the pink group higher than the blue group?

]
.w-30.pa3[
<div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-md"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-tie"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>6</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-astronaut"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#C8008F'><i class="fas fa-user-ninja"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>9</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-injured"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-nurse"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#027EB6'><i class="fas fa-user-secret"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-alt"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#746FB2'><i class="fas fa-user-graduate"></i></span></div></div> <div class='viewer-choice'><div class='choice'>10</div><div class='viewer'><span style='color:#6F7C4D'><i class="fas fa-user-ninja"></i></span></div></div>

]]

---

# Do you notice anything from the results?

.overflow-scroll.h-70[
<table class=" lightable-classic lightable-hover" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:right;"> Poll </th>
   <th style="text-align:right;"> Choice </th>
   <th style="text-align:left;">  </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#746FB2"><i class="fas fa-user-md"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-alt"></i></span> <span style="color:#C8008F"><i class="fas fa-user-md"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-injured"></i></span> <span style="color:#C8008F"><i class="fas fa-user-nurse"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-tie"></i></span> <span style="color:#746FB2"><i class="fas fa-user-injured"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#C8008F"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#027EB6"><i class="fas fa-user-tie"></i></span> <span style="color:#027EB6"><i class="fas fa-user-graduate"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#027EB6"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-ninja"></i></span> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#027EB6"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#C8008F"><i class="fas fa-user-tie"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#C8008F"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-ninja"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-alt"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user-md"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 8 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-tie"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#027EB6"><i class="fas fa-user-ninja"></i></span> <span style="color:#746FB2"><i class="fas fa-user-injured"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-tie"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> <span style="color:#746FB2"><i class="fas fa-user-injured"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:left;"> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user-md"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#C8008F"><i class="fas fa-user-tie"></i></span> <span style="color:#027EB6"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-alt"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#C8008F"><i class="fas fa-user"></i></span> <span style="color:#027EB6"><i class="fas fa-user-tie"></i></span> <span style="color:#027EB6"><i class="fas fa-user-injured"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-tie"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#027EB6"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> <span style="color:#C8008F"><i class="fas fa-user"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-md"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#C8008F"><i class="fas fa-user-nurse"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 5 </td>
   <td style="text-align:left;"> <span style="color:#746FB2"><i class="fas fa-user-tie"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user-ninja"></i></span> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#746FB2"><i class="fas fa-user-injured"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-tie"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:left;"> <span style="color:#C8008F"><i class="fas fa-user-secret"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 9 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-injured"></i></span> </td>
  </tr>
  <tr>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:left;"> <span style="color:#027EB6"><i class="fas fa-user-md"></i></span> <span style="color:#027EB6"><i class="fas fa-user-alt"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-tie"></i></span> <span style="color:#C8008F"><i class="fas fa-user-injured"></i></span> <span style="color:#746FB2"><i class="fas fa-user-md"></i></span> <span style="color:#C8008F"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-tie"></i></span> <span style="color:#C8008F"><i class="fas fa-user-astronaut"></i></span> <span style="color:#027EB6"><i class="fas fa-user-graduate"></i></span> <span style="color:#746FB2"><i class="fas fa-user-secret"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-secret"></i></span> <span style="color:#746FB2"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-astronaut"></i></span> <span style="color:#C8008F"><i class="fas fa-user-tie"></i></span> <span style="color:#746FB2"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user-ninja"></i></span> <span style="color:#C8008F"><i class="fas fa-user-graduate"></i></span> <span style="color:#027EB6"><i class="fas fa-user"></i></span> <span style="color:#027EB6"><i class="fas fa-user-astronaut"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user"></i></span> <span style="color:#746FB2"><i class="fas fa-user"></i></span> <span style="color:#C8008F"><i class="fas fa-user-ninja"></i></span> <span style="color:#027EB6"><i class="fas fa-user-nurse"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-nurse"></i></span> <span style="color:#027EB6"><i class="fas fa-user-secret"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-alt"></i></span> <span style="color:#746FB2"><i class="fas fa-user-graduate"></i></span> <span style="color:#6F7C4D"><i class="fas fa-user-ninja"></i></span> </td>
  </tr>
</tbody>
</table>

]

* Let's have a closer look at <span style='color:#C8008F'><i class="fas fa-user-secret"></i></span>.
--
 This person chose Plot 6 for 4 lineups out of 5 answered.
--

* Is this likely to happen by chance? What do you think about the data quality?
--

* Are there any other results you would question about?

---

# Pre-emptive data quality checks in mass-scale visual inference surveys

1. records the demographic of participants (e.g. gender, age and education),
{{content}}
]
--

1. records the choices of multiple lineups from each participant,
{{content}}
--

1. records the reaction time for selecting their choice, and
{{content}}
--

1. includes some lineups with an "obvious" data plot.
{{content}}
--

Post data collection then you can check:

* if the survey was representative of the population by checking the demographic information,
{{content}}
--

* if participant fails to detect the data plot in the "obvious" lineups, it means that they may not be answering sincerely or they did not understand the instructions, which means you may want to remove their data,
{{content}}
--

* if particpant appears to be selecting too quickly, they may not actually be processing the plots appropriately.

---

# How do choices in the lineup design effect the inference?

---

# Lineup with `\(m = 2\)` plots

]
.w-50.pa3[
<div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-md"></i></div></div>

]]

---

# Lineup with `\(m = 4\)` plots

]]

---

# Lineup with `\(m = 4\)` plots

* In fact in visual inference if `\(x\)` people detect out of `\(n\)` people from the same lineup, the frequeny distribution for selection of null plots does *not* change the visual inference `\(p\)`-value nor the power of the lineup
* So all the outcomes below yield the *same* visual inference `\(p\)`-values and power estimate

* Is that an issue?

---

# When null data has features under alternative hypothesis

.panelset[
.panel[.panel-name[📊]
<img src="images/week12A/cheat-plot-1.png" width="432" style="display: block; margin: auto;" />
]
.panel[.panel-name[data]
.f4[

```r
set.seed(1)
x1 <- rnorm(20, 0, 1)
x2 <- rnorm(20, 1, 1)
df3 <- tibble(x = c(x1, x2),
              group = rep(c("A", "B"), times = c(length(x1), length(x2))))
```
]

]
.panel[.panel-name[R]
.f4[

```r
set.seed(29)
gcheat <- lineup(null_permute("group"), true = df3, n = 4, pos = 3) %>% 
  ggplot(aes(group, x, color = group)) +
  ggbeeswarm::geom_quasirandom() +
  #geom_boxplot() + 
  facet_wrap(~.sample) +
  theme(axis.text = element_blank(),
        axis.title = element_blank(),
        axis.ticks.length = unit(0, "pt")) +
  scale_color_discrete_qualitative() +
  guides(color = "none")
gcheat
```
]
]
]

]
.w-55[

* Occasionally, null data demonstrates features that are more aligned with data generated from the alternative hypothesis 
{{content}}

]]

* For example, the data plot is in Plot 3 for the lineup on the left but Plot 1 demonstrates a bigger difference in the mean (and smaller standard deviation) of the two groups
{{content}}
--

* There is indeed a significant mean difference between the two groups for the data in Plot 3, but this can be overshadowed by null data in Plot 1
{{content}}
--

* This case is extremely rare; in fact, I cheated by generating 100 lineups of the same dimension and took the extreme case. 
{{content}}
--

* While this case is rare, it can happen so we need to becareful in generalising the results based on a test on one lineup

---

# Multiple lineups for the same data plot

.flex[
.w-50[
<img src="images/week12A/unnamed-chunk-36-1.png" width="576" style="display: block; margin: auto;" />

]
.w-50.pa3[
.flex[
.w-50[
**Lineup 1**
<div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div>

**Lineup 3**
<div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div>
]
.w-50[
**Lineup 2**
<div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-ninja"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div>

**Lineup 4**
<div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-secret"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-astronaut"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-tie"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>1</div><div class='viewer'><i class="fas fa-user-injured"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-md"></i></div></div> <div class='viewer-choice'><div class='choice'>2</div><div class='viewer'><i class="fas fa-user-nurse"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user"></i></div></div> <div class='viewer-choice'><div class='choice'>4</div><div class='viewer'><i class="fas fa-user-alt"></i></div></div> <div class='viewer-choice'><div class='choice'>3</div><div class='viewer'><i class="fas fa-user-graduate"></i></div></div>

]]
<style type="text/css">
.h-10 {
  height: 10%;
}
</style>

.overflow-scroll.h-10[
**Lineup 1**
<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 10 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 30 </td>
  </tr>
</tbody>
</table>

**Lineup 2**
<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 25 </td>
  </tr>
</tbody>
</table>

**Lineup 3**
<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 25 </td>
  </tr>
</tbody>
</table>

**Lineup 4**
<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 15 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 25 </td>
  </tr>
</tbody>
</table>

**All together**
<table class=" lightable-classic" style='font-family: "Arial Narrow", "Source Sans Pro", sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
  <tr>
   <th style="text-align:left;"> Choices </th>
   <th style="text-align:right;">  1 </th>
   <th style="text-align:right;">  2 </th>
   <th style="text-align:right;">  3 </th>
   <th style="text-align:right;">  4 </th>
   <th style="text-align:right;"> Total </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Frequency </td>
   <td style="text-align:right;"> 22 </td>
   <td style="text-align:right;"> 14 </td>
   <td style="text-align:right;"> 55 </td>
   <td style="text-align:right;"> 14 </td>
   <td style="text-align:right;"> 105 </td>
  </tr>
</tbody>
</table>

]

]]

---

# What if the positions of the plots has an effect on detection?

.flex[
.w-50[
* If the position of the plots has an effect on detection, we should take this into account as a potential source of variation and consider using position as a "block" in randomisation
* In another words, we should randomise the position of the data plot in the lineup such that each position appears roughly equally number of times for the data plot
* For example, in the lineups on the right, the data plot is highlighted with the red color and you can see each position appears exactly once
]
.w-50[
.flex[
<img src="images/week12A/unnamed-chunk-47-1.png" width="288" style="display: block; margin: auto;" />

]
.flex[

<img src="images/week12A/unnamed-chunk-50-1.png" width="288" style="display: block; margin: auto;" />
]

]]

---

# Statistical significance and practical significance

```r
library(nullabor)
set.seed(1)
sim <- tibble(id = 1:10000000) %>% 
         mutate(y = c(rnorm(n()/2), rnorm(n()/2, 0.001)),
                group = rep(c("A", "B"), each = n()/2))
with(sim, mean(y[group=="A"]) - mean(y[group=="B"]))
```

```
## [1] -0.001443504
```

```r
with(sim, t.test(y[group=="A"], y[group=="B"]))
```

```
## 
## 	Welch Two Sample t-test
## 
## data:  y[group == "A"] and y[group == "B"]
## t = -2.2819, df = 1e+07, p-value = 0.0225
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.0026833804 -0.0002036271
## sample estimates:
##    mean of x    mean of y 
## 0.0001819234 0.0016254271
```

]
.w-30[

* Notice here the real difference in the two groups is small (0.001) here.
* The two groups have a slightly different but the true difference is small, you might not care. 
* The **practical significance** takes into account the effect size.

]

---

# Lineup of small effect difference

<img src="images/week12A/sim-lineup-1.png" width="936" style="display: block; margin: auto;" />
.font_small[For computational reasons, only 10,000 data points for each plot are used above.]

---

# Statistical significance of the data plot

* Unlike conventional hypothesis testing, visual inference p-value depends on:
   * the visual test statistic `\(V\)`,
   * the individuals' visual perceptions,
   * the null generation method,
   * the number of `\(n\)` observers,
   * the size `\(m\)` of the lineup, and 
   * the effect size (or detection probability `\(p\)`). 
* The concept of conventional p-value is difficult for those that are not trained in statistics.
* The lineup is easier to understand to both novices and experts.
* If you reject the null hypothesis, you can find out *why* that might be the case

---

background-size: cover
class: title-slide
background-image: url("images/bg-01.png")

<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

.bottom_abs.width100[

Lecturer: *Emi Tanaka*

<i class="fas fa-envelope"></i>  ETC5521.Clayton-x@monash.edu

<i class="fas fa-calendar-alt"></i> Week 12 - Session 1

<br>

]