Possible features of a single continuous variable
| Feature | Example | Description |
|---|---|---|
| Asymmetry |  |
The distribution is not symmetrical. |
| Outliers |  |
Some observations are that are far from the rest. |
| Multimodality |  |
There are more than one "peak" in the observations. |
| Gaps |  |
Some continuous interval that are contained within the range but no observations exists. |
| Heaping |  |
Some values occur unexpectedly often. |
| Discretized |  |
Only certain values are found, e.g. due to rounding. |
| Implausible |  |
Values outside of plausible or likely range. |
Numerical features of a single contiuous variables
- A measure of central tendency, e.g. mean, median and mode
Numerical features of a single contiuous variables
- A measure of central tendency, e.g. mean, median and mode
- A measure of dispersion (also called variability or spread), e.g. variance, standard deviation and interquartile range
Numerical features of a single contiuous variables
- A measure of central tendency, e.g. mean, median and mode
- A measure of dispersion (also called variability or spread), e.g. variance, standard deviation and interquartile range
- There are other measures, e.g. skewness and kurtosis that measures "tailedness", but these are not as common as the measures of first two
Numerical features of a single contiuous variables
- A measure of central tendency, e.g. mean, median and mode
- A measure of dispersion (also called variability or spread), e.g. variance, standard deviation and interquartile range
- There are other measures, e.g. skewness and kurtosis that measures "tailedness", but these are not as common as the measures of first two
- The mean is also the first moment and variance, skewness and kurtosis are second, third, and fourth central moments
Numerical features of a single contiuous variables
- A measure of central tendency, e.g. mean, median and mode
- A measure of dispersion (also called variability or spread), e.g. variance, standard deviation and interquartile range
- There are other measures, e.g. skewness and kurtosis that measures "tailedness", but these are not as common as the measures of first two
- The mean is also the first moment and variance, skewness and kurtosis are second, third, and fourth central moments
Significance tests or hypothesis tests
- Testing for H0:μ=μ0 vs. H1:μ≠μ0 (often μ0=0)
- The t-test is commonly used if the underlying data are believed to be normally distributed
Case study 1 2019 Australian Federal Election Part 1/8
Context
- There are 151 seats in the House of Representative for the 2019 Australian federal election
- The major parties in Australia are:
- the Coalition, comprising of the:
- Liberal,
- Liberal National (Qld),
- National, and
- Country Liberal (NT) parties, and
- the Australian Labor party
- the Coalition, comprising of the:
- The Greens party is a small but notable party
Case study 1 2019 Australian Federal Election Part 5/8
What does this graph tell you?
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)skimr::skim(tdf3)## ── Data Summary ────────────────────────## Values## Name tdf3 ## Number of rows 151 ## Number of columns 6 ## _______________________ ## Column type frequency: ## character 3 ## numeric 3 ## ________________________ ## Group variables None ## ## ── Variable type: character ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max empty n_unique whitespace## 1 DivisionID 0 1 3 3 0 151 0## 2 DivisionNm 0 1 4 15 0 151 0## 3 State 0 1 2 3 0 8 0## ## ── Variable type: numeric ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist ## 1 votes_GRN 0 1 9821. 5581. 2744 6555 8676 11532. 45876 ▇▂▁▁▁## 2 votes_total 0 1 99925. 9801. 51009 96372. 100936 105588 116216 ▁▁▁▇▅## 3 perc_GRN 0 1 9.87 5.63 2.89 6.43 8.55 11.4 47.8 ▇▂▁▁▁tdf3 %>% ggplot(aes(perc_GRN)) + geom_histogram(color = "white", fill = "#00843D") + labs(x = "Percentage of first preference votes per division", y = "Count", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 5/8
What does this graph tell you?
- Majority of the country does not have first preference for the Greens
- Some constituents are slightly more supportive than the others
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)skimr::skim(tdf3)## ── Data Summary ────────────────────────## Values## Name tdf3 ## Number of rows 151 ## Number of columns 6 ## _______________________ ## Column type frequency: ## character 3 ## numeric 3 ## ________________________ ## Group variables None ## ## ── Variable type: character ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max empty n_unique whitespace## 1 DivisionID 0 1 3 3 0 151 0## 2 DivisionNm 0 1 4 15 0 151 0## 3 State 0 1 2 3 0 8 0## ## ── Variable type: numeric ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist ## 1 votes_GRN 0 1 9821. 5581. 2744 6555 8676 11532. 45876 ▇▂▁▁▁## 2 votes_total 0 1 99925. 9801. 51009 96372. 100936 105588 116216 ▁▁▁▇▅## 3 perc_GRN 0 1 9.87 5.63 2.89 6.43 8.55 11.4 47.8 ▇▂▁▁▁tdf3 %>% ggplot(aes(perc_GRN)) + geom_histogram(color = "white", fill = "#00843D") + labs(x = "Percentage of first preference votes per division", y = "Count", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 5/8
What does this graph tell you?
- Majority of the country does not have first preference for the Greens
- Some constituents are slightly more supportive than the others
What further questions would you ask?
02:00
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)skimr::skim(tdf3)## ── Data Summary ────────────────────────## Values## Name tdf3 ## Number of rows 151 ## Number of columns 6 ## _______________________ ## Column type frequency: ## character 3 ## numeric 3 ## ________________________ ## Group variables None ## ## ── Variable type: character ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max empty n_unique whitespace## 1 DivisionID 0 1 3 3 0 151 0## 2 DivisionNm 0 1 4 15 0 151 0## 3 State 0 1 2 3 0 8 0## ## ── Variable type: numeric ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist ## 1 votes_GRN 0 1 9821. 5581. 2744 6555 8676 11532. 45876 ▇▂▁▁▁## 2 votes_total 0 1 99925. 9801. 51009 96372. 100936 105588 116216 ▁▁▁▇▅## 3 perc_GRN 0 1 9.87 5.63 2.89 6.43 8.55 11.4 47.8 ▇▂▁▁▁tdf3 %>% ggplot(aes(perc_GRN)) + geom_histogram(color = "white", fill = "#00843D") + labs(x = "Percentage of first preference votes per division", y = "Count", title = "First preference votes for the Greens party")Formulating questions for EDA
- Think broad (open-ended) questions that promotes discussion and divergent thinking
- Polar questions (i.e. the answer is yes or no) are restrictive in exploring the data
- For example,
Is the outlying observation the electoral district that won the seat?
Formulating questions for EDA
- Think broad (open-ended) questions that promotes discussion and divergent thinking
- Polar questions (i.e. the answer is yes or no) are restrictive in exploring the data
- For example,
Is the outlying observation the electoral district that won the seat?
What is characterising the distribution of the percentage of first preference votes for the Greens party?
Formulating questions for EDA
- Think broad (open-ended) questions that promotes discussion and divergent thinking
- Polar questions (i.e. the answer is yes or no) are restrictive in exploring the data
- For example,
Is the outlying observation the electoral district that won the seat?
What is characterising the distribution of the percentage of first preference votes for the Greens party?
- What promotes a higher level of exploration?
Robust measure of central tendency
- Mean is a non-robust measure of location.
- Median is the 50% quantile of the observations
- Trimmed mean is the sample mean after discarding observations at the tails.
- Winsorized mean is the sample mean after replacing observations at the tails with the minimum or maximum of the observations that remain.
| Plot | Mean | Median | Trimmed Mean* | Winsorized Mean* |
|---|---|---|---|---|
| 1 | 0.109 | 0.114 | 0.120 | 0.103 |
| 2 | 0.054 | -0.045 | -0.016 | -0.029 |
| 3 | 1.177 | 0.729 | 0.820 | 0.888 |
| 4 | 0.523 | 0.552 | 0.531 | 0.530 |
| 5 | 0.432 | 0.285 | 0.337 | 0.361 |
| 6 | 2.972 | 2.477 | 2.624 | 2.721 |
* Both trimmed and Winsorized mean trimmed 20% of the tails.
Robust measure of dispersion
- Standard deviation or its square, variance, is a popular choice of measure of dispersion but is not robust to outliers
- Standard deviation for sample x1,...,xn is calculated as √n∑i=1(xi−ˉx)2n−1
- Interquartile range is the difference between 1st and 3rd quartile and is a more robust measure of spread
- Median absolute deviance (MAD) is also more robust and defined as median(|xi−median(xi)|)
Measure of dispersion |
|||||
|---|---|---|---|---|---|
| Plot | SD | IQR | MAD | Skewness | Kurtosis |
| 1 | 0.898 | 1.186 | 0.870 | -0.072 | 3.008 |
| 2 | 0.986 | 1.411 | 1.077 | 0.358 | 2.212 |
| 3 | 1.326 | 1.176 | 0.793 | 1.944 | 7.184 |
| 4 | 0.288 | 0.450 | 0.335 | -0.126 | 1.837 |
| 5 | 0.468 | 0.499 | 0.343 | 1.691 | 6.372 |
| 6 | 2.784 | 5.362 | 2.984 | -0.351 | 1.678 |
Case study 1 2019 Australian Federal Election Part 7/8
We should plot the data!
- The width of the boxplot is proportional to the number of electoral districts in the corresponding state (which is roughly proportional to the population)
What do you notice from this graph?
01:30
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + geom_boxplot(varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Outliers
Outliers are observations that are significantly different from the majority.
- Outliers can occur by chance in almost all distributions, but could be indicative of:
- a measurement error,
- a different population, or
- an issue with the sampling process.
Closer look at the boxplot
- Observations that are outside the range of lower to upper thresholds are referred at times as outliers
- Plotting boxplots for data from a skewed distribution will almost always show these "outliers" but these are not necessary outliers
- Some definitions of outliers assume a symmetrical population distribution (e.g. in boxplots or observations a certain standard deviations away from the mean) and these definitions are ill-suited for asymmetrical distributions
Closer look at the boxplot
- Observations that are outside the range of lower to upper thresholds are referred at times as outliers
- Plotting boxplots for data from a skewed distribution will almost always show these "outliers" but these are not necessary outliers
- Some definitions of outliers assume a symmetrical population distribution (e.g. in boxplots or observations a certain standard deviations away from the mean) and these definitions are ill-suited for asymmetrical distributions
But are there some things we cannot see from boxplots?
Case study 1 2019 Australian Federal Election Part 8/8
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + ggbeeswarm::geom_quasirandom(groupOnX = FALSE, varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 8/8
Now what do you notice from this graph that you didn't notice before?
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + ggbeeswarm::geom_quasirandom(groupOnX = FALSE, varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 8/8
Now what do you notice from this graph that you didn't notice before?
- There are only two electoral districts in NT!
- And only 3 and 5 electoral districts in ACT and TAS, respectively!
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + ggbeeswarm::geom_quasirandom(groupOnX = FALSE, varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 8/8
Now what do you notice from this graph that you didn't notice before?
- There are only two electoral districts in NT!
- And only 3 and 5 electoral districts in ACT and TAS, respectively!
- We have not computed the number of electoral districts for each state so far!
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + ggbeeswarm::geom_quasirandom(groupOnX = FALSE, varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Case study 1 2019 Australian Federal Election Part 8/8
Now what do you notice from this graph that you didn't notice before?
- There are only two electoral districts in NT!
- And only 3 and 5 electoral districts in ACT and TAS, respectively!
- We have not computed the number of electoral districts for each state so far!
Both numerical and graphical summaries can either reveal and/or hide aspects of the data
tdf3 <- df1 %>% group_by(DivisionID) %>% summarise(DivisionNm = unique(DivisionNm), State = unique(StateAb), votes_GRN = TotalVotes[which(PartyAb=="GRN")], votes_total = sum(TotalVotes)) %>% mutate(perc_GRN = votes_GRN / votes_total * 100)tdf3 %>% mutate(State = fct_reorder(State, perc_GRN)) %>% ggplot(aes(perc_GRN, State)) + ggbeeswarm::geom_quasirandom(groupOnX = FALSE, varwidth = TRUE) + labs(x = "Percentage of first preference votes per division", y = "State", title = "First preference votes for the Greens party")Case study 2 Melbourne Housing Prices Part 2/5
Observations arranged by Suburb and Date:
Comparing distribution of room number for observations with missing and non-missing price records:
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))skimr::skim(df2)## ── Data Summary ────────────────────────## Values## Name df2 ## Number of rows 63023 ## Number of columns 13 ## _______________________ ## Column type frequency: ## character 8 ## Date 1 ## numeric 4 ## ________________________ ## Group variables None ## ## ── Variable type: character ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max empty n_unique whitespace## 1 Suburb 0 1 3 18 0 380 0## 2 Address 0 1 7 27 0 57754 0## 3 Type 0 1 1 1 0 3 0## 4 Method 0 1 1 2 0 9 0## 5 SellerG 0 1 1 27 0 476 0## 6 Postcode 0 1 4 4 0 225 0## 7 Regionname 0 1 16 26 0 8 0## 8 CouncilArea 0 1 17 30 0 34 0## ## ── Variable type: Date ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max median n_unique## 1 Date 0 1 2016-01-28 2018-10-13 2017-09-03 112## ## ── Variable type: numeric ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist ## 1 Rooms 0 1 3.11 0.958 1 3 3 4 31 ▇▁▁▁▁## 2 Price 14590 0.768 997898. 593499. 85000 620000 830000 1220000 11200000 ▇▁▁▁▁## 3 Propertycount 0 1 7618. 4424. 39 4380 6795 10412 21650 ▅▇▅▂▁## 4 Distance 0 1 12.7 7.59 0 7 11.4 16.7 64.1 ▇▆▁▁▁df2 %>% select(Suburb, Rooms, Type, Price, Date) %>% arrange(Suburb, Date) %>% visdat::vis_miss()df2 %>% mutate(miss = ifelse(is.na(Price), "Missing", "Recorded")) %>% count(Rooms, miss) %>% group_by(miss) %>% mutate(perc = n / sum(n) * 100) %>% ggplot(aes(as.factor(Rooms), perc, fill = miss)) + geom_col(position = "dodge") + scale_fill_viridis_d() + labs(x = "Rooms", y = "Percentage", fill = "Price")Case study 2 Melbourne Housing Prices Part 2/5
Observations arranged by Suburb and Date:
Comparing distribution of room number for observations with missing and non-missing price records:
- Okay nothing notable as far as I can see
- What next?
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))skimr::skim(df2)## ── Data Summary ────────────────────────## Values## Name df2 ## Number of rows 63023 ## Number of columns 13 ## _______________________ ## Column type frequency: ## character 8 ## Date 1 ## numeric 4 ## ________________________ ## Group variables None ## ## ── Variable type: character ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max empty n_unique whitespace## 1 Suburb 0 1 3 18 0 380 0## 2 Address 0 1 7 27 0 57754 0## 3 Type 0 1 1 1 0 3 0## 4 Method 0 1 1 2 0 9 0## 5 SellerG 0 1 1 27 0 476 0## 6 Postcode 0 1 4 4 0 225 0## 7 Regionname 0 1 16 26 0 8 0## 8 CouncilArea 0 1 17 30 0 34 0## ## ── Variable type: Date ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate min max median n_unique## 1 Date 0 1 2016-01-28 2018-10-13 2017-09-03 112## ## ── Variable type: numeric ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────## skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist ## 1 Rooms 0 1 3.11 0.958 1 3 3 4 31 ▇▁▁▁▁## 2 Price 14590 0.768 997898. 593499. 85000 620000 830000 1220000 11200000 ▇▁▁▁▁## 3 Propertycount 0 1 7618. 4424. 39 4380 6795 10412 21650 ▅▇▅▂▁## 4 Distance 0 1 12.7 7.59 0 7 11.4 16.7 64.1 ▇▆▁▁▁df2 %>% select(Suburb, Rooms, Type, Price, Date) %>% arrange(Suburb, Date) %>% visdat::vis_miss()df2 %>% mutate(miss = ifelse(is.na(Price), "Missing", "Recorded")) %>% count(Rooms, miss) %>% group_by(miss) %>% mutate(perc = n / sum(n) * 100) %>% ggplot(aes(as.factor(Rooms), perc, fill = miss)) + geom_col(position = "dodge") + scale_fill_viridis_d() + labs(x = "Rooms", y = "Percentage", fill = "Price")Case study 2 Melbourne Housing Prices Part 3/5
What can we say from this plot?
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count")Case study 2 Melbourne Housing Prices Part 3/5
What can we say from this plot?
- The housing prices are right-skewed
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count")Case study 2 Melbourne Housing Prices Part 3/5
What can we say from this plot?
The housing prices are right-skewed
There appears to be a lot of outlying housing prices (how can we tell?)
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count")Case study 2 Melbourne Housing Prices Part 4/5
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count") + scale_x_log10()Case study 2 Melbourne Housing Prices Part 4/5
- The x-axis has been log10 transformed in this plot
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count") + scale_x_log10()Case study 2 Melbourne Housing Prices Part 4/5
- The x-axis has been log10 transformed in this plot
- The plot appears more symmetrical now
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count") + scale_x_log10()Case study 2 Melbourne Housing Prices Part 4/5
- The x-axis has been log10 transformed in this plot
The plot appears more symmetrical now
What is a measure of central tendancy here?
With no transformation:
| Mean | Median | Trimmed Mean | Winsorised Mean |
|---|---|---|---|
| $997,898 | $830,000 | $871,375 | $903,823 |
With log transformation (and back transformed to original scale):
| Mean | Median | Trimmed Mean | Winsorised Mean |
|---|---|---|---|
| $874,166 | $830,000 | $847,973 | $859,325 |
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6)) + geom_histogram(color = "white") + labs(x = "Price ($1,000,000)", y = "Count") + scale_x_log10()Case study 2 Melbourne Housing Prices Part 5/5
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6, as.factor(Rooms))) + geom_violin() + geom_boxplot(width = 0.1) + scale_x_log10() + labs(x = "Price ($1,000,000)", y = "# of Rooms")Case study 2 Melbourne Housing Prices Part 5/5
- You can see that drawing separate violin and box plots for each room number show that higher number of rooms generally are pricier
- You could not see this, however, when the data are combined
df2 <- read_csv(here::here("data/MELBOURNE_HOUSE_PRICES_LESS.csv"), col_types = cols( .default = col_character(), Rooms = col_double(), Price = col_double(), Date = col_date(format = "%d/%m/%Y"), Propertycount = col_double(), Distance = col_double()))df2 %>% ggplot(aes(Price/1e6, as.factor(Rooms))) + geom_violin() + geom_boxplot(width = 0.1) + scale_x_log10() + labs(x = "Price ($1,000,000)", y = "# of Rooms")
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Lecturer: Emi Tanaka
ETC5521.Clayton-x@monash.edu
Week 4 - Session 1