Attempt to Teaching the integration of large language models into the data science workflow

Presenter: Dr. Emi Tanaka

Australian National University

16th July 2026

Acknowledgement

Thank you to Jiajia Li for evaluating text analysis tasks using various large language models

Professional development online workshops

Large language models for the Statistical Society of Australia

Workshop 1

Workshop 2

Workshop 3

Workshop 4

• Workshops focus on interfacing LLMs via R code:
  • Assumes proficiency in basic R programming
• LLMs and Agentic AI are evolving at an accelerated pace
• Chat interface with natural language is becoming the everyday standard

How to use AI for data science?

• API via provider
  • OpenAI
  • Anthropic
  • Google
  • Microsoft
• Local AI models via Ollama

• ChatGPT
• Claude
• Gemini
• Copilot

• OpenAI Codex
• Claude Code
• Cursor
• GitHub Copilot

• Snowflake Cortex Analyst
• Databricks Mosaic AI
• Posit AI (previously: Positron AI and Databot)

Career trajectory in statistics

Technical Contributor Management & Leadership

Industry
Track

Graduate Data Analyst

Senior Data Analyst

Analytics Manager

Director of Analytics

Academic
Track

PhD Student
Postdoctoral Fellow

Lecturer
Assistant Professor

Associate Professor

Professor

• Education in statistics prepares you to be a technical contributor
• But not necessarily to lead analytic work

Drawing parallels to management skills

Leading a data science team

Managing AI analysts in the classroom

Plan and set priorities

Team management
(recruitment, wellbeing, leave, negotiation, etc)

Set up AI

Communicate with the team regularly

Give clear and specific instructions to AI

Review work, train and mentor team members

Review work and give other instructions to guide work

Make decisions under uncertainty

Make decisions under uncertainty

Report outcomes to stakeholders

Report outcomes to peers or teaching team

Shouldn’t you crawl before you walk?

Before you can walk lead, you need:

1. To have mastery of relevant technical knowledge
2. Experience applying the knowledge
3. Time to learn

But:

• Learning is not a linear path
• Assuming an identity can facilitate learning¹

1. Ibarra (1999) Provisional Selves: Experimenting with Image and Identity in Professional Adaptation. Administrative Science Quarterly

We live in a pivotal time

• Anyone now can get a team of data analysts on hand
  And their names include: ChattyG¹, Claude, Copilot, Gemini, and so on

• Zhu et al. (2024)² found best performance of GPT-4o at 65% based on an evaluation dataset consisting of over 11,000 examples for statistical tasks (mainly descriptive statistics and hypothesis testing)
• But latest models are likely much better at it…

1. Name credit to Jo Potts

2. Zhu et al. (2024) Are Large Language Models Good Statisticians? Advances in Neural Information Processing Systems
   Thanks to Bronwen Whiting for letting me know about this article!

Salary survey

Context

• A survey on Ask A Manager gathered salary and respondent information from 28,083 participants
• job_title and country are required free text entries in the survey

timestamp	age	industry	job_title	job_title_context	salary_annual	salarly_additional	currency	currency_other	salary_context	country	state	city	experience_overall	experience_in_field	education	gender	race
2021-04-27 11:02:10	25-34	Education (Higher Education)	Research and Instruction Librarian	NA	55000	0	USD	NA	NA	United States	Massachusetts	Boston	5-7 years	5-7 years	Master's degree	Woman	White
2021-04-27 11:02:22	25-34	Computing or Tech	Change & Internal Communications Manager	NA	54600	4000	GBP	NA	NA	United Kingdom	NA	Cambridge	8 - 10 years	5-7 years	College degree	Non-binary	White
2021-04-27 11:02:38	25-34	Accounting, Banking & Finance	Marketing Specialist	NA	34000	NA	USD	NA	NA	US	Tennessee	Chattanooga	2 - 4 years	2 - 4 years	College degree	Woman	White
2021-04-27 11:02:41	25-34	Nonprofits	Program Manager	NA	62000	3000	USD	NA	NA	USA	Wisconsin	Milwaukee	8 - 10 years	5-7 years	College degree	Woman	White
2021-04-27 11:02:42	25-34	Accounting, Banking & Finance	Accounting Manager	NA	60000	7000	USD	NA	NA	US	South Carolina	Greenville	8 - 10 years	5-7 years	College degree	Woman	White
2021-04-27 11:02:46	25-34	Education (Higher Education)	Scholarly Publishing Librarian	NA	62000	NA	USD	NA	NA	USA	New Hampshire	Hanover	8 - 10 years	2 - 4 years	Master's degree	Man	White
2021-04-27 11:02:51	25-34	Publishing	Publishing Assistant	NA	33000	2000	USD	NA	NA	USA	South Carolina	Columbia	2 - 4 years	2 - 4 years	College degree	Woman	White

Beginner task country distribution

• Question: What is the country representation in the survey?
• This task requires standardising the country variable and:
  • Is based on a real data
  • Has a low barrier to understanding the process
  • Has some uncertain decision-making
  • Revises skills in data wrangling and string manipulation
  • Is a very tedious task
    • actual task is menial,
    • provides little intellectual challenge, and
    • most would be happy to automate with AI

ChatGPT

GPT-5.5

Chat interface alone doesn’t have agency to run code

salary.csv

Please read the attached salary data and standardise the country variable, e.g., such that US is USA. Look at the city or state variable to determine the correct country.

I’ve cleaned and standardised the country variable by:

• Converting all common variants of the United States to USA, including:
• US, Us, us
• USA, Usa, usa
• U.S.A.
• United States, United states, united states
• United States of America
• Converting all common variants of the United Kingdom to UK, including:
• United Kingdom
• UK, Uk

The cleaned file is available here:

📄 salary_cleaned.csv

If you’d like, I can also perform a more comprehensive standardisation using the city and state columns to:

• infer missing countries,
• correct misspellings (e.g. “The Netherlands” → “Netherlands”),
• standardise all countries to ISO-3 codes (e.g. USA, GBR, AUS, CAN) or a consistent set of country names.

Please provide me a code of how you did this in R

Here’s a tidyverse approach that standardises the country names and then uses the city and state variables to infer the country where it is missing or ambiguous.

library(dplyr)
library(stringr)
library(countrycode)
library(readr)

# Read data
salary <- read_csv("salary.csv")

salary_clean <- salary %>%
  mutate(
    # Remove leading/trailing whitespace
    country = str_trim(country),

    # Standardise common variants
    country = case_when(
      str_to_lower(country) %in% c(
        "us", "usa", "u.s.a.", "u.s.", "united states",
        "united states of america"
      ) ~ "USA",

      str_to_lower(country) %in% c(
        "uk", "united kingdom", "great britain"
      ) ~ "UK",

      TRUE ~ country
    ),

    # Fill missing/blank country using state
    country = case_when(
      (is.na(country) | country == "") &
        state %in% state.abb ~ "USA",

      (is.na(country) | country == "") &
        state %in% state.name ~ "USA",

      (is.na(country) | country == "") &
        state %in% c(
          "NSW","VIC","QLD","WA","SA","TAS","NT","ACT"
        ) ~ "Australia",

      TRUE ~ country
    ),

    # Fill missing/blank country using city
    country = case_when(
      (is.na(country) | country == "") &
        city %in% c(
          "New York","Los Angeles","Chicago","Seattle",
          "Boston","Austin","San Francisco","San Diego",
          "Houston","Atlanta","Denver","Phoenix"
        ) ~ "USA",

      (is.na(country) | country == "") &
        city %in% c(
          "Sydney","Melbourne","Brisbane","Perth",
          "Adelaide","Canberra","Hobart","Darwin"
        ) ~ "Australia",

      (is.na(country) | country == "") &
        city %in% c(
          "London","Manchester","Birmingham","Leeds",
          "Liverpool","Edinburgh","Glasgow"
        ) ~ "UK",

      TRUE ~ country
    ),

    # Convert everything else to ISO-3 codes where possible
    country = countrycode(
      country,
      origin = "country.name",
      destination = "iso3c",
      warn = FALSE
    ),

    # Keep UK as UK rather than GBR if preferred
    country = recode(country, GBR = "UK")
  )

write_csv(salary_clean, "salary_cleaned.csv")

A more robust approach

Rather than hard-coding cities and states, I would normally use countrycode together with maps, world.cities, or geonames to infer countries from cities. That scales much better to thousands of locations. If your salary dataset is the one from the Ask a Manager survey (≈28,000 responses), I’d recommend that approach over maintaining lists of city names.

Posit AI

Claude Sonnet 5

Agentic AI can be aware of the code environment and execute code

Examining code

Review work

By ChatGPT

library(dplyr)
library(stringr)
library(countrycode)
library(readr)

# Read data
salary <- read_csv("salary.csv")

salary_clean <- salary %>%
  mutate(
    # Remove leading/trailing whitespace
    country = str_trim(country),

    # Standardise common variants
    country = case_when(
      str_to_lower(country) %in% c(
        "us", "usa", "u.s.a.", "u.s.", "united states",
        "united states of america"
      ) ~ "USA",

      str_to_lower(country) %in% c(
        "uk", "united kingdom", "great britain"
      ) ~ "UK",

      TRUE ~ country
    ),

    # Fill missing/blank country using state
    country = case_when(
      (is.na(country) | country == "") &
        state %in% state.abb ~ "USA",

      (is.na(country) | country == "") &
        state %in% state.name ~ "USA",

      (is.na(country) | country == "") &
        state %in% c(
          "NSW","VIC","QLD","WA","SA","TAS","NT","ACT"
        ) ~ "Australia",

      TRUE ~ country
    ),

    # Fill missing/blank country using city
    country = case_when(
      (is.na(country) | country == "") &
        city %in% c(
          "New York","Los Angeles","Chicago","Seattle",
          "Boston","Austin","San Francisco","San Diego",
          "Houston","Atlanta","Denver","Phoenix"
        ) ~ "USA",

      (is.na(country) | country == "") &
        city %in% c(
          "Sydney","Melbourne","Brisbane","Perth",
          "Adelaide","Canberra","Hobart","Darwin"
        ) ~ "Australia",

      (is.na(country) | country == "") &
        city %in% c(
          "London","Manchester","Birmingham","Leeds",
          "Liverpool","Edinburgh","Glasgow"
        ) ~ "UK",

      TRUE ~ country
    ),

    # Convert everything else to ISO-3 codes where possible
    country = countrycode(
      country,
      origin = "country.name",
      destination = "iso3c",
      warn = FALSE
    ),

    # Keep UK as UK rather than GBR if preferred
    country = recode(country, GBR = "UK")
  )

write_csv(salary_clean, "salary_cleaned.csv")

clean_country.R by Posit AI

library(tidyverse)

# ---- Load data ----
df <- read_csv("data/salary.csv", show_col_types = FALSE)

us_state_set <- c(state.name, "District of Columbia")

# Country/place names that look like plausible "countries" after cleaning
# but are actually US states/cities mistakenly entered in the country field.
known_us_places <- c("california", "san francisco", "hartford", "virginia")

# Free-text / garbage tokens that are not country names at all.
known_garbage <- c(
  "is", "u.a.", "ua", "uxz", "y", "united y", "currently finance",
  "contracts", "policy", "global", "international", "europe", "africa",
  "remote", "na", "n/a"
)

# ---- Standardization function ----
# Order of operations matters:
#   1. Match well-known country name variants (regex-based) for the most
#      common countries in the data (USA, UK, Canada, NZ, Netherlands, Ireland).
#   2. For anything unmatched, treat the (cleaned/title-cased) country text
#      itself as the country name, IF it looks plausible (not a sentence,
#      not a known US place, not garbage, not literally a US state name).
#   3. Only as a last resort — if the country text still isn't resolved —
#      fall back to inferring "USA" from the `state` column (state is a
#      valid US state name). This must come last so that rows with a
#      clearly-named foreign country (e.g. country = "France", state =
#      "Alaska" from a bad state selection) are NOT overwritten with "USA".
standardize_country <- function(country, state) {
  x <- str_trim(country)
  x_low <- str_to_lower(x)

  out <- rep(NA_character_, length(x))

  is_us <- str_detect(
    x_low,
    "united stat|unite states|unites states|u\\.?\\s?s\\.?a?\\.?$|^usa?$|^us$|^america$|🇺🇸|isa|i\\.s\\."
  ) | str_detect(x_low, "^u\\.?s\\.?a?\\.?\\s*$")

  is_uk <- str_detect(
    x_low,
    "united kingdom|^uk$|^u\\.k\\.?$|england|scotland|wales|britain|northern ireland"
  ) & !str_detect(x_low, "new zealand")

  is_canada <- str_detect(x_low, "canad")
  is_nz <- str_detect(x_low, "new.?zealand|^nz$|aotearoa")
  is_neth <- str_detect(x_low, "netherlands")
  is_ireland <- str_detect(x_low, "^ireland$|^ireland,|republic of ireland") &
    !str_detect(x_low, "northern")

  out[is_uk] <- "United Kingdom"
  out[is_canada] <- "Canada"
  out[is_nz] <- "New Zealand"
  out[is_neth] <- "Netherlands"
  out[is_ireland] <- "Ireland"
  out[is_us] <- "USA"

  # Treat remaining unmatched country text as a standalone country name,
  # if it looks plausible.
  still_na <- is.na(out)
  candidate <- x[still_na]
  cand_low <- str_to_lower(candidate)

  plausible <-
    str_length(candidate) <= 25 &
    !str_detect(candidate, "[0-9]") &
    !str_detect(
      cand_low,
      "\\bbut\\b|\\bwork\\b|\\bi \\b|\\bthe \\b|\\bfor \\b|remote|govt|government|based|company|contract|bonus|raise|withheld"
    ) &
    !(cand_low %in% known_us_places) &
    !(cand_low %in% known_garbage) &
    !(str_to_title(cand_low) %in% state.name)

  cleaned <- str_to_title(cand_low)
  out[still_na][plausible] <- cleaned[plausible]

  # Last resort: infer USA from `state` only for rows still unresolved.
  still_na2 <- is.na(out)
  us_by_state <- still_na2 & state %in% us_state_set
  out[us_by_state] <- "USA"

  out
}

# ---- Apply standardization ----
df <- df |>
  mutate(country_clean = standardize_country(country, state))

# ---- Manual fixes for residual typos / alternate spellings not caught by regex ----
df <- df |>
  mutate(country_clean = case_when(
    country_clean %in% c("Canda", "Csnada", "Can")                          ~ "Canada",
    country_clean %in% c("Australi", "Australian")                          ~ "Australia",
    country_clean %in% c("Englang", "United Kindom", "Unites Kingdom",
                          "London")                                         ~ "United Kingdom",
    country_clean %in% c("Danmark")                                        ~ "Denmark",
    country_clean %in% c("Nederland", "Nl")                                ~ "Netherlands",
    country_clean %in% c("Luxemburg")                                      ~ "Luxembourg",
    country_clean %in% c("Hong Konh")                                      ~ "Hong Kong",
    country_clean %in% c("Uae", "United Arab Emirates")                    ~ "United Arab Emirates",
    country_clean %in% c("Mainland China")                                 ~ "China",
    country_clean %in% c("Brasil")                                        ~ "Brazil",
    country_clean %in% c("México")                                        ~ "Mexico",
    country_clean %in% c("Panamá")                                        ~ "Panama",
    country_clean %in% c("Cote D'ivoire")                                 ~ "Cote d'Ivoire",
    str_detect(country_clean, "^Usa")                                      ~ "USA",
    country_clean %in% c("Japan, Us Gov Position")                        ~ "Japan",
    TRUE ~ country_clean
  ))

# ---- Sanity checks ----
n_distinct(df$country_clean)      # ~97 distinct standardized values (down from 318)
sum(is.na(df$country_clean))      # rows left NA: unresolvable free text / garbage

df |> count(country_clean, sort = TRUE) |> print(n = 20)

# ---- Save ----
write_csv(df, "data/salary_clean.csv")

Decisions, decisions, decisions

• Entries shown where one of the cleaned names is incorrect

• Removing unidentifiable countries:
  • ChatGPT got 98.2% correct
  • Posit AI got 99.8% correct

Discussion points in class

How do we assess it was correct?

Encourage sanity checks

Why don’t we have AI directly give us the standardised answers? Why get code?

Capturing the process as code allows us to reproduce the results and review the work

Was it ethical to upload or expose the data to the AI? Was all the cleaning by AI okay?

Decisions are made sometimes under uncertainty

Takeaway messages

Technical Contributor Management & Leadership

• AI “analysts” are accelerating the need for management skills
• Psychologically reframing AI analysts as “junior” analysts whose work requires review, while holding learners accountable for the final decisions, encourages learners to adopt a managerial perspective
• This broadens opportunities to discuss in the classroom the role of data analysts from a technical coder to that of a manager

Technical Contributor Management & Leadership

The slides, written using Quarto and Quarto extensions by Emil Hvitfeldt, can be found at emitanaka.org/slides/ICOTS2026