library(tidyverse)
library(ellmer)
This demo assumes that you have all the software requirements.
We’ll use LLMs to help us with the following data processing tasks:
- give a sentiment score from text,
- summarise the text,
- cluster the text entries,
- classify the text into particular categories, and
- standardise the data.
Make sure to validate by doing spot checks or otherwise!
LLM may propagate majority view so take caution on its usage, particularly for classification tasks.
1 AirBnB reviews data
The airbnb_reviews data contains AirBnB reviews in Sydney, Australia. We look at the listing “Heart of Rockdale 2” (ID 35428495) with host Jason. The host describes the neighbourhood of the listing as below.
good location easy to access all Sydney has to offer..
This listing has 20 reviews shown below.
airbnb_reviews <- readRDS(url("https://emitanaka.org/workshop-LLM-2025/data/airbnb_reviews.rds"))
reviews <- airbnb_reviews |>
filter(listing_id == 35428495) |>
pull(review_text)
reviews [1] "Excellent place to stay for single person travelling. All necessities are there hair dryer, kitchenette, cutlery, clean towels and so on. Highly recommended."
[2] "Good privacy and self contained."
[3] "Convenient location, east check in and clean space. Would stay again."
[4] "다음날 공항에 가기 전 묵기 좋았습니다<br/>깔끔해서 묵기 편했습니다<br/>하지만 난방기구가 하나도 없어서 저는 밤새 덜덜 떨면서 자야 했습니다...🥲<br/>너무 추운거 빼고는 완벽했습니다"
[5] "Great location, walking distance to train station, cafes, restaurants and Lady Robinson's beach. If you need to work or stream, bring your own internet."
[6] "Good stay overall!"
[7] "Great little single person room, good for short .<br/>It has a kitchenette, clean and usable toilet and shower.<br/>Lots of good food in Rockdale within walking distance"
[8] "Thanks Jason, was perfect for what I needed"
[9] "Clean & tidy place, Convient location with proximity to shops,airport & train."
[10] "Very last minute booking due to delayed flight, very close to airport. Thanks for the accomodation Jason! Highly recommend :)"
[11] "it's a tidy clean home. and Jason is nice . it's just the time i arrived was late and the light is not on. as a girl i'll be a bit scared ,and really suggest jason to change the blind into some thicker material, that blind for girls are not very safe. and could put a inside locker chain. i know it's a female thing. but it will definitely make us feel safer. ty"
[12] "An older style room with older furniture, but very clean and functional. Great communication from the owner."
[13] "Had a great stay. Fantastic location close to the airport and Jason was very accommodating and easy to communicate with."
[14] "Everything was perfect, great host"
[15] "Good basic accommodation for one person needing to make an early start for the airport."
[16] "Excellent for a quick stay"
[17] "Stayed here for 2.5 weeks and was really happy with the place. The unit was tiny, but had all the amenities I needed as a solo traveller. 10 min walk to the ocean and beaches, Rockdale train station, Anytime Fitness, shopping centre and Coles supermarket. I work remotely, so the only thing I wish was better was the wifi, which was pretty spotty most days. I used my wifi hotspot or worked from a café, so that worked for me"
[18] "Super handy to Sydney Airport, very prompt and clear communication, and totally reasonable, simple accommodation with everything that any reasonable maintenance person would happily need."
[19] "thank you, clean convenient and quiet neighbourhood"
[20] "Perfect for one night for solo travellors. All u need in a small bedsit. Can be cold though so either ask for another blanket or take a spare one." 1.1 Sentiment analysis
sentimenter <- chat_ollama(
model = "llama3.1:8b",
system_prompt = "Score the sentiment of the input text from 1 to 10 where 1 is the most negative and 10 is the most positive. Just return the sentiment score only.",
seed = 1,
api_args = list(top_p = 0.1, temperature = 0),
echo = FALSE
)
# for each review, ask the `sentimenter` to score.
sentiment <- map_dbl(reviews, ~as.numeric(sentimenter$chat(.x)))
mean(sentiment)[1] 8.75So the mean sentiment of this listing is 8.75 out of 10. You can see the review text and sentiment score computed by the LLM in Table 1. Do you agree with the scoring? It’s a good idea to do some sample checks that it’s working as intended before processing your data in large scale.
| reviews | sentiment |
|---|---|
| Excellent place to stay for single person travelling. All necessities are there hair dryer, kitchenette, cutlery, clean towels and so on. Highly recommended. | 9 |
| Good privacy and self contained. | 8 |
| Convenient location, east check in and clean space. Would stay again. | 9 |
| 다음날 공항에 가기 전 묵기 좋았습니다 깔끔해서 묵기 편했습니다 하지만 난방기구가 하나도 없어서 저는 밤새 덜덜 떨면서 자야 했습니다…🥲 너무 추운거 빼고는 완벽했습니다 |
6 |
| Great location, walking distance to train station, cafes, restaurants and Lady Robinson’s beach. If you need to work or stream, bring your own internet. | 9 |
| Good stay overall! | 8 |
| Great little single person room, good for short . It has a kitchenette, clean and usable toilet and shower. Lots of good food in Rockdale within walking distance |
9 |
| Thanks Jason, was perfect for what I needed | 10 |
| Clean & tidy place, Convient location with proximity to shops,airport & train. | 9 |
| Very last minute booking due to delayed flight, very close to airport. Thanks for the accomodation Jason! Highly recommend :) | 10 |
| it’s a tidy clean home. and Jason is nice . it’s just the time i arrived was late and the light is not on. as a girl i’ll be a bit scared ,and really suggest jason to change the blind into some thicker material, that blind for girls are not very safe. and could put a inside locker chain. i know it’s a female thing. but it will definitely make us feel safer. ty | 7 |
| An older style room with older furniture, but very clean and functional. Great communication from the owner. | 8 |
| Had a great stay. Fantastic location close to the airport and Jason was very accommodating and easy to communicate with. | 10 |
| Everything was perfect, great host | 10 |
| Good basic accommodation for one person needing to make an early start for the airport. | 8 |
| Excellent for a quick stay | 9 |
| Stayed here for 2.5 weeks and was really happy with the place. The unit was tiny, but had all the amenities I needed as a solo traveller. 10 min walk to the ocean and beaches, Rockdale train station, Anytime Fitness, shopping centre and Coles supermarket. I work remotely, so the only thing I wish was better was the wifi, which was pretty spotty most days. I used my wifi hotspot or worked from a café, so that worked for me | 9 |
| Super handy to Sydney Airport, very prompt and clear communication, and totally reasonable, simple accommodation with everything that any reasonable maintenance person would happily need. | 10 |
| thank you, clean convenient and quiet neighbourhood | 9 |
| Perfect for one night for solo travellors. All u need in a small bedsit. Can be cold though so either ask for another blanket or take a spare one. | 8 |
1.2 Translate text
If you have reviews in multiple languages, you may wish to translate them to a common language. The 4th review was not in English so let’s translate it to English.
reviews[4][1] "다음날 공항에 가기 전 묵기 좋았습니다<br/>깔끔해서 묵기 편했습니다<br/>하지만 난방기구가 하나도 없어서 저는 밤새 덜덜 떨면서 자야 했습니다...🥲<br/>너무 추운거 빼고는 완벽했습니다"translater <- chat_ollama(
model = "llama3.1:8b",
system_prompt = "You have a review in a different language. Translate the input text to English. Just return the translated text.",
seed = 1,
api_args = list(top_p = 0.1, temperature = 0),
echo = FALSE
)
translater$chat(reviews[4]) [1] "The hotel was great the night before we had to leave for the airport. It was clean and comfortable. However, there were no heaters in the room so I shivered all night."And if you did not know what language the review was in, you can ask the LLM to identify the language for you.
translater$chat(c("What language is this?", reviews[4]))[1] "This text is in Korean."1.3 Summarise texts
For summarising the text, we need to parse all reviews. To do this we have to collapse all the review text into one but also have a way to signal that the reviews are separate. To do this we’ll quote each review and separate them by a comma.
review_all <- paste0(paste0('"', reviews,'"'), collapse = ", ")
summariser <- chat_ollama(model = "llama3.1:8b",
system_prompt = "The user will input reviews of a listing. Summarise the input text into one paragraph including the overall sentiment. Just return the summary.",
seed = 1,
api_args = list(top_p = 0.1,
temperature = 0))
summariser$chat(review_all) |> cat()The listing is a simple, self-contained accommodation suitable for single
travelers, with all the necessary amenities such as kitchenette, clean towels,
and hair dryer. The location is convenient, close to public transport, shops,
and beaches, making it ideal for short stays or early departures from Sydney
Airport. While some guests noted that the room can be cold, especially at
night, most reviewers praised the cleanliness, tidiness, and good communication
with the owner. Overall, the sentiment is overwhelmingly positive, with many
guests recommending the listing to others.
The listing is a simple, self-contained accommodation suitable for single travelers, with all the necessary amenities such as kitchenette, clean towels, and hair dryer. The location is convenient, close to public transport, shops, and beaches, making it ideal for short stays or early departures from Sydney Airport. While some guests noted that the room can be cold, especially at night, most reviewers praised the cleanliness, tidiness, and good communication with the owner. Overall, the sentiment is overwhelmingly positive, with many guests recommending the listing to others.What do you think about the summary it produced? Again, do some spot checks!
2 Salary
The salary data contains the survey results from a 2021 Ask a Manager Salary Survey.
We’ll take a 400 random responses to reduce the time for demonstration.
set.seed(2025)
salary <- readRDS(url("https://emitanaka.org/workshop-LLM-2025/data/salary.rds"))
salary_sample <- slice_sample(salary, n = 400)2.1 Clustering
The responders entered the job title by text, as such there are as many job titles as many responders. We may wish to group them thematically so we’ll cluster them using the LLM as below.
In this process, you may like to cluster the group based on a sub-sample (200 below) as LLM have a context window and the long list of all job titles may make the input text unwieldy long.
job_all <- paste0(paste0('"', sample(salary_sample$job_title, 200), '"'), collapse = ", ")
job_grouper <- chat_ollama(model = "llama3.1:8b",
system_prompt = "The input text is a list of job titles from survey responses. Clean the job titles and group similar job titles together. Just return the groups only as a json object.",
seed = 1,
echo = FALSE,
api_args = list(top_p = 0.5,
temperature = 0,
response_format = list(type = "json_object")))
job_groups <- job_grouper$chat(job_all) |>
jsonlite::fromJSON()
job_groups$Management
[1] "Data Engineering Manager"
[2] "Program Director"
[3] "Library director"
[4] "Sr Contracts Manager"
[5] "Director of Customer Experience"
[6] "office manager"
[7] "Payor Support Specialist"
[8] "Head of Devops"
[9] "Senior Program Manager"
[10] "Digital Account Manager"
[11] "Manager"
[12] "Customer Success Manager"
[13] "EA"
[14] "Admin"
[15] "Executive assistant"
[16] "VP Marketing/Merchandising/Creative"
[17] "Sr Manager"
[18] "Human Resource Manager"
[19] "Office Manager"
[20] "HR Coordinator"
[21] "IT Helpdesk Assistant"
[22] "HR Operations Manager"
[23] "Account coordinator"
[24] "Research scientist"
[25] "Government worker"
[26] "Service Manager"
[27] "Marketing Coordinator"
[28] "Consultant"
[29] "Administrative Assistant & Systems Analyst"
[30] "Alliance Manager"
[31] "Marketing Manager"
[32] "Administrative Assistant"
[33] "Manager, Access Services"
[34] "Sr. Account Relationship Manager"
[35] "RN coordinator"
[36] "Digital print technician"
[37] "Benefits & Compensation Manager"
[38] "Director of Development"
[39] "Senior Associate"
[40] "Library Director"
[41] "Team Lead Inbound Scheduling"
[42] "Grants Manager"
[43] "HR Manager"
[44] "Graduate Student"
[45] "Director of Software Engineering"
[46] "Clinical Trainer"
[47] "Quality Manager"
[48] "Veterinary technician supervisor"
[49] "Executive Director"
$`Science and Technology`
[1] "Postdoc" "Communications Logistics Analyst"
[3] "Frontend Engineer" "Director of Data Science"
[5] "Cognitive implementation engineer" "Data Engineer"
[7] "Water Resources Engineer" "Graduatae Assistant"
[9] "Staff Engineering Analyst" "Devops Engineer"
[11] "Machine Learning Engineer" "Senior Software Engineer"
[13] "Senior Engineer" "Senior Research Manager"
[15] "Faculty Manager" "Senior Front End Developer"
[17] "Senior Programmatic Associate" "Digital Director"
[19] "IT Business Analyst" "Statistician"
[21] "Web Developer" "Software Dev"
[23] "Security engineer"
$Education
[1] "Teacher"
[2] "World Language teacher"
[3] "High school teacher"
[4] "EL kindergarten teacher"
[5] "Adjunct professor and exhibiting artist"
[6] "Associate Professor"
[7] "Professor of [Humanities Discipline]"
[8] "Instructor"
$Healthcare
[1] "Psychotherapist" "Financial Controller"
[3] "Director of Research Services" "Technical Writer"
[5] "Forecasting Analyst" "Customer Success Manager"
[7] "RN coordinator" "Licensed Clinical Psychologist"
[9] "Licensed Mental Health Counselor" "Actuary"
[11] "Director of Revenue Cycle & EHR"
$`Finance and Accounting`
[1] "Attorney"
[2] "Policy Analyst"
[3] "Communications Strategist"
[4] "Director of Customer Experience"
[5] "office manager"
[6] "Payor Support Specialist"
[7] "Head of Devops"
[8] "Senior Program Manager"
[9] "Digital Account Manager"
[10] "Manager"
[11] "Customer Success Manager"
[12] "EA"
[13] "Admin"
[14] "Executive assistant"
[15] "VP Marketing/Merchandising/Creative"
[16] "Sr Manager"
[17] "Human Resource Manager"
[18] "Office Manager"
[19] "HR Coordinator"
[20] "IT Helpdesk Assistant"
[21] "HR Operations Manager"
[22] "Account coordinator"
[23] "Research scientist"
[24] "Government worker"
[25] "Service Manager"
[26] "Marketing Coordinator"
[27] "Consultant"
[28] "Administrative Assistant & Systems Analyst"
[29] "Alliance Manager"
[30] "Marketing Manager"
[31] "Administrative Assistant"
[32] "Manager, Access Services"
[33] "Sr. Account Relationship Manager"
[34] "RN coordinator"
[35] "Digital print technician"
[36] "Benefits & Compensation Manager"
[37] "Director of Development"
[38] "Senior Associate"
[39] "Library Director"
[40] "Team Lead Inbound Scheduling"
[41] "Grants Manager"
[42] "HR Manager"
[43] "Graduate Student"
[44] "Director of Software Engineering"
[45] "Clinical Trainer"
[46] "Quality Manager"
[47] "Veterinary technician supervisor"
[48] "Executive Director"
$`Arts and Entertainment`
[1] "TV commercial assistant editor"
[2] "product photographer"
[3] "Games Specialist"
[4] "Solutions Consultant"
[5] "Database Administrator"
[6] "Lead Consultant - Software Development"
[7] "Global Event Manager"
$`Sales and Marketing`
[1] "Digital Account Manager"
[2] "Marketing Coordinator"
[3] "Service Manager"
[4] "Marketing Manager"
[5] "Administrative Assistant & Systems Analyst"
[6] "Alliance Manager"
[7] "Marketing Manager"
[8] "Administrative Assistant"
[9] "Manager, Access Services"
[10] "Sr. Account Relationship Manager"
[11] "RN coordinator"
[12] "Digital print technician"
$`Human Resources`
[1] "Recruiter"
[2] "Lawyer"
[3] "Law Firm Administrator"
[4] "Project Engineer"
[5] "Customer Success Manager"
[6] "EA"
[7] "Admin"
[8] "Executive assistant"
[9] "VP Marketing/Merchandising/Creative"
[10] "Sr Manager"
[11] "Human Resource Manager"
[12] "Office Manager"
[13] "HR Coordinator"
[14] "IT Helpdesk Assistant"
[15] "HR Operations Manager"
[16] "Account coordinator"
[17] "Research scientist"
[18] "Government worker"
[19] "Service Manager"
[20] "Marketing Coordinator"
[21] "Consultant"
[22] "Administrative Assistant & Systems Analyst"
[23] "Alliance Manager"
[24] "Marketing Manager"
[25] "Administrative Assistant"
[26] "Manager, Access Services"
[27] "Sr. Account Relationship Manager"
[28] "RN coordinator"
[29] "Digital print technician"
[30] "Benefits & Compensation Manager"
$Other
[1] "Postdoc" "Communications Logistics Analyst"
[3] "Frontend Engineer" "Director of Data Science"
[5] "Cognitive implementation engineer" "Data Engineer"
[7] "Water Resources Engineer" "Graduatae Assistant"
[9] "Staff Engineering Analyst" "Devops Engineer"
[11] "Machine Learning Engineer" "Senior Software Engineer"
[13] "Senior Engineer" "Senior Research Manager"
[15] "Faculty Manager" "Senior Front End Developer"
[17] "Senior Programmatic Associate" "Digital Director"
[19] "IT Business Analyst" "Statistician"
[21] "Web Developer" "Software Dev"
[23] "Security engineer" 2.2 Classification
Once we identify the classification groups like in Section 2.1 or otherwise, we can classify each job title to one of these groups as below. We’ll only classify 20 observations as this will be time consuming for a demonstration.
groups <- paste0(names(job_groups), collapse = ", ")
job_classifier <- chat_ollama(model = "llama3.1:8b",
system_prompt = paste0("Classify the input text to [", groups,"]. Just return the group only."),
seed = 1,
echo = FALSE,
api_args = list(top_p = 0.5,
temperature = 0))
jobs <- map_chr(salary_sample$job_title[1:20], ~job_classifier$chat(.x))The resulting classification can be seen in Table 2. Do you agree with the classifications? Some I don’t agree with (e.g. Technical Writer grouped as Engineering but most others look good to me).
| Job Title | Group |
|---|---|
| Technical Writer | Engineering |
| Executive assistant | Management |
| Archivist | Other |
| Associate Director of Admissions | Education |
| Senior examiner (claims) | Law |
| Library Director | Education |
| Machine Learning Engineer | Engineering |
| Assistant Director, Finance | Finance |
| Librarian | Education |
| Learning Designer | Education |
| Senior policy analyst | Law |
| Operations Manager | Management |
| Lead Marketing Analyst | Marketing |
| Production manager | Engineering |
| Sr. Executive Assistant | Management |
| Regulatory Manager | Law |
| Finance Director | Finance |
| Assistant manager | Management |
| Solicitor | Law |
| Billing Manager | Finance |
It is best to scan all classification and if there is only a small issue, you can manually correct these. If there are more, you can give some examples of what is the correct classification to LLM and reclassify all.
2.3 Standardise data
In the survey, respondents entered their country by text entry. If you look at the entries below, you’ll notice that there are many variations of the same country, e.g. “United States” and “USA”.
countries <- unique(salary_sample$country)
countries [1] "United States" "US"
[3] "Australia" "USA"
[5] "United states" "Ireland"
[7] "United States of America" "Canada"
[9] "usa" "UK"
[11] "na" "U.S"
[13] "Sweden" "Germany"
[15] "United Kingdom" "Usa"
[17] "U.S." "Spain"
[19] "Uk" "United Stateds"
[21] "Philippines" "Austria"
[23] "New Zealand" "U.S.A."
[25] "United States of america" "England"
[27] "Estonia" "Belgium"
[29] "International" "United State of America"
[31] "France" "Czechia"
[33] "Puerto Rico" "united states" Most standardisation seems correct but the “na” entry was standardised to “Norway” and may require further checking to see if this is indeed correct.
standardiser <- chat_ollama(model = "llama3.1:8b",
system_prompt = "The input text is a list of country names.",
seed = 1,
echo = FALSE)
country_dict <- standardiser$extract_data(countries,
type = type_array("Country",
items = type_object(original = type_string("All the original inputs"),
standard = type_enum("Standardised country names", values = countries::list_countries())))) |>
mutate(standard = as.character(standard))
as_tibble(country_dict) |>
print(n = Inf)# A tibble: 31 × 2
original standard
<chr> <chr>
1 United States United States of America
2 US United States of America
3 Australia Australia
4 USA United States of America
5 United states United States of America
6 Ireland Ireland
7 Canada Canada
8 UK United Kingdom of Great Britain and Northern Ireland
9 na Norway
10 U.S United States of America
11 Sweden Sweden
12 Germany Germany
13 UK United Kingdom of Great Britain and Northern Ireland
14 United Kingdom United Kingdom of Great Britain and Northern Ireland
15 Usa United States of America
16 U.S. United States of America
17 Spain Spain
18 Austria Austria
19 Philippines Philippines
20 New Zealand New Zealand
21 U.S.A. United States of America
22 United states of america United States of America
23 England United Kingdom of Great Britain and Northern Ireland
24 Estonia Estonia
25 Belgium Belgium
26 International Norway
27 United State of America United States of America
28 France France
29 Czechia Czechia
30 Puerto Rico Puerto Rico
31 united states United States of America Notice that it failed to standardise some of the countries. You can check which countries were not standardised by using the code below.
setdiff(countries, country_dict$original)[1] "United States of America" "usa"
[3] "Uk" "United Stateds"
[5] "United States of america"Since there is only a handful left, you can manually correct these. We then make a dictionary to standardise the country names.
full_country_dict <- c(deframe(country_dict),
c("United States of America" = "United States of America",
"usa" = "United States of America",
"Uk" = "United Kingdom of Great Britain and Northern Ireland",
"United Stateds" = "United States of America",
"United States of america" = "United States of America"))Then let’s apply this dictionary to the salary_sample data.
salary_sample |>
mutate(country_std = full_country_dict[as.character(country)]) |>
select(country, country_std)# A tibble: 400 × 2
country country_std
<chr> <chr>
1 United States United States of America
2 US United States of America
3 Australia Australia
4 USA United States of America
5 USA United States of America
6 United states United States of America
7 USA United States of America
8 United States United States of America
9 United States United States of America
10 Ireland Ireland
# ℹ 390 more rowsThis process may be easier to do via batch processing, but this is still experimental in ellmer. You can check the hellmer package for an implementation on batch processing.
🎉 Mm! You’ve reached the end of this demo! You are stylish! Proceed to …