library(ellmer)
1 response_format
The default response format is text, but you can have a more structured output with JSON output.
1.1 JSON format
The response_format is useful for getting a structured response. For data processing tasks, this is arguably the most useful feature. If you use JSON format, you need to make sure that you tell the system that you want the format as JSON somewhere, otherwise you may end up getting a lot of white spaces.
be_statistician <- "Respond like a statistician but keep it concise. No more than 80 words."
statistician <- chat_ollama(model = "llama3.1:8b",
seed = 1,
system_prompt = paste0(be_statistician, " Return the format as json."),
api_args = list(response_format = list(type = "json_object"),
temperature = 0),
echo = TRUE)The answer is in JSON format.
out <- statistician$chat("Could you tell me what the probability of getting a head four times if I toss the coin 4 times? Just return the probability."){
"probability": 0.0625
}But it’s stored as a character.
class(out)[1] "character"Let’s convert this to a list object by converting the character to list using jsonlite::fromJSON().
jsonlite::fromJSON(out)$probability
[1] 0.0625The user prompt asks to specifically return the probability only, but sometimes the output format can have different JSON output.
1.2 Structured output
Suppose that we want the output with a structure shown in Figure 1.
flowchart LR
response("response<br>(list)") --> steps("steps<br>(data.frame)")
response --> final_answer("final_answer<br>(numeric)")
steps --> explanation("explanation<br>(character)")
steps --> output("output<br>(character)")
final_answer
linkStyle default stroke: black
The schema in Figure 1 is coded as a list below.
json_schema <- type_object(.description = "Provide the reasoning behind the answer.",
steps = type_array(description ="Provide the steps to the answer.",
items = type_object(explanation = type_string("Explanation for the steps."),
output = type_string("What the corresponding output is."))),
final_answer = type_number())The schema stored in json_schema is parsed to response_format below.
be_statistician <- "Respond like a statistician but keep it concise. No more than 80 words."
tame_statistician <- chat_openai(
system_prompt = be_statistician,
model = "gpt-4o-mini",
params = params(seed = 1,
temperature = 0)
)The output is shown below.
out <- tame_statistician$chat_structured("Could you tell me what the probability of getting a head four times if I toss the coin 4 times?",
type = json_schema)
out$steps
explanation
1 Identify the number of trials (n) and the number of successes (k). Here, n=4 and k=4.
2 Calculate the probability of success (p) for a fair coin, which is 0.5.
3 Use the binomial probability formula: P(X=k) = (n choose k) * p^k * (1-p)^(n-k).
4 Calculate (4 choose 4) = 1, (0.5)^4 = 0.0625, and (0.5)^0 = 1.
5 Thus, the final probability is 0.0625.
output
1 n=4, k=4
2 p=0.5
3 P(X=4) = (4 choose 4) * (0.5)^4 * (0.5)^0
4 P(X=4) = 1 * 0.0625 * 1
5 P(X=4) = 0.0625
$final_answer
[1] 0.0625Looking at the structure, you can see it matches with the desired output in Figure 1.
str(out)List of 2
$ steps :'data.frame': 5 obs. of 2 variables:
..$ explanation: chr [1:5] "Identify the number of trials (n) and the number of successes (k). Here, n=4 and k=4." "Calculate the probability of success (p) for a fair coin, which is 0.5." "Use the binomial probability formula: P(X=k) = (n choose k) * p^k * (1-p)^(n-k)." "Calculate (4 choose 4) = 1, (0.5)^4 = 0.0625, and (0.5)^0 = 1." ...
..$ output : chr [1:5] "n=4, k=4" "p=0.5" "P(X=4) = (4 choose 4) * (0.5)^4 * (0.5)^0" "P(X=4) = 1 * 0.0625 * 1" ...
$ final_answer: num 0.06252 Parallel processing
If you have large data then you’ll find generally that the text processing with LLM will take too long. In this instance, you can speed it up by parallel processing with multiple simultaneous chats.
2.1 Parallel structured output
Suppose we have a series of prompts about a person and their age. We want to convert each of these prompts to extract the person’s name and their age.
prompts <- list(
"They call me Ronan — I'm 28 and currently wandering across South America.",
"At 45, I’ve finally learned how to skateboard. I’m Grace, by the way.",
"I'm 34, and the name’s Layla. I restore vintage motorcycles for a living.",
"Hi! The name’s Mira. I turned 22 last week and still don’t like coffee.",
"I'm Jules. Just hit 39, and I still can’t figure out taxes.",
"Name’s Andre, age 31. I collect rare vinyl and make terrible dad jokes.",
"I’m 26, and people know me as Sam — the guy who bakes at midnight.",
"My name? Camila. I’m 48 and recently became a certified yoga instructor.",
"Hey, I’m Leo. Age 20. Professionally confused, but full of ambition.",
"Just turned 30! I’m Nia, and I write greeting cards for a living.")
schema_person <- type_object(name = type_string(),
age = type_number())We can start multiple simultaneous chats by using parallel_chat_structured by parsing the chat object, prompts and the schema for structured output.
chat <- chat_openai(model = "gpt-4.1",
params = params(seed = 0))
parallel_chat_structured(chat, prompts, schema_person)[working] (0 + 0) -> 5 -> 5 | ■■■■■■■■■■■■■■■■ 50%[working] (0 + 0) -> 0 -> 10 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 100% name age
1 Ronan 28
2 Grace 45
3 Layla 34
4 Mira 22
5 Jules 39
6 Andre 31
7 Sam 26
8 Camila 48
9 Leo 20
10 Nia 302.2 Batch chat
batch_chat() and batch_chat_structured() are similar to parallel_chat() and parallel_chat_structured(), but these are submitted as batch requests. This process can take longer to complete, but you generally pay less to the provider.
🎉 Huh! You’ve reached the end of this exercise! You are mind-blowing! Proceed to …