Prompt Engineering

Text Analysis with large language models

Emi Tanaka

3th June 2025

These slides are best viewed on a modern browser like Google Chrome on a desktop or laptop. You can navigate the slides using the left and right arrows.

Table of contents

• Prompt engineering
• Instruction-based prompt
• Zero-shot prompt
• In-context learning
• Chain-of-thought
• Prompt engineering components
• Prompt design
• Communicating effectively with LLM
• Self-consistency
• Evaluation of a LLM
  • Benchmark: MMLU
  • Benchmark: MuSR
  • Benchmark: HumanEval
  • Other evaluations
• Exercise 4

Prompt engineering

• Prompt engineering is the process of designing and refining the prompts for a language model to generate specific types of output.

chat1 <- ellmer::chat_ollama(model = "llama3.1:8b", 
                            seed = 1,
                            api_args = list(temperature = 0),
                            echo = TRUE)

• Suppose we want to classify the sentiment of the text as positive or negative.

chat1$chat('Classify the text as positive or negative.
           "This is a great book!"') 

I would classify this text as **positive**. The use of the word "great" 
indicates a strong and enthusiastic endorsement of the book.

Instruction-based prompt

• You can give more specific instructions regarding the format.

chat2 <- ellmer::chat_ollama(model = "llama3.1:8b", 
                            system_prompt = "Just give the answer.",
                            seed = 1,
                            api_args = list(temperature = 0),
                            echo = TRUE)

• The response is now concise.

chat2$chat('Classify the text as positive or negative.
           "This is a great book!"')

Positive.

• In this case, a similar answer would be generated if system prompt was included in the user prompt.

Zero-shot prompt

• When LLM are given no examples of the task to carry out, this is referred to as zero-shot prompt.

chat2$chat('Which discipline does machine learning belong to?')

Computer Science.

chat2$chat('Which discipline does logistic regression belong to?')

Statistics and Machine Learning.

• LLM can be fine-tuned by training on a new labelled data, but this is computationally expensive and out of scope for typical analysts.

• Prompt engineering with one- or few-shot prompt is a low cost approach to provide in-context learning.

In-context learning

• If examples of expected response are provided in the prompt, these are called one-shot prompt (if one example) or a few-shot prompt (if more than one example).

chat2$chat('Which discipline does machine learning belong to?
           Examples:

           Technique: Regression
           Discipline: Statistics
           
           Technique: Deep learning
           Discipline: Data Science
           
           Technique: Database Management
           Discipline: Computer Science')

Data Science.

chat2$chat('Which discipline does logistic regression belong to?')

Statistics and Machine Learning, but more specifically, it belongs to the 
discipline of Statistics.

Chain-of-thought

• Chain-of-thought aims to make the LLM “think” before answering.

• Reasoning is a core component of human intelligence and LLM can mimic “reasoning” from memorisation and pattern matching trained from large corpus of text.

chat2$chat('Regression is a primarly tool for statisticians. 
           Which discipline does logistic regression belong to?')

Statistics.

• Here is another example where we want to calculate the median:

chat2$chat("What is the median of 1, 4 and 5?")

3.

• Provide some cue for reasoning.

chat2$chat("The median is the middle value of the sorted list of numbers. 
           What is the median of 1, 4 and 5?")

Since the numbers are already in order (1, 4, 5), the median is the middle 
number, which is 4.

Prompt engineering components

Some common components include:

• Instruction The task (be specific as possible).
• Format The response format. E.g. “return the value only” and “return as JSON”.
• Example Example(s) of input and expected response.
  
• Context Additional information about the context of the task.
• Persona Describe the role of the LLM. E.g. “You are a statistics tutor”.
• Audience Describe the target audience. E.g. “Explain it to a high school student”.
• Tone The tone of the text. E.g. “Respond professionally”.

Prompt design

Prompt = Verb + Focus + Context + Alignment + Constraints

• Verb: summarise, synthesise, assess, review, model
• Focus: topic, results, themes, methods
• Context: domain, audience, framework
• Alignment: goal, objective, purpose
• Constraints: format, length, style, tone

Example: “Please critique the statistical methodology used in the article on educational technology adoption by teachers. Assess strengths and limitations from the viewpoint of an academic journal peer reviewer with statistical expertise. Focus the critique on the sampling design, statistical tests, and interpretation of results. Provide a detailed response in a professional tone limited to 500 words.”

Bron Eager (2025) AI-Powered Scholar. Routledge.

Communicating effectively with LLM

• Be unambiguous and specific in the prompt.
• Use natural language or be conversational.
• Provide contextual framing.
• Offer representative examples.
• Break down larger instructions into single-focused tasks.
• Specify perspective.
• Build a shared history.

Bron Eager (2025) AI-Powered Scholar. Routledge.

Self-consistency

• To ensure the responses are reliable, you can generate the response to the same prompt multiple times and select the mode.

chat2$chat("What is the mean of 1, 9 and 5?")

(1 + 9 + 5) / 3 = 15 / 3 = 5.

chat2$chat("What is the mean of 1, 9 and 5?")

15.

chat2$chat("What is the mean of 1, 9 and 5?")

(1 + 9 + 5) / 3 = 15 / 3 = 5.

chat2$chat("What is the mean of 1, 9 and 5?")

15/3 = 5.

chat2$chat("What is the mean of 1, 9 and 5?")

15.

Evaluation of a LLM

Benchmark: MMLU

• Measuring Massive Multitask Language Understanding (MMLU)¹ is a public benchmark containing ~16,000 multiple choice questions (4 choices) spanning 57 academic subjects.

Which three systems of the human body function together to move and control body parts?

1. nervous, skeletal, and muscular
2. muscular, endocrine, and excretory
3. digestive, excretory, and reproductive
4. circulatory, endocrine, and respiratory

Correct answer:

• An enhanced version MMLU-Pro² containing 12,000 multiple choice questions (10 choices) across 14 domains with more challenging college-level problems.

Correct: A

1. D. Hendrycks et al. (2021) Measuring Massive Multitask Language Understanding. http://arxiv.org/abs/2009.03300

2. Y. Wang et al (2024) MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark http://arxiv.org/abs/2406.01574

Benchmark: MuSR

• Multistep Soft Reasoning (MuSR)¹: free text narratives on murder mysteries, object placements and team allocation.

Emily took her final stroll in the park last night, forever, when her life was snuffed out under the mask of night. The cause of death was a single fatal shot from a pistol. Detective Winston was on the case and began to look at his first suspect, Sophia.

Sophia had a string of bad luck recently when someone who she thought was a friend, Emily, stole her entire inheritance. Her evening strolls in the park became franc pacing while she reconciled the fortune she lost. Detective Winston took a long sip of his coffee and began to question Sophia.

‘Quite the marksmen I see’ - pointing to a picture of her holding a recently shot buck up.

‘Yeah, my dad loved taking me shooting’ - Sophia replied sheepishly

Identify the killer. Killers have a motive, means, and opportunity …

1. Z. Sprague et al. (2024) MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. http://arxiv.org/abs/2310.16049

Benchmark: HumanEval

• HumanEval¹ is a dataset of hand-written coding problems.
• The evaluation of the LLM is based on the unit tests associated with the code it generates.

Example 1

You will be given a string of words separated by commas or spaces. Your task is to split the string into words and return an array of the words.

For example:

words_string(“Hi, my name is John”) == [“Hi”, “my”, “name”, “is”, “John”]

words_string(“One, two, three, four, five, six”) == [“One”, “two”, “three”, “four”, “five”, “six”]

Example 2

def is_prime(n):  
   """Return true if a given number is prime, and false otherwise.
   >>> is_prime(6) 
   False 
   >>> is_prime(101) 
   True 
   >>> is_prime(11) 
   True 
   >>> is_prime(13441) 
   True 
   >>> is_prime(61) 
   True 
   >>> is_prime(4) 
   False 
   >>> is_prime(1) 
   False 
   """

1. M. Chen et al. (2021) Evaluating Large Language Models Trained on Code. http://arxiv.org/abs/2107.03374

Other evaluations

• Human evaluation can involve:
  • Accept/Reject of the response
  • Scoring the quality of the response on various aspects (e.g. answer relevancy, correctness, hallucination, responsible metrics, and so on).
  • Pick preferred response between two responses (reinforcement learning)

My interaction with ChatGPT on 2024-10-09

• LLM evaluating LLM¹: e.g. Prometheus² uses LLM to evaluate other LLMs

1. L. Zheng et al. (2024) Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Proceedings of the 37th International Conference on Neural Information Processing Systems

2. S. Kim et al. (2023) Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. http://arxiv.org/abs/2310.08491

Exercise 4

Go to Exercise 4

Next slide

Back to start