Parameters of Large Language Models

Text Analysis with large language models

Emi Tanaka

3th June 2025

These slides are best viewed on a modern browser like Google Chrome on a desktop or laptop. You can navigate the slides using the left and right arrows.

Predictive model

Large language model at its core predicts the next word given a sequence of words.

Input: All models are wrong, but some are

Updated input:

All models are wrong, but some are useful

Updated input:

All models are wrong, but some are useful.

LLM

useful

<|end|>

Output: useful.

Tokenization

Input Where there’s a will, there’s a

Token Where there ’s a will , there ’s a

Token ID 11977 1354 802 261 738 11 1354 802 261

LLM

2006

way

Tokens are not necessary words

Every LLM has its own tokenizer with varying vocabulary size.
Rule of thumb for common English text: 1 token = ~4 characters = ~0.75 words

summary summarise summarize summarising summarizations

summary summ ar ise summ ar ize summ ar ising summ ar izations

3861 141249 277 1096 750 5066 25434

Special tokens

In practice, summarise may be tokenized as summ ##ar ##ise where “##” indicates another token prefixes it.

Due to popularity of chat tokens since 2023, tokenizers have adapted to a conversational direction with special tokens to indicate speaker role, such as
- <|system|> – high-level instructions
- <|user|> – user (usually human) queries or prompts,
- <|assistant|> – typically the model’s response, and
- other, e.g. <|tool|> .

Token distribution

Input: Where there’s a will, there’s a

Output is randomly sampled from likely tokens weighted by their respective probabilities.

Original distribution:

Token Probability
2006

301

4443

…

Small top_p:

Token Probability
2006

301

High temperature

Token Probability
2006

301

4443

…

seed ensures the same random sample given the same input (important for reproducibility!), but the same seed may not yield the same result across different systems.

Exercise 3

Go to Exercise 3

Next slide

Back to start