Architecture of Large Language Models

Text Analysis with large language models

Emi Tanaka

4th June 2025

These slides are best viewed on a modern browser like Google Chrome on a desktop or laptop. You can navigate the slides using the left and right arrows.

Table of contents

• Overview of training a LLM
• Corpora for training data
• Corpus size used for training increasing over time
• Tokenization
• Embeddings
• Artificial neuron
• Visualising an artifical neuron
• Artifical neural network: regression
• Artifical neural network: classification
• Feed forward neural network
• Embedding model: Word2Vec for illustration of an embedding process
• Transformer model
• Positional embeddings
• Attention head: Projection matrices
• Attention Head: Relevance scoring
• Attention Head: Combining information
• Summary
• Exercise 7

Overview of training a LLM

Corpus (raw)

Preprocessed Training Data Tokenized Training Data Base LLM Fine-tuned LLM

Corpus (labelled)

Corpora for training data

Corpus (raw)

• LLMs are trained on large diverse datasets such as books, Wikipedia, websites, code, scientific articles and social network platforms.

• For example, LLaMA¹ used:
  • (67%) CommonCrawl: web archives from 2017 to 2020
  • (15%) C4: pre-processed CommonCrawl
  • (4.5%) GitHub: repos distributed under Apache, BSD, and MIT licenses
  • (4.5%) Wikipedia: online encyclopaedia
  • (4.5%) Gutenberg and Books3: books from Project Gutenberg and the Pile dataset
  • (2.5%) ArXiv: scientific preprints
  • (2%) Stack Overflow: a high quality Q&A website

1. Touvron et al. (2023) “LLaMA: Open and Efficient Foundation Language Models,” Feb. 27, 2023, arXiv preprint: arXiv:2302.13971

Corpus size used for training increasing over time

Figure 1

• Note: not all vendors publicly document their corpus size for training.

Tokenization

Input Where there’s a will, there’s a

Token Where  there ’s  a  will ,  there ’s  a

Token ID 11977 1354 802 261 738 11 1354 802 261

LLM

2006

way

See also Open AI tokenizer: https://platform.openai.com/tokenizer

Embeddings

• Language modelling typically use (token) embeddings.
• Embedding models convert items to numerical vectors such that similar items are closer together than dissimilar items based on this embedding space.

• Text embedding models are often based on artificial neural networks.

Artificial neuron

• The artificial neuron is an elementary unit of an artificial neural network.

• The artificial neuron receives inputs \boldsymbol{x} = (x_{1}, \dots,x_{p})^\top that is typically combined as a weighted sum: z = b + \sum_{j=1}^pw_jx_{j} = b + \boldsymbol{w}^\top\boldsymbol{x}, where \boldsymbol{w} = (w_1, \dots, w_p)^\top are weights and b referred to as bias.

• The z then get passed into the activation function, h(z), e.g. 

  • Linear: h(z) = z,
  • Sigmoid: h(z) = (1+e^{-z})^{-1} (a.k.a. a logistic function), and
  • ReLU: h(z) = \max(0, z).

Visualising an artifical neuron

Input layer

1

x_1

x_2

Output layer

h = ReLU

h(z)

• When x_1 = 1 and x_2 = 3, then \begin{align*}z &= b + w_1x_1 + w_2 x_2\\ &= 1 + 0.5 \times 1 - 3\times 3 = -7.5.\end{align*}

• Using ReLU, the prediction is \max(0, z) = 0.

Artifical neural network: regression

Input layer

1

x_1

x_2

Hidden layer
h_1 = ReLU

1

h_1(z_{11})

h_1(z_{12})

Output layer
h_2 = Linear

h_2(z_{21})

• The number of nodes in the input layer is constrained by the (modified) input data.
• The number of nodes in the output layer is constrained by the desired output (regression or classification).

Artifical neural network: classification

Input layer

1

x_1

x_2

Hidden layer
h_1 = ReLU

1

h_1(z_{11})

h_1(z_{12})

Output layer
h_2 = Softmax

h_2(\boldsymbol{z}, 1)

h_2(\boldsymbol{z}, 2)

h_2(\boldsymbol{z}, 3)

Assume classification to K classes.

Softmax:

h(\boldsymbol{z}, i) = \dfrac{\exp(z_i)}{\sum_{j=1}^K\exp(z_j)}

Output node i contains the “probability score” associated with class i.

Feed forward neural network

• Feed forward neural networks, also called deep neural networks, add more hidden layers.
• The output depends on the parameters, \boldsymbol{\theta} = (\underbrace{\boldsymbol{b}^\top}_{\text{biases}},\underbrace{\boldsymbol{w}^\top}_{\text{weights}})^\top.
• The number of parameters increases exponentially with more layers, but you gain flexibility in modelling.
• These parameters are calibrated (or trained) by:
  1. Defining an objective (or loss) function (like the mean square error), and
  2. Finding parameters \boldsymbol{\theta} that optimise this objective function (using techniques like stochastic gradient descent and backpropagation).

Embedding model: Word2Vec for illustration of an embedding process

• Word2Vec¹ is one of the early unsupervised word embedding models based on a three-layer neural network using a fixed window size.

Training data: Statisticians analyse data Training mode: Continuous Bag Of Words Vocabulary size: 5     Window size: 3     Aim: Predict middle word

Statisticians

1
0
0
0
0

data

0
0
1
0
0

Input

 
 
 
 
 

 
 
 

Output

 
 
 
 
 

0.01
0.89
0.04
0.01
0.01

Actual:

analyse

0
1
0
0
0

1. Tomas Mikolov et al (2013) Efficient Estimation of Word Representations in Vector Space. https://arxiv.org/abs/1301.3781

Transformer model

• Popular LLM are variants of a pre-trained transformer model.
• The original transformer model¹ consist of:
  • Input embeddings of current and previous positions
  • Positional embeddings
  • A stack of transformer blocks (N=6) with each containing:
    • An attention layer with multiple attention heads
    • Normalisation layers
    • Feed forward neural network with 3 hidden layers:
      Linear, ReLU, then Linear
  • Un-embedding layer (Softmax)

Original transformer model by Vaswani et al (2017)

https://kikaben.com/transformers-encoder-decoder/

https://www.datacamp.com/tutorial/how-transformers-work

1. Vaswani et al. (2017) Attention is All you Need. 31st Conference on Neural Information Processing Systems

Positional embeddings

Input The paired t-test p-value is 0.049, so it is statistically significant

Token The paired  t  -test p  -value  is  0 . 049 , so it is statistically signicant

• Context length: below 10 (typically much larger, e.g. 4096 for llama3.1:8b)

The paired  t  -test p  -value  is  0 . 049

• Transformer blocks are blind to position of token, so incorporate information about relative or absolute positions of tokens for the subsequent steps.

• Position embedding in the original transformer is represented by unique frequencies and offsets of the wave:

Figure 2

• PE(pos, 2i) = \sin(pos/10000^{2i/n}) and
• PE(pos, 2i + 1) = \cos(pos/10000^{2i/n})

where i is dimension in the encoding and pos is the token position.

Attention head: Projection matrices

• The goal of attention layer is to incorporate relevant information of previous tokens into the current token.

The paired  t  -test p  -value  is  0 . 049

• Suppose the current position is 049 – the neural network doesn’t know what this token refers to without looking at the previous tokens.
• The attention head consists of query, key, and value projection matrices.

Attention Head: Relevance scoring

• Embedding matrix is multiplied with each of the projection matrices

\times

The query vector of the current position:

=

Result \rightarrow Softmax(Result)

Attention Head: Combining information

\times

=

Sum above and the new token embedding that incorporates relevant information from other tokens is:

Summary

Input:

All models are wrong, but some are

Updated input:

All models are wrong, but some are useful

Updated input:

All models are wrong, but some useful.

Tokenizer

Embedding layer

Transformer blocks
(artificial neural network)

Un-embedding layer

Token	Token ID
All	2594
models	7015
are	553
wrong	8201
,	11
but	889

Token embeddings

Token

Numerical vectors

2594
7015
0553
8201
0011
0889

Token ID	Probability
1236	0.844
1991	0.004
12698	0.001
...	...

8316 = useful

13 = .

13 = .

XXXX = <|end|>

XXXX = <|end|>

Output: useful.

LLM

Exercise 7

Go to Exercise 7

Next slide

Back to start