How to build a LLM

A language model is a program that predicts the next token. Give it "the fat cat" and it produces a probability for every token that could follow. The entire field of large language models, from autocomplete to chatbots, rests on that one operation repeated thousands of times.

emmllm is my attempt to build one from scratch and understand every piece. The goal is not to compete with ChatGPT. It is to get past the usual diagram where text enters a box labelled "transformer" and somehow comes out coherent. I wanted to know what is inside the box, what shape every number takes, and how the simple goal of predicting one token leads to 138 million adjustable numbers.

This article assumes no machine learning background. It builds every concept from school-level maths and introduces every symbol before using it.

The model is roughly GPT-2 sized but uses a few newer techniques. For reference: it processes text in chunks of up to 2,048 tokens, represents each token with 768 numbers, and stacks 12 processing layers. These are configuration choices that set the model’s scale, not facts it learns from data.

Turning text into numbers

A token is a short piece of text: a whole word, part of a word, a space, punctuation, or a single byte. Each token is assigned a number, its token ID, and the text is converted into a sequence of token IDs before any neural network maths happens.

One option is to use one number per character. the fat cat in a typical ASCII encoding is:

This always works, but every token adds cost at every layer of the model, so spelling text out one character at a time is expensive. Using one ID per whole word solves the length problem but creates a coverage problem: code, names, misspellings, emojis, and other languages would all fall outside the dictionary.

The middle ground is byte-pair encoding, or BPE. It starts with all 256 possible single-byte values, then scans a corpus of training text and repeatedly merges the most common adjacent pair into a new token.

Starting from bytes is what makes BPE foolproof. A computer stores all text as bytes (numbers 0–255), so any text can be encoded before a single merge is learned; merges only make common sequences shorter, never anything unrepresentable.

At each step the algorithm picks whichever pair appears most often. That is the choice that removes the most tokens in one round: each occurrence of the chosen pair becomes one token instead of two. If an appears 10,000 times and ba appears 2,000 times, merging an shortens the corpus by 10,000 tokens immediately.

The merge list is constructed once, before the neural network exists, and then frozen: it is a fixed list of pair-joining rules, not a set of trainable parameters.

After this process, common text collapses into a few large tokens while unusual text can always fall back to individual bytes. My tokeniser was given a 32,768-token budget. After reserving 256 tokens for individual bytes and 3 to mark the user, the model, and the end of a message, that left room for 32,509 learned merges. A larger budget could shorten encoded text but would also enlarge the embedding and output tables, so 32,768 is the chosen tradeoff. The merges were trained on 256 MiB of text from the Dolma dataset.

The result is a reversible mapping: the same text always produces the same IDs, and decoding recovers the original text exactly. Tokenisation only segments the bytes; it does not decide what the text means.

Embeddings

The model cannot calculate with text directly, so it represents each token with a list of numbers. That list is called an embedding. emmllm uses 768 numbers per token, making each embedding a 768-dimensional vector.

The 768 numbers are coordinates in the embedding space. Just as $(x,y)$ marks a point in 2D, an embedding marks a point in 768D. Similar embeddings are processed similarly by later matrices.

After training, these embeddings encode properties of each token. For cat, different dimensions may capture animal-related meaning, feline traits, and grammatical role. These properties are spread across many dimensions rather than stored in one labelled slot.

The embedding matrix stores one of these vectors for every token. A token ID is simply the row number used to retrieve it: ID 47 selects row 47. The lookup returns that token’s embedding, projecting the token into embedding space. The ID itself says nothing about the token’s meaning.

This matters because vectors can be compared while raw IDs cannot. The integer 481 is not "close to" 12 in any useful sense, but two 768-dimensional vectors can point in similar directions. During training, the dog ran and the cat ran produce many of the same prediction errors, so training pushes the dog and catrows in similar directions. They won't become identical, but they end up nearby — both are animals, both appear in similar sentences, both take similar verbs — so the rest of the network learns to treat them similarly.

The basic operation for comparing two vectors is the dot product: multiply matching coordinates and sum the results. A larger dot product means the vectors point in more similar directions. The model uses this throughout to decide which tokens are relevant to each other.

The tensor shapes

A tensor is just a block of numbers with a particular shape. A single number is a 0-dimensional tensor, a list of numbers is 1-dimensional (a vector), a grid of numbers is 2-dimensional (a matrix), and a stack of grids is 3-dimensional. The word describes the container, not the meaning.

Three letters describe the shape of nearly every tensor in the model. T is the sequence length in tokens, D is the model width of 768, and B is the batch size. Training does not process one sequence at a time — it stacks many independent sequences into a single batch and pushes them all through the model in one pass.

Batching exists for two reasons. Modern GPUs are designed around massively parallel matrix multiplication, so processing a batch of 32 sequences is far less than 32 times slower than processing one. It also makes training updates more stable. One unusual sentence might push a parameter sharply in one direction; averaging the correction across 31 other sequences smooths out that noise.

The sequences in a batch never interact. B is simply a stack dimension, so every shape in the table below begins with it, and every operation the model performs runs independently for each item in that stack.

Attention, with tiny tensors

An embedding vector is context-free. The token "bank" retrieves the same 768 numbers from the table regardless of whether the sentence is about a river or a financial institution. The embedding space captures properties of the word in isolation, but language is not made of isolated words. Meaning depends on context.

Attention is the mechanism that adds context. It allows each token’s vector to become a weighted sum of information from earlier tokens. Beside "river", the new vector for "bank" may copy a large share of the value carried by "river". Beside "deposit", it may instead copy from "deposit". The two occurrences began with the same embedding row but leave attention with different numbers.

Think of attention as a soft lookup. A normal lookup finds the one row whose label matches exactly. Attention compares a search term against every label at once, scores how well each matches, and returns a blend of information weighted by those scores.

That gives the three letters their jobs. The query is what the current token is searching for. Each key is a label describing what another token can be matched on. Its value is the vector that will actually be added into the current token when that key receives weight. Queries and keys choose the amounts; values supply the numbers being mixed.

The labels and stored information are not written by a person — they are vectors learned during training. The model makes all three from the same token vectors $X$ using three different learned matrices:

Here $X$ is the input: $T$ rows, one per token, with $d_{\text{model}}=768$ numbers in each row. Multiplying one row by $W_Q$ compresses those 768 numbers down to $d_k=64$ query numbers. Each of those 64 output numbers is a weighted combination of the 768 inputs, with the weights stored in $W_Q$. The matrix decides which aspects of the token’s 768-number representation matter for the "searching" role. $W_K$ and $W_V$ do the same compression with different learned weights, producing a 64-number key and a 64-number value from the same input. Repeating this for all $T$ rows produces $Q$, $K$, and $V$, each with one row per token.

The full equation performs four concrete steps. First, $Q\cdot K^{\mathsf T}$ creates a $T\times T$ table. Cell $(i,j)$ contains the dot product between token $i$’s query and token $j$’s key, so it scores how strongly token $i$ should read from token $j$.

Second, each score is divided by $\sqrt{d_k}$. A dot product adds $d_k$ coordinate-products, so its typical size grows as more coordinates are added. This matters because softmax, applied two steps later, exponentiates every score: raw scores of 16 and 8 would end up a factor of $e^{16}/e^8=e^8\approx2{,}981$ apart. Here $d_k=64$ and $\sqrt{64}=8$, so those scores become 2 and 1, whose ratio is only $e^2/e^1=e\approx2.72$. Scaling stops the width alone from pushing almost all the weight onto one token.

Third, $M$ is a causal mask — explained in the next section. Finally, softmax turns each row into weights that sum to 1, and multiplying by $V$ blends the value vectors in those proportions.

The query and key use separate matrices because searching and being searchable are different jobs. In the cat sat, satcan produce a query pattern associated with "find my subject", while catcan produce a key pattern associated with "I can be a subject". The value is separate again: once cat matches, its value determines which numbers are copied into sat’s updated vector.

Here is the full attention calculation for the cat sat, using only two dimensions so every row and column can be labelled:

Each result cell pairs its row’s query token with its column’s key token. For example, the cat row and sat column meet at 1. cat’s query is $\begin{bmatrix}0&1\end{bmatrix}$ and keysat is $\begin{bmatrix}0&1\end{bmatrix}$, so their dot product is $0\cdot0+1\cdot1=1$. Every cell in the score matrix is made by that same multiply-then-add rule.

Stopping the model from cheating

During training the full sentence is loaded at once, so without a mask each token could just read ahead to the answer. A causal mask sets every forward-looking score to negative infinity, which softmax converts to exactly zero weight. Each token can only attend to itself and earlier tokens.

Softmax turns a row of raw scores into a probability distribution: exponentiate, then divide by the total so the row sums to one. Larger scores get larger shares of weight.

The first token can only see itself, so it receives all of its own weight. The second token distributes its weight across tokens one and two. The third spreads its weight across all three tokens, placing half on the second and a quarter on each of the others.

These weights determine how much information to collect from each value vector. Multiplying the weight matrix by $V$ takes each row of proportions and applies it to every coordinate of the value vectors. The second row, $\begin{bmatrix}0.33&0.67&0\end{bmatrix}$, combines the first two values as $0.33\begin{bmatrix}1&0\end{bmatrix} +0.67\begin{bmatrix}0&2\end{bmatrix} =\begin{bmatrix}0.33&1.34\end{bmatrix}$.

Each output row is a weighted mixture of information from the tokens that token was allowed to see. It still has 64 numbers, but those numbers now include selected parts of earlier tokens’ value vectors. This row is the attention update passed to the output projection.

Twelve heads at once

A single set of Q, K and V matrices can only learn one pattern of attention per layer. Every relationship a sentence contains, subject to verb, adjective to noun, pronoun to its referent, would have to compete for the same set of weights, and the dominant pattern would crowd out the rest.

Multi-head attention solves this by creating 12 separate lookups, called heads. Each head projects the full 768-number input into its own 64-number queries, keys, and values, then runs the complete score, mask, softmax, and value-mixing calculation independently. 12 is a configured choice; 64 follows from dividing the 768-number output evenly across those heads.

Because the heads share no weights, one can learn to track syntactic relationships like "attend to the previous token" while another learns semantic patterns like "attend to the most recent noun".

The 12 outputs of 64 numbers are concatenated back into a single 768-dimensional vector, reassembled in slice order. Call that tensor $H$. It is multiplied by a learned $768\times768$ output-weight matrix, written $W_O$. Without it, head one would always occupy coordinates 1–64, head two 65–128, and so on. $W_O$ lets every one of the 768 output coordinates take a learned weighted combination of coordinates from all 12 heads.

Where position comes from

The query-key dot product contains no distance information. The causal mask says only whether a token is before or after another token; it does not say whether an allowed token is one place back or 100 places back. Without another signal, identical query and key vectors receive the same compatibility score at both distances.

emmllm uses rotary positional embeddings, usually called RoPE, to inject position information. The idea relies on rotation: if you plot two numbers as a point on a flat grid, rotating that point means swinging it around the origin by some angle while keeping the same distance from the centre, like the hand of a clock moving forward. Rope takes each query and key vector and reads it as a series of number pairs, treating each pair as a point on that kind of grid. For pair $r$ at token position $p$, it rotates that point by $p\,\theta_r$ radians (a way of measuring angles). $\theta_r$ is a fixed rate for that pair: moving one token to the right adds the same extra angle every time.

The key property emerges when a rotated query meets a rotated key. Consider a query at position 5 and a key at position 3. When both are rotated, their relative angle for pair $r$ is $5\theta_r-3\theta_r=2\theta_r$. The dot product therefore depends on the distance of two tokens, not their absolute addresses.

So the model never needs to learn anything specific about position 5 or position 3. It only learns about relative distances, "two tokens apart", and that knowledge can be reused at other absolute positions, such as 105 and 103.

One transformer layer

A transformer layer combines the two operations described above into a single repeatable block. First, attention moves information between tokens. Then, a feed-forward network processes each token row independently.

In the diagram, $x$ is the matrix of token vectors entering the layer, and $x_1$ is that matrix after the attention update. The + steps are residual connections. The input vector that entered the operation is added back to whatever the operation produced: $y=x+f(x)$. If the attention or feed-forward branch outputs all zeros, then $y=x$ and the layer copies its input exactly. The branch only has to produce the coordinates that should be added or subtracted.

Residual connections also matter during training. Correction signals must travel backwards through every layer to reach the earliest parameters, and each operation can shrink them. Because $y=x+f(x)$ contains a direct copy of $x$, part of the backward signal skips past $f$ entirely — this is what makes 12 stacked layers trainable.

RMSNorm handles scale. If one token vector has coordinates around 0.1 and another has coordinates around 100, the same projection matrix produces dot products about a thousand times apart. That can make softmax put nearly all its weight on one entry for one token and spread it almost flat for another. RMSNorm divides each token vector by a single number computed from its own coordinates, making its root-mean-square coordinate close to 1 before the next learned matrix sees it.

Every coordinate is divided by the same RMS number, so their ratios stay unchanged: 3:4 becomes 0.85:1.13. The small $\varepsilon$ inside the square root prevents division by zero. Afterwards, a learned gain $g_i$ multiplies coordinate $i$. If training repeatedly benefits from making coordinate 20 twice as influential, its gain can move towards 2 without allowing the whole vector’s scale to drift.

The feed-forward network

Attention is mainly a routing operation: it decides which token values to mix into each token row. After that routing, the model still needs a large per-token computation that can react differently to different combinations of coordinates—for example, "activate this output when feature A is present and feature B is absent". That is the feed-forward network’s job.

One projection creates 3,072 candidate features — four times the model width of 768, a configured choice — and another creates 3,072 gate values. SiLU bends the gates non-linearly, then each gate is multiplied by its matching candidate — a gate near zero kills that candidate, a positive gate lets it through. A final projection mixes the survivors back into 768 coordinates. $\odot$ in the diagram means this element-wise multiplication, not matrix multiplication.

$\operatorname{SiLU}(x)=x\sigma(x)$ is the non-linear part. $\sigma(x)$ is the sigmoid function, which returns a number between 0 and 1. For $x=-4$, SiLU returns about -0.07, so multiplying by that gate nearly removes the matching candidate. For $x=4$, it returns about 3.93, so the candidate is passed through and scaled up. Because the gate depends on the current token vector, the same candidate coordinate can be active for one token and suppressed for another.

A single layer makes only a modest update to the representation. But stacking the block 12 times allows later layers to build on features assembled by earlier ones: early layers might learn basic syntactic patterns, middle layers might combine those into phrase-level structures, and later layers might capture relationships across longer distances.

The final step needs to convert a 768-dimensional vector into 32,768 scores, one per token in the vocabulary. The straightforward approach would be another large matrix. Instead, the model reuses the embedding table it already has.

To score each candidate token, the model takes the dot product of the final vector $h$ with that token’s embedding row. For vocabulary item $j$, its score — called a logit — is $z_j=h\cdot E_j$: multiply the 768 matching coordinates and add them. Training raises this number for the correct next token and lowers it relative to incorrect tokens. The same table that turns an ID into a vector on the way in therefore supplies the candidate vectors used to produce scores on the way out.

This is called weight tying. Each token's embedding serves as both its input representation and its target direction when predicted as output. When training moves the embedding for cat, that one update changes both how the model reads cat and how easily it can predict cat.

Learning the next token

Training uses a single sequence as both input and answer key by shifting it one token. After every input token, the model’s job is to predict which token comes next.

After the first token the model has seen the and should assign fat a high score, and so on down the sequence. The causal mask ensures each token can attend only to itself and earlier tokens, so up to 2,048 predictions can be computed in a single forward pass. The batch dimension B adds more independent sequences to that same pass. The final training loss is the average over all target tokens in all B sequences.

The logits are unconstrained: any real number, positive or negative, and a row of them does not sum to anything meaningful. Softmax converts them into a proper probability distribution, where every value is positive and the row sums to one.

The loss function is the negative natural logarithm of the probability assigned to the correct token, known as cross-entropy loss. This makes the scale concrete: assigning the correct token 90% probability costs $-\ln(0.9)\approx0.11$, 50% costs $0.69$, and 1% costs $4.61$. Confident correct predictions cost little; confident wrong predictions cost much more.

Softmax and cross-entropy also produce a simple correction at the logits. For the correct token it is $p_{\text{correct}}-1$. At 90% probability that is -0.1; at 1% it is -0.99. Gradient descent therefore pushes much harder to raise the correct logit when the model is confidently wrong.

Backpropagation computes a gradient for each of the 138.4 million parameters: how much the loss would change for a tiny increase in that parameter. +0.4 means increasing it would raise the loss, so it should be lowered; -0.4 means the opposite. It obtains these numbers by applying the chain rule backwards through the forward pass.

Gradient descent then applies the update below. $w$ is one parameter, $\mathcal L$ is the loss, and $\nabla_w\mathcal L$ is that parameter’s gradient. The gradient points uphill, towards larger loss, so the minus sign moves $w$ downhill.

$\eta$ is the learning rate. Too small and training crawls; too large and each step overshoots, causing the loss to oscillate or diverge.

A single update barely changes the model. But repeating across billions of tokens rewards changes that raise the correct token's probability: embeddings used in similar contexts drift together, attention projections that retrieve helpful tokens are reinforced, and feed-forward gates that reduce error open wider.

The full picture

To generate text, the model takes a prompt, runs it through all 12 layers, picks the highest-scoring next token, appends it, and repeats. Every step is a lookup, matrix multiplication, or element-wise function — 138.4 million numbers adjusted one batch at a time.

If you made it this far, thanks for reading. The code for emmllm is on GitHub if you want to poke around. If you have questions or spot something wrong, send me an email at emmanuelk@emmanuelk.com.au.