Learn How Large Language Models (LLM) Work

The project is designed for illustration, experimentation, and structural understanding; and not for production use.

Modern GPT Chat Agents are built on a simple objective: given some text, predict the next word.

Repeated many times, this produces full conversations.

What we recognize as a GPT-type chatbot is built on making useful next-token predictions over and over, using the context of text seen so far. (Chat assistants are then trained further to follow instructions.)

During training, an LLM is exposed to large amounts of human language and learns patterns in how words follow one another. When generating a response, the model works step by step, using the words so far (its context window) to choose the next one. At each step, it makes a best guess based on what it has learned.

Generation proceeds one token at a time.

These examples illustrate how changing:

1. the context window (e.g., 0, 1, 2, or 3 prior words) and
2. the training text

affects next-token predictions. The results determine how well the conversation goes.

We include a neutral cat-dog corpus (linked below) to illustrate cases where context beyond one previous word provides no additional benefit: which noun follows "on the" depends on both the subject and the verb, which a window of three tokens cannot see.

Training is a large, one-time computational effort. Each conversation with a trained model costs far less than training did, but across many users the total cost of running a model can rival or exceed it.

Training requires processing large amounts of human-written text and repeatedly adjusting the model to improve predictions.

Large systems may take days or weeks of computation across many machines. The pre-trained models in Toy GPT reflect the same process at much smaller scale.

It takes space to track each word in context (even if the context is only 2 or 3 words prior). The context-3 model stores one weight row per unique 3-token context, so the weight matrix grows as vocab³:

[1] Size of train-400-context-3\artifacts\02_model_weights.csv.

Our simple Context-3 model (without embeddings) grew to 428 MB, consisting mostly of zeros (because relatively few combinations appear in context).

This rapid growth makes direct counting approaches impractical and motivates more compact representations for capturing similarities across words.

Modern GPT-style models use the transformer architecture from "Attention Is All You Need"¹. They use:

• embeddings (numeric representations of words) and
• attention (selecting which prior words matter most)

to represent tokens and weigh which prior words are most relevant for prediction.

These mechanisms produce much more compact output. Attention is included in Toy GPT for completeness, but its full effect is hard to demonstrate on such tiny corpora.

¹ Vaswani et al. (2017), Attention Is All You Need.

Interactive visualization of next-word (token) prediction in GPT-style language models.

• App -interactive visualization of next-word (token) prediction in GPT-style language models.
• App Repo

Each repository contains pre-trained artifacts for one model–corpus combination. Two things vary independently: the model and the corpus.

Select a model (how many previous tokens are used to predict the next one):

Select a corpus (the text the model is trained on):

• ⚠️ Too large to commit (428 MB of mostly zeros).
• ℹ️ Attention requires a larger corpus to produce meaningful position weighting.

Legend: X indicates a completed training run with committed artifacts. Blank cells indicate combinations intentionally not explored.

Model codes increase with architectural complexity.

Predicts the next token using only overall frequency, without considering surrounding context.

Predicts the next token conditioned on the immediately preceding token.

Predicts the next token using the previous two tokens as context (a trigram model in n-gram terms). Unlike the bigram (200), it can capture dependencies two tokens back. It is stored as a sliding context window so the same structure extends to context-3 (400), embeddings (500) and attention (600).

Predicts the next token using the previous three tokens as context.

Learns numeric vector representations of tokens so related tokens occupy nearby locations in a learned space.

Dynamically weights which prior tokens matter most when predicting the next token, rather than treating all context equally.

See the paper Attention Is All You Need for background on attention-based models.

Corpus IDs are three digits and follow a project-wide standard:

• 000–009: canonical reference corpora
• 010–049: curated explanatory corpora for focused demonstrations
• 100–799: additional contributions

Each corpus file is named NNN_slug.txt.

A corpus deliberately constructed so that context beyond one previous token adds no benefit (context-2 and context-3 should not beat the bigram). Some adjacency signal remains (for example, "sat" is always followed by "on"), so a bigram still beats a unigram. Used as a negative control for longer contexts.

A corpus where adjacency or proximity carries meaningful predictive signal (e.g., attributes, phrases, or semantic groupings). Increasing context length should measurably improve predictions.

A model with no context: it predicts the next token from overall token frequency alone.

A model whose context is one previous token: it predicts from adjacent pairs.

This organization emphasizes clarity, reproducibility, and explicit boundaries between models, data, corpora, and claims.