This is a real GPT-style transformer, the same kind of architecture that sits behind ChatGPT and Claude, only more than two million times smaller than GPT-3. It trains right here on your device; no data leaves your browser.
The model predicts the next token, calculates its error against the real text, and adjusts its weights to lower that error. Then it does it again, hundreds of times. That is all training is.
The text is cut into pieces and each piece gets a number. This model learns about 250 word pieces from its text; large models use about 100,000.
Each token becomes a vector: a list of 48 numbers. During training, tokens used in similar ways end up with similar vectors.
For every position, the model decides which earlier tokens matter for guessing the next one. Two layers with four attention heads each.
It predicts the next token and calculates the error (the loss). Then it nudges every weight a little to lower that error. Repeat hundreds of times.
Part 1
Follow the four steps. Each one links to the chapter that explains it.
1Choose a text
2Train
Loading TensorFlow.js…
Watch for: a steep drop first, then a long flattening tail. Why? Chapter 3 →
3Let it write
Try: press it twice and compare, and look for sentences that sound right but are not in the text (that is a hallucination), then move Creativity (temperature) to 0.2 and to 1.5. Train more and try 1.5 again: a better-trained model stays readable at higher temperature. Why is it different every time? Chapter 4 →
4Look inside
Type anything. The bars show the model's guesses for the next token, live, also while it trains. Chapter 2 →
Every 50 steps the model writes a sample. Watch noise turn into language. The weights changing live are shown in Chapter 2.
Part 2
Six short chapters, in the order the model works: text becomes tokens, tokens flow through the model, the model learns, then writes.
Chapter 1
A model never sees words, only numbers. The tokenizer decides how text is cut into pieces. Large models learn their pieces with byte-pair encoding (BPE): start with single letters, then keep merging the pair that appears most often. Watch it happen on your own sentence.
Fewer tokens means the model sees more meaning per step. GPT-4's tokenizer learned about 100,000 such pieces from a huge text collection; the model above learns up to 250 pieces from its own training text.
Chapter 2
Every token becomes a vector of 48 numbers, its embedding: a point in a 48-dimensional space. At first these points are random. During training, tokens used in similar ways move close together. That is how meaning is stored: as positions in a space of numbers. GPT-3 uses vectors of 12,288 numbers, i.e. 12,288 dimensions.
Why 48? It is a design choice by whoever builds the model, not a law of nature. Each extra dimension adds capacity to store patterns, but also parameters and compute. 48 is enough for one page of text and still trains in a browser in a minute. It also has to split evenly across the 4 attention heads: each head works with 12 of the 48 numbers.
The model is built from two identical blocks stacked on top of each other, like two floors of an office. On each floor, two things happen in turn: the tokens look at each other (attention), then each token works through what it has gathered (the hidden layer). Block 1 is the ground floor and learns simple patterns; block 2 builds on its results. GPT-3 has 96 such floors.
The attention weights turn each token's vector into three new vectors. The query asks “what am I looking for?”, the key says “this is what I offer”, the value is the information passed on. Where a query matches a key, the token pays attention and takes in that value. In “Look inside”, the amber chips show where it looks.
In short: 192 small pattern detectors check each token's vector. The ones that fire add what they found, and the enriched vector moves on to the next block. This is where the model stores most of what it has learned.
Why the activation matters: without this “bend”, any stack of layers would collapse into one big multiplication, and the network could only learn straight-line relationships. GPT models use a smoother variant (GELU); early networks used the S-shaped sigmoid.
| Model | Neurons per block | Blocks |
|---|---|---|
| This model | 192 | 2 |
| GPT-3 (2020) | 49,152 | 96 |
| Llama 3.1 405B (open-weight) | 53,248 | 126 |
| DeepSeek-V3 (open-weight)256 experts, 8 used per token | 256 × 2,048 | 61 |
| Frontier modelsClaude, GPT, Gemini | not published | – |
Each pixel is one weight of block 1, the model's first layer: about 16,000 of its parameters. Press Train in Part 1 and watch faint random noise turn into structure, live.
Chapter 3
The model reads a piece of the training text and guesses every next token. The loss measures how wrong those guesses were. Then the optimizer (Adam) nudges every weight a little in the direction that would have made the guess less wrong. One step. Repeat hundreds of times. This is backpropagation, the chain rule from calculus.
Before training, the weights are faint random noise: the model knows nothing. Early on, the pictures change quickly and the loss falls fast. Later, the changes get smaller and the patterns settle. Stripes and bands appear where groups of weights learn to work together.
The first steps bring big gains, each further step less. Researchers measured that a language model's loss falls as a power law of compute, data and parameters (Kaplan et al., 2020): every tenfold increase in compute buys roughly the same small improvement.
Even when the curve flattens, this model is far from usable. That is why training real models costs so much in compute and parameters, and why it still gets cheaper as the cost per computation falls. A low loss here means it has nearly memorised one page; a big model needs far more text to generalise.
Chapter 4
Because of step 3. The probabilities are the same each time, but the dice roll differs, and one different token changes everything after it. ChatGPT and Claude work exactly like this, which is why the same question gets slightly different answers.
The Creativity slider is the temperature. It loads the dice: at 0.2 the model almost always takes the favourite: consistent, but repetitive. At 1.5 the outsiders get a real chance: more original, but more mistakes. Many business tools run at low temperature for exactly this reason.
Better trained, more headroom. The better a model is trained, the higher the temperature can go before it talks nonsense. A well-trained model is confident: the right token stands far ahead of the alternatives, so it takes much more “loading of the dice” before the outsiders start to win. Try it: let it write at 1.5 after a short training, then train more and try 1.5 again.
Every single token is a likely choice given the ones before it, so the sentence reads well. But the model checks only plausibility, step by step, never truth. A chain of likely pieces can add up to a statement that is unlikely or false. This demo did it trained on the white paper: it wrote “A gateway is not ready”, stitched together from two true sentences, “A gateway screens, logs and enforces policy” and “Our data is not ready”. Large models do the same with facts, figures and sources, only far more convincingly. That is why answers that matter are grounded in your own documents and checked by a person.
A model writes the most plausible continuation of what it was trained on. Trained on a vast share of human writing, that is an excellent starting point. Going beyond what already exists still needs your own judgement and creativity.
Chapter 5
The neural network is written with TensorFlow.js, the JavaScript edition of Google's TensorFlow, the library behind many Python AI projects. Instead of running on a server, it runs on your own device: training uses your graphics chip through WebGL. Nothing is installed and no text leaves your browser.
The architecture is a decoder-only transformer, the GPT design: text goes in as tokens, passes through stacked transformer blocks (attention plus a small hidden layer), and comes out as a probability for every possible next token.
| Feature | This model | GPT-3 (2020) |
|---|---|---|
| Parametersnumbers the model adjusts while learning | 62,496 | 175 billion |
| Transformer blocks (layers)each: attention, then a hidden layer | 2 | 96 |
| Attention heads per blockparallel “views” on the context | 4 | 96 |
| Embedding sizenumbers describing each token | 48 | 12,288 |
| Hidden layer per blockfeed-forward neurons, ReLU activation | 192 | 49,152 |
| Context windowhow far back it can read | 32 tokens | 2,048 tokens |
| Vocabularydistinct tokens it knows | – | 50,257 pieces |
| Trainingoptimizer, batch, objective | Adam · 16 × 32 tokens | Adam · 3.2 M tokens |
| Runs onwhere training happens | your browser | data centre |
Both models also use layer normalisation and residual connections, and learn by minimising cross-entropy: how surprised they are by the true next token. GPT-3 figures from Brown et al., “Language Models are Few-Shot Learners”, 2020.
And today's models? The providers of current frontier models, such as Claude Opus and Claude Fable, GPT and Gemini, do not publish these figures. Open-weight models do, and show the scale: Meta's Llama 3.1 405B (2024) uses vectors of 16,384 dimensions, 126 blocks and 405 billion parameters; DeepSeek-V3 (2024) uses 7,168 dimensions, 61 blocks and 671 billion parameters, of which only about 37 billion work on each token (“mixture of experts”). The vector size grows only moderately; models grow mainly through more blocks, more parameters and, above all, far more training data.
Chapter 6
Almost every idea on this page is decades old. What changed is that we finally have the computing power and the data to train them at scale. The key papers are listed in the appendix.
McCulloch and Pitts describe a neuron as a weighted sum and a threshold. Still the basic unit today.
Turing asks the question and proposes the imitation game.
Rosenblatt builds a machine that learns its own weights from examples.
Minsky and Papert show the limits of single-layer networks. Funding dries up for years.
Fukushima's layered network for recognising patterns, forerunner of today's image networks.
Rumelhart, Hinton and Williams popularise how multi-layer networks learn: the chain rule from 17th-century calculus.
Hochreiter and Schmidhuber (TU Munich, IDSIA Lugano) give networks a memory for sequences.
AlexNet wins ImageNet by a wide margin, trained on two gaming graphics cards.
“Attention Is All You Need” introduces the architecture used on this page.
The same ideas, scaled up with huge data and compute, reach everyone.
Weighted sums, derivatives, probability. A good engineering student knows every equation in this model.
Graphics chips built for video games turned out to be ideal for training networks. Your browser uses one right now.
The internet made trillions of words available in digital format. Scale, not a secret formula, made the leap.
It is all maths: multiplications, additions and probabilities.
Which raises an old question in a new way: if a few equations and enough data can produce language, how much of our own thinking is maths too?
Part 3
The same machinery runs the models you use at work, at a vastly larger scale. Here is what follows for leaders.
Built from scratch, trained in your browser
e.g. Llama, Mistral, Qwen, DeepSeek
e.g. Claude, GPT, Gemini
All three share the same core: tokens, embeddings, attention, next-token prediction. The difference is scale, training data and who controls the weights. That is why my white paper proposes tiering workloads: rented frontier models for most work, self-hosted open-weight models for the most sensitive data.
The model is only the engine. The harness is everything you build around it so it does useful, safe work in your company: your data, your rules, your workflows, your connections and your checks. The engine can be swapped for a better one every year. The harness carries your know-how, so it is what you should own.
A model has no memory between calls. Its context window is only what it can read in one go, and it is emptied after every answer. Every time you ask something, the harness assembles the context again: instructions, the conversation so far, relevant documents and stored memory. So the context belongs to the harness, not to the model. Swap the model, and the harness hands the same context to the new one, like a project file handed to a new consultant.
↻ The loop runs on every session, so context and memory evolve on their own.
Even this page is a tiny harness: the tokenizer, the temperature slider and the interface around a model that you can retrain at any time. Swap the training text and the model changes; the page stays the same.
A language model writes the most plausible continuation of everything it was trained on. Trained on a vast share of human writing, that is an excellent starting point: a strong first draft in seconds. Going beyond what already exists still needs your own judgement and creativity, and answers that matter need grounding in your documents and a human check.
The method fits on one page; the leap comes from data and computing power, with diminishing returns on every extra step. Training a frontier model is a race few can afford, and each generation is overtaken by a cheaper, better one. So companies should not bet on one model. They should own the interface around it (their data, workflows, rules and checks) and keep the model behind it swappable.
No budget, no project, no vendor. Anyone in leadership can see how this works first-hand. Waiting is a choice, not a strategy.
Excuses are not a strategy. My white paper on why executives must adopt AI now, and how to do it with control over data, cost and vendors.
About
Dipl.-Ing. · MBA
Operator and AI leader in aerospace and space.
Most recently Chief Operating Officer of Beyond Gravity. Before that, as Head of its Satellites Division, I led the turnaround of the business and returned it to profitability. Earlier roles at SR Technics, Jet Aviation, Ed. Züblin and Roland Berger, across Switzerland, Germany, Austria, Sweden and the United States.
I believe AI cannot be delegated down the line: leaders have to understand it first-hand. This lab is part of that, alongside MIT Professional Education's Applied AI and Data Science Program and my white paper “Excuses are not a strategy”.
Today's lead of US models is not only the result of venture capital. It rests on decades of academic and industrial research, much of it publicly funded, and most of it done in the United States. Canada, Europe and Japan supplied essential pieces too, and many of their researchers later joined US labs. Today's models are the natural consequence.
| Year | Paper and authors | Where | What it added |
|---|