Oliver Grassmann · AI lab

A small language model, trained in your browser

This is a real GPT-style transformer, the same kind of architecture that sits behind ChatGPT and Claude, only more than two million times smaller than GPT-3. It trains right here on your device; no data leaves your browser.

  1. PlayTrain the model and watch it learn. About 2 minutes.
  2. UnderstandSix short chapters on what you just saw.
  3. What it meansFrom this demo to AI in your company.

In 30 seconds: how it trains

The model predicts the next token, calculates its error against the real text, and adjusts its weights to lower that error. Then it does it again, hundreds of times. That is all training is.

Tokens

The text is cut into pieces and each piece gets a number. This model learns about 250 word pieces from its text; large models use about 100,000.

Embeddings

Each token becomes a vector: a list of 48 numbers. During training, tokens used in similar ways end up with similar vectors.

Attention

For every position, the model decides which earlier tokens matter for guessing the next one. Two layers with four attention heads each.

Learning

It predicts the next token and calculates the error (the loss). Then it nudges every weight a little to lower that error. Repeat hundreds of times.

Part 1

Play

Follow the four steps. Each one links to the chapter that explains it.

1Choose a text

2Train

–parameters
–vocabulary
0training step
–loss

Loading TensorFlow.js…

Watch for: a steep drop first, then a long flattening tail. Why? Chapter 3 →

3Let it write

0.8
Before training, the model writes random letters. Try it, then train and try again.

Try: press it twice and compare, and look for sentences that sound right but are not in the text (that is a hallucination), then move Creativity (temperature) to 0.2 and to 1.5. Train more and try 1.5 again: a better-trained model stays readable at higher temperature. Why is it different every time? Chapter 4 →

4Look inside

Its next guess

Type anything. The bars show the model's guesses for the next token, live, also while it trains. Chapter 2 →

Training diary

Every 50 steps the model writes a sample. Watch noise turn into language. The weights changing live are shown in Chapter 2.

  • Not startedPress Train model.

Part 2

Understand

Six short chapters, in the order the model works: text becomes tokens, tokens flow through the model, the model learns, then writes.

  1. Tokens
  2. Inside the model
  3. Learning
  4. Writing
  5. The model card
  6. History

Chapter 1

Tokens: how text becomes numbers

A model never sees words, only numbers. The tokenizer decides how text is cut into pieces. Large models learn their pieces with byte-pair encoding (BPE): start with single letters, then keep merging the pair that appears most often. Watch it happen on your own sentence.

0characters
0tokens
0vocabulary
  1. Split the text into single letters.
  2. Count every pair of neighbours.
  3. Merge the most frequent pair into one new token.
  4. Repeat until no pair appears twice.

    Fewer tokens means the model sees more meaning per step. GPT-4's tokenizer learned about 100,000 such pieces from a huge text collection; the model above learns up to 250 pieces from its own training text.

    Chapter 2

    Inside the model: embeddings, attention, blocks

    Embeddings

    Every token becomes a vector of 48 numbers, its embedding: a point in a 48-dimensional space. At first these points are random. During training, tokens used in similar ways move close together. That is how meaning is stored: as positions in a space of numbers. GPT-3 uses vectors of 12,288 numbers, i.e. 12,288 dimensions.

    Why 48? It is a design choice by whoever builds the model, not a law of nature. Each extra dimension adds capacity to store patterns, but also parameters and compute. 48 is enough for one page of text and still trains in a browser in a minute. It also has to split evenly across the 4 attention heads: each head works with 12 of the 48 numbers.

    Blocks, like floors

    The model is built from two identical blocks stacked on top of each other, like two floors of an office. On each floor, two things happen in turn: the tokens look at each other (attention), then each token works through what it has gathered (the hidden layer). Block 1 is the ground floor and learns simple patterns; block 2 builds on its results. GPT-3 has 96 such floors.

    Attention: query, key, value

    The attention weights turn each token's vector into three new vectors. The query asks “what am I looking for?”, the key says “this is what I offer”, the value is the information passed on. Where a query matches a key, the token pays attention and takes in that value. In “Look inside”, the amber chips show where it looks.

    The hidden layer: 192 neurons

    In short: 192 small pattern detectors check each token's vector. The ones that fire add what they found, and the enriched vector moves on to the next block. This is where the model stores most of what it has learned.

    1. WeighEach of the 192 neurons computes one weighted sum of the token's 48 numbers: one column in the weights image. 48 → 192 numbers.
    2. Fire or stay silentThe activation function decides. This model uses ReLU: positive values pass, negative ones become zero. A neuron either reports its pattern or stays silent.
    3. Fold backA second set of weights turns the 192 values back into 48 numbers. They are added to the token's vector: it keeps what it had and gains what the neurons found. 192 → 48.
    4. Pass onThe next block receives these enriched vectors, one per token, and starts again with attention.

    Why the activation matters: without this “bend”, any stack of layers would collapse into one big multiplication, and the network could only learn straight-line relationships. GPT models use a smoother variant (GELU); early networks used the S-shaped sigmoid.

    ModelNeurons per blockBlocks
    This model1922
    GPT-3 (2020)49,15296
    Llama 3.1 405B (open-weight)53,248126
    DeepSeek-V3 (open-weight)256 experts, 8 used per token256 × 2,04861
    Frontier modelsClaude, GPT, Gemininot published–

    See it: the weights of block 1

    Its weights, changing

    Each pixel is one weight of block 1, the model's first layer: about 16,000 of its parameters. Press Train in Part 1 and watch faint random noise turn into structure, live.

    Attention weights · 48 × 144 = 6,912 weights
    Hidden layer · 48 × 192 = 9,216 weights
    positivenegativebrighter = stronger

    Chapter 3

    Learning: guess, measure, adjust

    The loop

    The model reads a piece of the training text and guesses every next token. The loss measures how wrong those guesses were. Then the optimizer (Adam) nudges every weight a little in the direction that would have made the guess less wrong. One step. Repeat hundreds of times. This is backpropagation, the chain rule from calculus.

    Watching the weights change

    Before training, the weights are faint random noise: the model knows nothing. Early on, the pictures change quickly and the loss falls fast. Later, the changes get smaller and the patterns settle. Stripes and bands appear where groups of weights learn to work together.

    Diminishing returns: a power law

    The first steps bring big gains, each further step less. Researchers measured that a language model's loss falls as a power law of compute, data and parameters (Kaplan et al., 2020): every tenfold increase in compute buys roughly the same small improvement.

    Why it matters

    Even when the curve flattens, this model is far from usable. That is why training real models costs so much in compute and parameters, and why it still gets cheaper as the cost per computation falls. A low loss here means it has nearly memorised one page; a big model needs far more text to generalise.

    Chapter 4

    Writing: one token at a time

    What “Let it write” does

    1. Read: the model takes your start text as tokens.
    2. Rate: it computes a probability for every token in its vocabulary to come next.
    3. Roll: it picks one by a weighted dice roll. Likely tokens win often, unlikely ones sometimes.
    4. Repeat: the picked token is added and the loop starts again. It never plans a sentence ahead; it only sees the last 32 tokens.

    Why the text is different every time

    Because of step 3. The probabilities are the same each time, but the dice roll differs, and one different token changes everything after it. ChatGPT and Claude work exactly like this, which is why the same question gets slightly different answers.

    Temperature sets the creativity

    The Creativity slider is the temperature. It loads the dice: at 0.2 the model almost always takes the favourite: consistent, but repetitive. At 1.5 the outsiders get a real chance: more original, but more mistakes. Many business tools run at low temperature for exactly this reason.

    Better trained, more headroom. The better a model is trained, the higher the temperature can go before it talks nonsense. A well-trained model is confident: the right token stands far ahead of the alternatives, so it takes much more “loading of the dice” before the outsiders start to win. Try it: let it write at 1.5 after a short training, then train more and try 1.5 again.

    Why models hallucinate

    Every single token is a likely choice given the ones before it, so the sentence reads well. But the model checks only plausibility, step by step, never truth. A chain of likely pieces can add up to a statement that is unlikely or false. This demo did it trained on the white paper: it wrote “A gateway is not ready”, stitched together from two true sentences, “A gateway screens, logs and enforces policy” and “Our data is not ready”. Large models do the same with facts, figures and sources, only far more convincingly. That is why answers that matter are grounded in your own documents and checked by a person.

    It predicts from what it has seen

    A model writes the most plausible continuation of what it was trained on. Trained on a vast share of human writing, that is an excellent starting point. Going beyond what already exists still needs your own judgement and creativity.

    Chapter 5

    The model card: how big is it?

    The neural network is written with TensorFlow.js, the JavaScript edition of Google's TensorFlow, the library behind many Python AI projects. Instead of running on a server, it runs on your own device: training uses your graphics chip through WebGL. Nothing is installed and no text leaves your browser.

    The architecture is a decoder-only transformer, the GPT design: text goes in as tokens, passes through stacked transformer blocks (attention plus a small hidden layer), and comes out as a probability for every possible next token.

    FeatureThis modelGPT-3 (2020)
    Parametersnumbers the model adjusts while learning62,496175 billion
    Transformer blocks (layers)each: attention, then a hidden layer296
    Attention heads per blockparallel “views” on the context496
    Embedding sizenumbers describing each token4812,288
    Hidden layer per blockfeed-forward neurons, ReLU activation19249,152
    Context windowhow far back it can read32 tokens2,048 tokens
    Vocabularydistinct tokens it knows–50,257 pieces
    Trainingoptimizer, batch, objectiveAdam · 16 × 32 tokensAdam · 3.2 M tokens
    Runs onwhere training happensyour browserdata centre

    Both models also use layer normalisation and residual connections, and learn by minimising cross-entropy: how surprised they are by the true next token. GPT-3 figures from Brown et al., “Language Models are Few-Shot Learners”, 2020.

    And today's models? The providers of current frontier models, such as Claude Opus and Claude Fable, GPT and Gemini, do not publish these figures. Open-weight models do, and show the scale: Meta's Llama 3.1 405B (2024) uses vectors of 16,384 dimensions, 126 blocks and 405 billion parameters; DeepSeek-V3 (2024) uses 7,168 dimensions, 61 blocks and 671 billion parameters, of which only about 37 billion work on each token (“mixture of experts”). The vector size grows only moderately; models grow mainly through more blocks, more parameters and, above all, far more training data.

    Chapter 6

    History: AI did not come out of the blue

    Almost every idea on this page is decades old. What changed is that we finally have the computing power and the data to train them at scale. The key papers are listed in the appendix.

    United StatesEuropeCanadaAsia
    1. 1943US · Chicago

      The artificial neuron

      McCulloch and Pitts describe a neuron as a weighted sum and a threshold. Still the basic unit today.

    2. 1950UK · Manchester

      Can machines think?

      Turing asks the question and proposes the imitation game.

    3. 1958US · Cornell

      The perceptron

      Rosenblatt builds a machine that learns its own weights from examples.

    4. 1969US · MIT

      First AI winter

      Minsky and Papert show the limits of single-layer networks. Funding dries up for years.

    5. 1980Japan · NHK Labs

      The neocognitron

      Fukushima's layered network for recognising patterns, forerunner of today's image networks.

    6. 1986US · San Diego

      Backpropagation

      Rumelhart, Hinton and Williams popularise how multi-layer networks learn: the chain rule from 17th-century calculus.

    7. 1997Germany · Switzerland

      Long short-term memory

      Hochreiter and Schmidhuber (TU Munich, IDSIA Lugano) give networks a memory for sequences.

    8. 2012Canada · Toronto

      Deep learning takes off

      AlexNet wins ImageNet by a wide margin, trained on two gaming graphics cards.

    9. 2017US · Google

      The transformer

      “Attention Is All You Need” introduces the architecture used on this page.

    10. 2022US · OpenAI

      ChatGPT

      The same ideas, scaled up with huge data and compute, reach everyone.

    The maths is old

    Weighted sums, derivatives, probability. A good engineering student knows every equation in this model.

    The compute is new

    Graphics chips built for video games turned out to be ideal for training networks. Your browser uses one right now.

    The data is new

    The internet made trillions of words available in digital format. Scale, not a secret formula, made the leap.

    It is all maths: multiplications, additions and probabilities.

    Which raises an old question in a new way: if a few equations and enough data can produce language, how much of our own thinking is maths too?

    Part 3

    What it means for your company

    The same machinery runs the models you use at work, at a vastly larger scale. Here is what follows for leaders.

    From this demo to the models you use at work

    This demo

    Built from scratch, trained in your browser

    Size
    About 60,000 parameters, trained on one page of text
    Who has the weights
    You, in your browser, for the moment
    Can do
    Show how the machinery works. Nothing more.

    Open-weight model

    e.g. Llama, Mistral, Qwen, DeepSeek

    Size
    From about a billion to several hundred billion parameters; the weights are published
    Who has the weights
    Anyone can download them and run them on their own servers
    Can do
    Strong everyday work. Sensitive data never has to leave your own infrastructure, and you can fine-tune it on your documents.

    Frontier model

    e.g. Claude, GPT, Gemini

    Size
    Not published; trained on a large share of the public internet, books and code, then refined with human feedback
    Who has the weights
    Only the provider; you rent access through an app or API
    Can do
    The best reasoning, writing and coding available, plus tools such as search and documents. Enterprise contracts exclude training on your data.

    All three share the same core: tokens, embeddings, attention, next-token prediction. The difference is scale, training data and who controls the weights. That is why my white paper proposes tiering workloads: rented frontier models for most work, self-hosted open-weight models for the most sensitive data.

    What is the harness?

    The model is only the engine. The harness is everything you build around it so it does useful, safe work in your company: your data, your rules, your workflows, your connections and your checks. The engine can be swapped for a better one every year. The harness carries your know-how, so it is what you should own.

    People and processesemployees, customers, daily tasks
    The harness: owned by you
    Your datadocuments, mail, contracts, drawings, retrieved with sources
    Rules and accessgateway, who may use which data, data tiers, logging
    Prompts and workflowshow a task is asked, step by step, in your language
    Tools and systemsERP, CRM, mail, calendar: the model can look up and act
    Model slotrented, swappable
    Frontier modelOpen-weightNext year's model
    Cost routingsimple tasks to small, cheap models; hard ones to the best
    Checks and reviewhuman sign-off, error measurement, tests before go-live
    Interfacewhere people actually work: chat, apps, Teams, documents
    Memory and feedbackwhat worked, corrections, improving over time
    Resultsdrafts, answers, decisions, actions

    Where does the context live?

    A model has no memory between calls. Its context window is only what it can read in one go, and it is emptied after every answer. Every time you ask something, the harness assembles the context again: instructions, the conversation so far, relevant documents and stored memory. So the context belongs to the harness, not to the model. Swap the model, and the harness hands the same context to the new one, like a project file handed to a new consultant.

    Keep it automatically: the memory loop

    1. WorkEmployees work with the assistant as usual: chats, documents, tasks.
    2. In the backgroundExtractAfter each session, a small model pulls out what lasts: decisions, facts, preferences, open points.
    3. In the backgroundCurateNew entries are merged, duplicates removed, outdated ones replaced, with date and source kept.
    4. RetrieveBefore the next call, the harness fetches only the memory relevant to the task, whichever model answers.

    ↻ The loop runs on every session, so context and memory evolve on their own.

    Practical recommendation

    1. Make memory a harness service, not a chat feature. Built-in memory in ChatGPT or Claude is convenient, but it belongs to the vendor's product. Put the memory loop in your own gateway or platform, so it survives a model change. Open-source memory layers such as Mem0 or Letta, or your platform's agent framework, provide the extract, curate and retrieve steps.
    2. Store it in open formats you own. Plain text or Markdown files and a search index on your own storage (SharePoint, Drive, a database). One file per person, project and topic, readable by people, not only by machines.
    3. Separate three layers. Instructions (rules, house style: maintained deliberately), documents (indexed automatically from where they already live), memory (written automatically by the loop).
    4. Give people control. Everyone can see, correct and delete what is stored about their work. Apply the same data tiers, retention rules and access rights as for any other company data.
    5. Test the swap. Once a quarter, run a few typical tasks with a different model on the same context. If the answers hold up, you are model-agnostic in practice, not just on paper.

    Even this page is a tiny harness: the tokenizer, the temperature slider and the interface around a model that you can retrain at any time. Swap the training text and the model changes; the page stays the same.

    What it teaches an executive

    It predicts from what it has seen.

    A language model writes the most plausible continuation of everything it was trained on. Trained on a vast share of human writing, that is an excellent starting point: a strong first draft in seconds. Going beyond what already exists still needs your own judgement and creativity, and answers that matter need grounding in your documents and a human check.

    Own the harness, stay model-agnostic.

    The method fits on one page; the leap comes from data and computing power, with diminishing returns on every extra step. Training a frontier model is a race few can afford, and each generation is overtaken by a cheaper, better one. So companies should not bet on one model. They should own the interface around it (their data, workflows, rules and checks) and keep the model behind it swappable.

    Understanding is two minutes away.

    No budget, no project, no vendor. Anyone in leadership can see how this works first-hand. Waiting is a choice, not a strategy.

    Excuses are not a strategy. My white paper on why executives must adopt AI now, and how to do it with control over data, cost and vendors.

    About

    Oliver Grassmann

    Dipl.-Ing. · MBA

    Operator and AI leader in aerospace and space.

    Most recently Chief Operating Officer of Beyond Gravity. Before that, as Head of its Satellites Division, I led the turnaround of the business and returned it to profitability. Earlier roles at SR Technics, Jet Aviation, Ed. Züblin and Roland Berger, across Switzerland, Germany, Austria, Sweden and the United States.

    I believe AI cannot be delegated down the line: leaders have to understand it first-hand. This lab is part of that, alongside MIT Professional Education's Applied AI and Data Science Program and my white paper “Excuses are not a strategy”.

    Connect on LinkedIn

    Appendix: the papers behind it

    Today's lead of US models is not only the result of venture capital. It rests on decades of academic and industrial research, much of it publicly funded, and most of it done in the United States. Canada, Europe and Japan supplied essential pieces too, and many of their researchers later joined US labs. Today's models are the natural consequence.

    YearPaper and authorsWhereWhat it added