A tiny GPT you can watch write.

Mbind is a 2.2-million-parameter language model trained on La Fontaine, Hugo, Maupassant and a few others. It runs in your browser and hides nothing: the tokens, the attention, the probabilities. It writes in French only.

2.2M parameters4 layers256-token contexttrained in 4 min on a CPU

Writing…

What it considered

Where it looked

It reads neither letters nor words.

Text is first cut into tokens: pieces of words learned from the books themselves. A common word fits in one token, a rare one takes several. The model only sees their numbers.

Type a sentence to see how it gets cut.

At every token, it looks back.

To choose what comes next, each attention head spreads its gaze over the earlier tokens. Some fix on the previous token, others on punctuation or the start of the sentence. Nobody taught them these roles: they appeared during training.

At first, it writes gibberish.

Its weights start out random. Then it reads, guesses the next token, gets it wrong, adjusts: 1 million times. Spelling comes fast, punctuation later, meaning never really.

What it can’t do

Understand what it writes

It predicts the next token, nothing more. Its sentences look French; often they mean nothing.

Remember for long

It only sees the last 256 tokens, about 750 characters. Beyond that, the start of the text no longer exists for it.

Talk about anything else

It has read 20 books, mostly from the 19th century. No news, no English, and sometimes La Fontaine’s old spelling (“il étoit”).

Compete with the big ones

2.2M parameters, tens of thousands of times fewer than a large modern model. On purpose: small enough to look at everything.

Give it a first sentence.