How it works

Mbind does one thing: given some text, it assigns a probability to each of the 2,048 tokens that could come next. Writing a sentence means repeating that calculation, one token at a time.

Tokens

A neural network only handles numbers, so the text first has to become a sequence of integers.

Characters would have worked: about a hundred symbols, very simple, but very long sequences, and a model that has to rebuild everything letter by letter. Mbind uses a BPE (byte-pair encoding) tokenizer trained on its own books instead. Start from characters, find the most frequent neighbouring pair, merge it into a new token, and repeat 1,945 times. On these books the first merge glues a space to a “d” (the start of “de”, “dans”, “dit”), the third joins “e” and “s”; the last ones produce whole words such as “sourire”, “chapeau” or “Fauchelevent”.

The result: a 2,048-token vocabulary, about 3.0 characters per token, with common words in one piece. The cost: the model never sees letters. That “Corbeau” and “corbeau” are the same word is something it has to learn.

Embeddings

Each token number points to a row in a big table: a vector of 192 numbers. A second table gives one vector per position (1st, 2nd… 256th), added to the first; without it the model would not know the word order.

These numbers start out random. Training moves them, and tokens that are used the same way end up with similar vectors.

Attention

This is the core of the transformer. For each token the model computes three vectors: a query (what it is looking for), a key (what it offers) and a value (what it passes on). The dot product between one token’s query and each earlier token’s key says how well they fit. Those scores go through a softmax, and the token receives the average of the values, weighted accordingly.

A token only looks back: before the softmax, the scores of later tokens are set to minus infinity. That mask is what draws the triangle in the attention maps.

Each layer has 6 heads, each with its own queries, keys and values, and Mbind stacks 4 layers: 24 ways of looking at the text. Between two attention steps, a small network (the MLP) transforms each vector on its own; much of what the model has picked up lives there. Each block adds its result to the vector instead of replacing it (the residual connection), which keeps training stable.

Softmax

After the last layer, the last token’s vector is compared with the embedding of each of the 2,048 tokens. That gives 2,048 scores, the logits. The softmax turns them into probabilities: the exponential of each score, divided by the sum of them all. Everything becomes positive, the total is 1, and one extra point of score multiplies the probability by e, about 2.7.

Sampling

Now a token has to be chosen. Always taking the most likely one produces text that soon goes round in circles. So we draw at random according to the probabilities, with three settings:

  • temperature divides the logits before the softmax. Below 1, likely choices crush the rest; above 1, everything flattens and the text wanders off;
  • top-k keeps only the k most likely tokens;
  • top-p keeps the most likely ones until their total reaches p.

The model’s real probabilities, renormalised over its ten best candidates.

The seed fixes the random draw: same seed, same settings, same text.

Training

The corpus: 20 public-domain books from Project Gutenberg (La Fontaine, Hugo, Maupassant, Flaubert, Daudet, Voltaire), 8.2 million characters. One passage in twenty is set aside and never used for training: that is the validation set, which tells us how the model does on text it has not seen.

Training repeats a single operation: take 32 excerpts of 256 tokens, have the model predict every next token, measure the error, then nudge the 2.2 million weights in the direction that reduces it. The error is the cross-entropy: minus the log of the probability given to the real token. The nudge comes from backpropagation and the AdamW optimiser. 100 steps, 4 min on 6 cores of a laptop, no GPU.

Nobody showed it a grammar rule. Everything it knows comes from that one instruction: guess what comes next.

In the browser

The weights are exported as float16 (4.4 MB) and the maths is written by hand in TypeScript, without a library, in a Web Worker. A check compares its output with PyTorch on the same text: same tokens, logits identical to within 10⁻⁵. Keys and values already computed are kept in memory, so each new token costs a single pass over the weights. Nothing is sent to a server.

Limits

Mbind imitates a style; it understands nothing. It invents names, mixes Hugo with La Fontaine, loses the thread from one sentence to the next, forgets the start of the text beyond 256 tokens and only speaks the French of the books it read, old spellings included.

A large modern model rests on the same ideas: tokens, attention, softmax, predicting what comes next. It has tens of thousands of times more parameters, data and compute, plus a second training phase to follow instructions, which Mbind does not have.