<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Yash Singh]]></title><description><![CDATA[I’m Yash Singh, a CS student exploring AI, machine learning and deep learning. I document what I learn and build through hands-on experiments and projects.]]></description><link>https://yashksingh.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6af20f149c80d9d1b498be7f/ce733a4c-a4f0-4a19-bc05-cab4501f7698.png</url><title>Yash Singh</title><link>https://yashksingh.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Fri, 20 Nov 2026 16:53:52 GMT</lastBuildDate><atom:link href="https://yashksingh.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Building BPE from Scratch: How a Tokenizer Learns Its Vocabulary]]></title><description><![CDATA[How I built a BPE tokenizer from scratch in Python: merge rules, pre-tokenization, special tokens, faster training, and why plain BPE fails on Unicode]]></description><link>https://yashksingh.hashnode.dev/transformer-from-scratch-part-2-bpe</link><guid isPermaLink="true">https://yashksingh.hashnode.dev/transformer-from-scratch-part-2-bpe</guid><category><![CDATA[Machine Learning]]></category><category><![CDATA[nlp]]></category><category><![CDATA[Python]]></category><category><![CDATA[AI]]></category><category><![CDATA[ML]]></category><dc:creator><![CDATA[Yash Singh]]></dc:creator><pubDate>Thu, 19 Nov 2026 09:41:35 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af20f149c80d9d1b498be7f/c863ef3d-0e03-43e0-84b2-ea4c3f6e8e97.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In the last post I went through what tokens are and why subword tokenization wins. This post is about how I actually built one: Byte Pair Encoding (BPE), step by step, with every problem I ran into along the way.</p>
<h2>Starting simple: a character tokenizer</h2>
<p>This was my roadmap for the tokenizer:</p>
<pre><code>LEVEL 0  — Foundations
LEVEL 1  — Character tokenizer
LEVEL 2  — Word tokenizer
LEVEL 3  — Vocabulary + Token IDs
LEVEL 4  — Special tokens + padding
LEVEL 5  — Subword intuition
LEVEL 6  — BPE from scratch
LEVEL 7  — Training a tokenizer on a corpus
LEVEL 8  — Encoding + decoding
LEVEL 9  — Handling unknown/rare text
LEVEL 10 — Byte-level BPE
LEVEL 11 — Production-style tokenizer
LEVEL 12 — Connect it to the Transformer
</code></pre>
<p><strong>Important:</strong> a tokenizer has two jobs:</p>
<ol>
<li>Converting text into token IDs (<strong>encoding</strong>)</li>
<li>Converting token IDs back into text (<strong>decoding</strong>)</li>
</ol>
<p>I started with the simplest one, a character tokenizer. Here each character in a text is a token, so character = token. But that's not true for all tokenizers. More generally, <strong>a token is an atomic unit produced by a tokenizer.</strong></p>
<pre><code class="language-python">text = "hello world!"
stoi = {}  # string to id
itos = {}  # id to string

chars = sorted(list(set(text)))
for i, char in enumerate(chars):
    stoi[char] = i
    itos[i] = char
vocab_size = len(chars)

def encoder(s):
    return [stoi[char] for char in s]

def decoder(t):
    return [itos[i] for i in t]
</code></pre>
<pre><code>Vocabulary: [' ', '!', 'd', 'e', 'h', 'l', 'o', 'r', 'w']
Vocab size: 9
Encoded: [4, 3, 5, 5, 6]
Decoded: ['h', 'e', 'l', 'l', 'o']
</code></pre>
<p>It worked, but two problems showed up straight away.</p>
<p><strong>First problem: unknown characters.</strong> If we use a "?", that's not in our vocabulary, so the encoder throws an error. For this we use a special token, <code>&lt;UNK&gt;</code> (unknown):</p>
<pre><code class="language-python">UNK_token = "&lt;UNK&gt;"
vocab = [UNK_token] + chars

def encoder(s):
    return [stoi[char] if char in stoi else stoi[UNK_token] for char in s]
</code></pre>
<p>So now, if a character is not in the vocabulary, it gets the <code>&lt;UNK&gt;</code> token ID. But then decoding produces <code>&lt;UNK&gt;</code> instead of the original characters: "hello <code>&lt;UNK&gt;&lt;UNK&gt;</code>". So we have lost the original information.</p>
<p><strong>This is why modern tokenizers try to avoid <code>&lt;UNK&gt;</code>.</strong> They try to make the vocabulary capable of representing essentially arbitrary text, often using subwords or bytes.</p>
<p><strong>Second problem: sequence length.</strong> When we tokenized "hello", it produced 5 tokens. "transformers" would be 12 tokens, and a paragraph would run into the thousands.</p>
<h2>The idea behind BPE</h2>
<p>Byte Pair Encoding (BPE) builds a vocabulary by starting from single characters and repeatedly merging the most frequent adjacent pair into a new token.</p>
<p>Imagine our corpus is:</p>
<pre><code>low
lower
lowest
newer
newest
</code></pre>
<p>With character tokenization, we start with every character as a token:</p>
<pre><code>l o w
l o w e r
l o w e s t
n e w e r
n e w e s t
</code></pre>
<p>Now look at the pattern. We repeatedly see the same pairs: l + o, o + w, w + e, e + r, e + s, s + t, n + e.</p>
<p>So if we merge l + o = lo, then low = lo + w. And if low is common, that also becomes a token.</p>
<p>"we" appears 4 times (lower, lowest, newer, newest), so it becomes a token. Our vocabulary size increases, but every occurrence of "we" now uses 1 token, not 2. Then we repeat: some other pair is now the most frequent, so it becomes a token and gets added to the vocabulary, and so on. Eventually, "lowest" might become low + est: only 2 tokens instead of 6.</p>
<p><strong>Important:</strong> we are not manually defining that "low" is a word and "est" is a suffix. We just say: these pieces occur together, so let's make them a token. That's why BPE can discover useful pieces by itself.</p>
<p><strong>BPE is closely related to compression.</strong> We start with "l o w e s t" and eventually discover lo, then low, and then est. We're saying: instead of storing these common combinations as separate pieces, combine them and give them a compact representation. (In fact, BPE started out in 1994 as a data compression algorithm, and was only adapted for language models in 2016.)</p>
<p>My first version of the training loop, on "low lower lowest lowering":</p>
<pre><code class="language-python">from collections import Counter

def count_pairs(words):
    counts = Counter()
    for word in words:
        for i in range(len(word) - 1):
            pair = (word[i], word[i + 1])
            counts[pair] += 1
    return counts

def merge_pairs(words, pairs):
    new_words = []
    for word in words:
        new_word = []
        i = 0
        while i &lt; len(word):
            if i &lt; len(word) - 1 and (word[i], word[i + 1]) in pairs:
                new_word.append(word[i] + word[i + 1])
                i += 2
            else:
                new_word.append(word[i])
                i += 1
        new_words.append(new_word)
    return new_words

for step in range(10):
    pair_counts = count_pairs(word_char_list)
    if not pair_counts:
        break
    most_common_pair = pair_counts.most_common(1)[0][0]
    new_token = "".join(most_common_pair)
    word_char_list = merge_pairs(word_char_list, {most_common_pair})
    vocab.append(new_token)
</code></pre>
<h3>When do we stop merging?</h3>
<p>What if we keep merging until every word becomes a token? Then the vocabulary will explode. That means a lot more parameters, and we'd slowly recreate the word tokenization problem. So we need to stop merging at some point, for example at a target vocabulary of 10,000 tokens.</p>
<p>More merges mean: larger vocabulary → longer tokens → fewer tokens per sentence → shorter sequences → less Transformer computation. So we have two competing costs:</p>
<pre><code>TOO FEW MERGES                      TOO MANY MERGES
      ↓                                    ↓
small vocabulary                    huge vocabulary
      ↓                              ↓              ↓
long sequences               large embeddings   shorter sequences,
      ↓                                         less computation
expensive attention
</code></pre>
<p>So tokenizer design is basically trying to find a <strong>sweet spot</strong>. That's one of the deeper reasons tokenization matters so much.</p>
<h2>Training vs encoding</h2>
<p>Up until now we were doing training, but a tokenizer has two different jobs:</p>
<ul>
<li><strong>Training:</strong> learn the vocabulary and the merge rules</li>
<li><strong>Encoding:</strong> use the vocabulary and merge rules on new text</li>
</ul>
<p>Given a training corpus of low, lower, lowest and lowering, we started with characters and then repeatedly: find the most frequent pair → merge the pair → repeat. Eventually we might learn merge rules such as:</p>
<pre><code>(l, o)    → lo
(lo, w)   → low
(e, r)    → er
(low, er) → lower
(e, s)    → es
(es, t)   → est
</code></pre>
<p>These <strong>merge rules are the knowledge learned by the tokenizer.</strong> And with every merge, our vocabulary grows:</p>
<p>$$
V_{final} = V_{initial} + N_{merges}
$$</p>
<p><strong>Saving rules:</strong> we need to save those rules, and the order matters. First l + o = lo, then e + r = er, and so on. The order matters, like a recipe.</p>
<p>So when we are encoding new words, we apply these rules. For example, to encode "lowering":</p>
<ol>
<li>Break it into characters: l + o + w + e + r + i + n + g</li>
<li>Apply the rules in order:<ol>
<li>l + o = lo, which gives lo + w + e + r + i + n + g</li>
<li>lo + w = low, which gives low + e + r + i + n + g</li>
<li>and so on…</li>
</ol>
</li>
</ol>
<p>So, even if the tokenizer doesn't know the word, it only needs to know the reusable pieces.</p>
<p>The training loop now also saves each merge:</p>
<pre><code class="language-python">merges = []
for step in range(10):
    pair_counts = count_pairs(word_char_list)
    if not pair_counts:
        break
    most_common_pair = pair_counts.most_common(1)[0][0]
    new_token = "".join(most_common_pair)
    vocab.append(new_token)
    word_char_list = merge_pairs(word_char_list, {most_common_pair})
    merges.append(most_common_pair)
</code></pre>
<p>So now we:</p>
<ol>
<li>Find the pair counts</li>
<li>Get the most common pair</li>
<li>Create a new token by joining the most common pair</li>
<li>Update the vocab by adding the token (this increases the vocab size)</li>
<li>Save the rule, for encoding new text later</li>
</ol>
<p>Then encoding takes the new text, applies the learned merge rules, and converts the result to token IDs. Decoding takes the IDs and converts them back to a string:</p>
<pre><code class="language-python">def encode_text(text, merges):
    tokens = list(text)
    for pairs_to_merge in merges:
        merged_token = "".join(pairs_to_merge)
        tokens = merge_pair(tokens, pairs_to_merge, merged_token)
    return [stoi[token] if token in stoi else stoi["&lt;UNK&gt;"] for token in tokens]

def decode_text(encoded_tokens):
    return "".join([itos[i] for i in encoded_tokens])
</code></pre>
<p>Then I tested it on "slower", which was never in the training corpus. It came out as just two token IDs, <code>[12, 22]</code>: "s" plus the "lower" token it had learned. This is a big advantage of BPE: it can encode even words it has never seen.</p>
<p><strong>Problem: whitespace.</strong> My first version ignored spaces completely. So if a new text contained a space, the space became <code>&lt;UNK&gt;</code>, because it was never in the vocabulary. To fix this, I built the initial vocabulary directly from the original text instead of from the split words. Now "lower loweing slowest glowing flowing running learning building" encoded and decoded back perfectly, spaces included.</p>
<h3>Ranking: the order the rules are applied in</h3>
<p>How the rules are applied matters. Each merge BPE learns becomes a rule, and its position in the list is its <strong>rank</strong>: the rule learned first has the lowest rank.</p>
<p>Looping over every merge in order (like my first <code>encode_text</code>) works for a small tokenizer, but it tries every single rule even for a short word. The standard way is: at each step, look at the pairs currently present and apply the one with the highest priority (the lowest rank), until no learned pair is left.</p>
<pre><code class="language-python">def apply_merges(tokens, merge_rank):
    while True:
        best_pair = None
        best_rank = float("inf")
        for i in range(len(tokens) - 1):
            pair = (tokens[i], tokens[i + 1])
            if pair in merge_rank:
                rank = merge_rank[pair]
                if rank &lt; best_rank:
                    best_rank = rank
                    best_pair = pair
        if best_pair is None:
            break

        merged_token = "".join(best_pair)
        new_tokens = []
        i = 0
        while i &lt; len(tokens):
            if i &lt; len(tokens) - 1 and (tokens[i], tokens[i + 1]) == best_pair:
                new_tokens.append(merged_token)
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1
        tokens = new_tokens
    return tokens
</code></pre>
<h2>Special tokens in practice</h2>
<p>Next I added the special tokens <code>&lt;UNK&gt;</code>, <code>&lt;PAD&gt;</code>, <code>&lt;BOS&gt;</code> and <code>&lt;EOS&gt;</code> to the BPE tokenizer, and this is where I realised some of them aren't really the tokenizer's job.</p>
<p><strong><code>&lt;UNK&gt;</code></strong> is only needed when the tokenizer cannot represent an input piece at all. Suppose our training corpus contained "low". Encoding "low" works. But if we encode "lowz", z was never in the training corpus, so the tokenizer doesn't know z, and it gets <code>&lt;UNK&gt;</code>. Note that <code>&lt;UNK&gt;</code> doesn't mean there was no BPE merge for it: if z was in the base vocabulary, it wouldn't need a merge, it could exist as it is. So "lowz" becomes low + <code>&lt;UNK&gt;</code>, and if we decode this, we cannot recover the original z.</p>
<p><strong><code>&lt;PAD&gt;</code>: padding is generally a batching concern, not a tokenizer concern.</strong> Padding only happens when multiple sequences are processed together. Suppose after tokenization we have:</p>
<pre><code>"low"    → [22]
"lower"  → [22, 5]
"lowest" → [22, 5, 8]
</code></pre>
<p>A model usually wants a rectangular tensor, so they become:</p>
<pre><code>[[22, 0, 0],
 [22, 5, 0],
 [22, 5, 8]]     where 0 = &lt;PAD&gt;
</code></pre>
<p>An important concept here is <strong>batch size</strong>, but batch size and padding are two different things:</p>
<ul>
<li><strong>Batch size</strong> = how many sequences are processed together. If batch size = 3, then sequences 1, 2 and 3 are processed together.</li>
<li><strong>Padding</strong> makes those 3 sequences the same length.</li>
</ul>
<p>And <strong>batching is decided by the training/inference system</strong>, not the tokenizer:</p>
<pre><code>Tokenizer → produces token IDs
    ↓
Data loader / inference system → chooses batch size, pads and batches sequences
    ↓
Transformer
</code></pre>
<p><strong>Context length</strong> is different again. A model might have a maximum context length of 4,096: that means one sequence cannot exceed 4,096 tokens for that model. <strong>This is independent of batch size.</strong> You could have a batch of 8 sequences of 4,096 tokens each.</p>
<p><strong>So how does the data loader decide the sequence length?</strong> There are two boundaries:</p>
<ol>
<li><strong>Maximum sequence length</strong> (the model's limit). Suppose max_length = 512. Then "hello" (5 tokens) is okay, "how are you" (3 tokens) is okay, and a very long text of 800 tokens gets truncated to 512.</li>
<li><strong>Actual batch length.</strong> Suppose one batch contains sequences of 5, 8 and 6 tokens. The batching logic can simply pad to the longest sequence <em>in that batch</em>, so the batch length is 8, not 512. That's usually much more efficient. Different batches can then have different lengths (80, 120, 200…), as long as none exceeds the model limit.</li>
</ol>
<p>And what if a sequence is longer than the limit, say 700 tokens with max_length = 512? We have choices: truncate it to 512 and throw away the rest, or split the 700 tokens into smaller sequences instead of wasting them. (In my actual training I went with splitting all the way: documents are packed back to back into 1,024-token sequences, so there's no padding at all. More on that in the training post.)</p>
<p><strong><code>&lt;BOS&gt;</code> and <code>&lt;EOS&gt;</code></strong> are different from padding. Padding is optional and handled by the training/inference system, but <code>&lt;BOS&gt;</code> and <code>&lt;EOS&gt;</code> mark the beginning and end of sequences, so the model can learn that a sequence is starting or ending. They serve as <strong>sequence boundary markers</strong>, so the encoder adds them itself:</p>
<pre><code class="language-python">tokens = ["&lt;BOS&gt;"] + tokens + ["&lt;EOS&gt;"]
</code></pre>
<h2>Whitespace, punctuation and the pre-tokenizer</h2>
<p>At this point my tokenizer was treating spaces as ordinary characters, which meant BPE could merge across word boundaries. Punctuation had the same issue.</p>
<h3>Handling whitespace</h3>
<p>There is no universal rule for how whitespace should be handled.</p>
<ol>
<li>One way is to keep the space as its own token: low + " " + lower. This makes sure BPE doesn't learn pairs involving whitespace: we simply don't count pairs where either side is a space.</li>
<li>Some BPE tokenizers deliberately allow whitespace to become part of tokens, learning things like " the", "ing" or " hello", because encoding the whitespace together with common words can be efficient.</li>
</ol>
<h3>Handling punctuation</h3>
<p>Punctuation should not be removed. It carries information:</p>
<pre><code>"You're coming."
"You're coming?"
"You're coming!"
</code></pre>
<p>Same words, completely different expression.</p>
<h3>The pre-tokenizer</h3>
<p>So before BPE, I added a <strong>pre-tokenizer</strong> that separates whitespace, punctuation and words. This defines the boundaries: what goes into BPE, and what stays as separate tokens.</p>
<pre><code class="language-python">PUNCTUATION = "!\"#$%&amp;'()*+,-./:;&lt;=&gt;?@[\\]^_`{|}~"

def pre_tokenize(text):
    tokens = []
    current_word = ""
    for char in text:
        if char.isspace() or char in PUNCTUATION:
            if current_word:
                tokens.append(current_word)
                current_word = ""
            tokens.append(char)
        else:
            current_word += char
    if current_word:
        tokens.append(current_word)
    return tokens
</code></pre>
<h3>Including whitespace in BPE training</h3>
<p>Then I hit the next problem. If we always keep whitespace as separate tokens, the token count goes up by a lot, because most words are followed by a space, and common words have spaces before or after them. The tokenizer ends up spending a token after every single word just to represent the space.</p>
<p>By including whitespace in BPE training, we allow it to learn pairs like "e ", "the " or " a". The frequency statistics eventually discover that "the" is usually followed by a space. We never tell it that "the" is a common word followed by a space; it learns that from the frequency of the word-and-space pair.</p>
<p>Tests after this change:</p>
<pre><code>learning to train → ['le', 'ar', 'n', 'ing', ' to', ' t', 'r', 'ain']
training data     → ['t', 'r', 'ain', 'ing', ' ', 'd', 'at', 'a']
hello world       → ['h', 'e', 'l', 'l', 'o', ' ', 'w', 'or', 'l', 'd']
</code></pre>
<p><strong>Problem: punctuation again.</strong> Now punctuation had the same problem whitespace had before: it was kept out of BPE training, so it could never merge with anything. So I included punctuation in BPE training too, letting it learn merges like "ing,". Now BPE training includes every segment.</p>
<h3>What my BPE learned</h3>
<p>Trained on a small corpus, it went from 78 initial tokens to a vocabulary of 200: 122 merges. The first few merges, in rank order:</p>
<pre><code>Rank 1:  ('i', 'n')   → 'in'
Rank 2:  ('s', ' ')   → 's '
Rank 3:  ('e', ' ')   → 'e '
Rank 4:  ('e', 'n')   → 'en'
Rank 5:  (' ', 't')   → ' t'
Rank 6:  ('e', 'r')   → 'er'
...
Rank 10: ('in', 'g')  → 'ing'
...
Rank 44: ('lear', 'n') → 'learn'
...
Rank 120: ('learn', 'ing') → 'learning'
</code></pre>
<p>And you can see the pieces being reused across related words:</p>
<pre><code>learn learning learner learners → [2, 121, 5, 197, 5, 121, 83, 5, 121, 83, 70, 3]
walk walking walker walkers     → [2, 196, 5, 196, 87, 5, 196, 83, 5, 196, 83, 70, 3]
</code></pre>
<p>(2 and 3 are <code>&lt;BOS&gt;</code> and <code>&lt;EOS&gt;</code>, and 5 is the space.) "learn" is token 121 everywhere, "walk" is 196, and "er" (83) and "s" (70) get reused as endings. It even handled words it never saw, like "retraining", "rebuilding" and "sleepwalking", by combining pieces it knew.</p>
<p>But not everything worked. "Hello, world!" decoded as "Hello, world<code>&lt;UNK&gt;</code>", because "!" never appeared in my small training corpus, and "GPT models" lost its G the same way. Every character the corpus didn't contain was lost. That problem comes back in a much bigger way at the end of this post.</p>
<h2>Problems I hit along the way</h2>
<h3>Problem: improper sequence boundaries during BPE training</h3>
<p>In my first implementation, the pre-tokenizer identified words, spaces, punctuation and newlines, but the training step then combined all the pieces from an entire line into a single character sequence. So a sentence like "Machine learning is a method." was effectively one long sequence:</p>
<pre><code>['M', 'a', 'c', 'h', 'i', 'n', 'e', ' ', 'l', 'e', 'a', 'r', 'n', 'i', 'n', 'g', ' ', 'i', 's', ' ', 'a', ...]
</code></pre>
<p>BPE was therefore allowed to keep merging across word boundaries. That's statistically valid for basic BPE, but it can make the tokenizer learn unnecessarily long phrase-level tokens when a phrase happens to occur often in a small training corpus. Mine actually did this: going down the merge list, it built up "s su", then "s such", then "s such a", and finally <strong>"s such as "</strong> as a single token. That's not what I wanted. I wanted BPE to learn reusable subword patterns, not entire phrases caused by accidental corpus frequency.</p>
<p><strong>Solution: use pre-tokenization to define independent BPE sequences.</strong> I redesigned the pre-tokenizer so that it doesn't just identify the different parts of the text, it directly produces the independent units BPE works on. Instead of:</p>
<pre><code>["Machine", " ", "learning", " ", "is", " ", "a", " ", "method", "."]
</code></pre>
<p>the pre-tokenizer now produces:</p>
<pre><code>["Machine", " learning", " is", " a", " method", "."]
</code></pre>
<p>Leading whitespace is attached to the following word, so whitespace can still take part in BPE merges and the tokenizer can learn tokens like " the", " to" or " learning". Punctuation stays its own unit: it can still take part in BPE if it's frequent enough, but unrelated words can never be merged together across a pre-tokenization boundary.</p>
<p>The pipeline became:</p>
<p><strong>Raw text → Pre-tokenization → Independent BPE sequences → BPE merging → Vocabulary</strong></p>
<p>This separates the responsibilities of the two stages: the pre-tokenizer decides the units within which BPE can operate, and BPE decides which pairs inside those units get merged, based purely on frequency.</p>
<h3>Problem: no morphological or positional awareness</h3>
<p>A basic BPE tokenizer learns tokens purely from the frequency of adjacent pairs. It doesn't understand prefixes, suffixes, roots or word structure. For example, "est" occurs at the beginning of <em>estimate</em> and at the end of <em>highest</em>. BPE can learn "est" as a token because it's frequent, but it doesn't understand that it plays a different role in the two words.</p>
<p>This doesn't mean we need separate token IDs for every position. The same "est" token can be used in both cases, because the surrounding tokens provide the positional and semantic information to the language model. Meaning is learned by the embedding and Transformer layers, not encoded into the tokenizer.</p>
<p><strong>Design decision:</strong> keep the frequency-based approach, and leave context and position to the model. Whitespace-aware tokens already give some boundary information for free: " est" (with a leading space, so the start of a word) and "est" can become different tokens, because they are different character sequences. So I intentionally didn't add morphological rules to the tokenizer. The tokenizer's job is to produce useful, compact subword units; meaning is learned later by the model.</p>
<p><strong>Don't make the tokenizer responsible for information that the language model is designed to learn.</strong></p>
<h3>Problem: the encoder didn't use the pre-tokenizer</h3>
<p>During training, the tokenizer pre-tokenized the raw text into independent units before learning the merge rules. But my encode function skipped this step and just used <code>list(text)</code>. So the tokenizer learned merge rules on one representation of the text and applied them to a different one, which meant merges could be attempted across boundaries never seen in training.</p>
<p><strong>Solution: use the same pre-tokenization pipeline for encoding.</strong> The encoder now calls <code>pre_tokenize()</code> first. Each unit is split into characters, the merges are applied inside that unit, and the results are joined into the final token stream. Training and inference now follow exactly the same steps:</p>
<p><strong>Raw text → Pre-tokenization → BPE merges → Special tokens → Token IDs</strong></p>
<p>This makes tokenization deterministic and guarantees that encoding respects exactly the merge boundaries learned during training.</p>
<h3>Problem: no newline token</h3>
<p>The pre-tokenizer gave us newline tokens (<code>\n</code>), but they were discarded while building the training sequences. So the vocabulary had no newline token and no ID for it, and any new text with a newline got <code>&lt;UNK&gt;</code>.</p>
<p><strong>Solution:</strong> include the newline (<code>\n</code>) in the vocabulary.</p>
<h2>Making training fast</h2>
<p>The tokenizer worked, but training was slow, and on a real corpus that matters a lot.</p>
<h3>Problem: recalculating pair frequencies</h3>
<p><code>count_pairs()</code> was being called again after every single merge:</p>
<pre><code>count pairs → find most frequent pair → merge pair → count pairs again → find most frequent pair → merge pair → ...
</code></pre>
<p>That meant the entire <code>word_char_list</code> was rescanned to recalculate every pair frequency, even though each merge only changes a small part of the corpus.</p>
<p><strong>Solution: incremental pair count updates.</strong> Now <code>count_pairs()</code> runs only once, at the start, to build the initial <code>pair_counts</code>. I also keep a <code>pair_locations</code> map that records, for every pair, which words contain it. After a pair is selected for merging, only the words containing that pair are processed. For each affected word:</p>
<ol>
<li>Count its current pairs.</li>
<li>Remove those old pair counts from the global <code>pair_counts</code>.</li>
<li>Merge the selected pair.</li>
<li>Count the pairs in the newly merged word.</li>
<li>Add the new pair counts back to <code>pair_counts</code>.</li>
</ol>
<p>Words that don't contain the selected pair are left untouched. So training becomes:</p>
<pre><code>Initial corpus
   ↓
count_pairs()  ← done once
   ↓
find most frequent pair
   ↓
merge the pair in affected words only
   ↓
update pair_counts
   ↓
find next most frequent pair
   ↓
...
</code></pre>
<p><strong>Verification:</strong> I compared the incrementally maintained <code>pair_counts</code> against a fresh <code>count_pairs()</code> after a merge: <code>incremental counts == fresh counts → True</code>. So the optimized version gives exactly the same frequencies as recounting everything, without the full-corpus rescans.</p>
<h3>Problem: repeated maximum search</h3>
<p>We were also searching the whole list of pairs for the most frequent one again and again, after every merge. That's slow, because it scans everything every time.</p>
<p><strong>Solution: <code>heapq</code>.</strong> We count the pair frequencies once and put the pairs and their frequencies into a heap. The important thing to understand here is that in <code>heapq</code>, the smallest value comes first. So to get the most frequent pair at the top, we store each frequency as a negative number. That way the most frequent pair comes out first.</p>
<h3>Problem: duplicate (stale) entries</h3>
<p>But there's another issue. Suppose the pair (i, n) has count 17, so the heap holds (−17, (i, n)). Then a merge somewhere else changes the count of (i, n) to 5, and we push the new entry (−5, (i, n)). Now the heap has two entries for the same pair, and the old one is outdated.</p>
<p><strong>Solution: keep only the latest count.</strong> Instead of searching the heap to delete the old entry, we leave it there. When an entry comes out of the heap, we check whether its count matches the current count in <code>pair_counts</code>. If it matches, the entry is correct; if not, it's stale, and we skip it.</p>
<pre><code class="language-python">def get_current_best_pair(self):
    while self.heap:
        neg_count, pair = heapq.heappop(self.heap)
        count = -neg_count
        if self.pair_counts.get(pair, 0) == count:
            return pair, count
    return None, None
</code></pre>
<p>And after every merge, only the pairs whose counts changed get pushed back:</p>
<pre><code class="language-python">changed_pairs = self.merge_pairs(most_common_pair)
for pair in changed_pairs:
    if pair in self.pair_counts:
        heapq.heappush(self.heap, (-self.pair_counts[pair], pair))
</code></pre>
<p>I put all of this into one <code>BPE</code> class with <code>pre_tokenize</code>, <code>train</code>, <code>encode</code>, <code>decode</code> and padding, training until a target vocabulary size instead of a fixed number of steps. (The full code is on GitHub.)</p>
<h2>Where plain BPE breaks: Unicode</h2>
<p>With the complete class done, I ran Unicode tests, and this is where it fell apart:</p>
<pre><code>'café'          → 'caf '
'naïve'         → 'na ve'
'Zürich'        → ' rich'
'北京'           → ' '
'日本語'         → ' '
'мир'           → ' '
'Ελλάδα'        → ' '
'🚀'             → ' '
'hello 👋 world'  → 'hello world'
'café ☕ résumé'  → 'caf r sum '
</code></pre>
<p>(My decoder prints <code>&lt;UNK&gt;</code> as a blank, which is why the outputs have gaps.) Every one of these failed. The accented letters, Chinese, Japanese, Russian, Greek and emoji all became <code>&lt;UNK&gt;</code>, because my base vocabulary only contained the characters that appeared in my training text. Any character the tokenizer never saw is lost, and Unicode has over 150,000 characters. No training corpus is going to contain all of them.</p>
<p>But then I looked at what these characters actually are underneath:</p>
<pre><code>'a'  → UTF-8 bytes [97]
'é'  → UTF-8 bytes [195, 169]
'€'  → UTF-8 bytes [226, 130, 172]
'🚀' → UTF-8 bytes [240, 159, 154, 128]
</code></pre>
<p>Every character, in every language, plus every emoji, is just a sequence of bytes, and there are only <strong>256 possible bytes</strong>. So if the base vocabulary were the 256 bytes instead of the characters seen in training, nothing could ever be unknown again.</p>
<p>That's the idea behind <strong>byte-level BPE</strong>, the tokenizer I actually trained my models with, and that's the next post.</p>
<hr />
<p><strong>Previous:</strong> <a href="https://yashksingh.hashnode.dev/transformer-from-scratch-part-1-tokenization">Part 1: Tokenization Basics</a><br /><strong>Next:</strong> Part 3: Byte-level BPE (coming soon)</p>
<p><em>This post is part 2 of the series "Building a Transformer from Scratch". Code: <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch">GitHub</a> · Results: <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch/blob/main/docs/technical_report.pdf">technical report (PDF)</a></em></p>
]]></content:encoded></item><item><title><![CDATA[Tokenization Basics: What the Model Actually Reads]]></title><description><![CDATA[What tokens are, how vocabulary and special tokens work, and why the tokenizer decides how much text fits in a model's context and what it costs.]]></description><link>https://yashksingh.hashnode.dev/transformer-from-scratch-part-1-tokenization</link><guid isPermaLink="true">https://yashksingh.hashnode.dev/transformer-from-scratch-part-1-tokenization</guid><category><![CDATA[Machine Learning]]></category><category><![CDATA[nlp]]></category><category><![CDATA[Python]]></category><category><![CDATA[AI]]></category><category><![CDATA[llm]]></category><dc:creator><![CDATA[Yash Singh]]></dc:creator><pubDate>Wed, 11 Nov 2026 13:05:51 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af20f149c80d9d1b498be7f/dff38a1a-db9f-417a-b713-888a3de2e559.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>This is the first post in a series about building a GPT-style language model completely from scratch, from the tokenizer to training two models (110M and 336M parameters) on two GPUs and evaluating them against GPT-2. Each post covers one stage, written from the notes I made while learning it, mistakes included. The code is on</em> <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch"><em>GitHub</em></a><em>, and the full results are in the</em> <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch/blob/main/docs/technical_report.pdf"><em>technical report</em></a><em>.</em></p>
<p>Before building anything, the first thing I had to understand was how text even gets into a model. A transformer never sees text. It only sees numbers.</p>
<h2>The basics</h2>
<h3>Tokens</h3>
<p><strong>A token is a piece of text that the model treats as one unit.</strong></p>
<p>A token is not necessarily a word, a linguistic unit or anything to do with grammar.</p>
<p>For example, "I Love Transformers" can be tokenized to <code>["I", "Love", "Transformers"]</code> or to <code>["I", "Love", "Transform", "ers"]</code>, depending upon the model's tokenizer.</p>
<p>Then these tokens are converted into numbers, such as:</p>
<ul>
<li>I → 40</li>
<li>Love → 1832</li>
<li>Transformers → 4034</li>
</ul>
<p>These numbers are called <strong>token IDs</strong>. So the model never sees "I Love Transformers", it sees <code>[40, 1832, 4034]</code>.</p>
<p>So the basic pipeline is:</p>
<pre><code class="language-plaintext">Text
 ↓
Tokenizer
 ↓
Tokens
 ↓
Token IDs
 ↓
Embedding
 ↓
Vectors
 ↓
Transformer
</code></pre>
<h3>Character vs Word vs Subword tokenization</h3>
<p><strong>1. Character tokenization</strong></p>
<p>Each character becomes a token. So "Hello" becomes <code>["H", "e", "l", "l", "o"]</code>, and "Hello World" becomes <code>["H", "e", "l", "l", "o", " ", "W", "o", "r", "l", "d"]</code>: 11 tokens, because the space counts too.</p>
<ul>
<li><strong>Advantage:</strong> small vocabulary. Since each character is a token, the vocabulary is just A–Z, digits, punctuation, symbols and special characters.</li>
<li><strong>Disadvantages:</strong> long sequence length. As each character is a token, even 2–3 lines of text become a lot of tokens, and attention becomes expensive.</li>
</ul>
<p><strong>2. Word tokenization</strong></p>
<p>Here each word becomes a token. So "Hello World" becomes <code>["Hello", "World"]</code>.</p>
<ul>
<li><strong>Advantages:</strong> simple, and a short sequence length (only 2 tokens for "Hello World", whereas character tokens gave 11).</li>
<li><strong>Disadvantages:</strong> vocabulary size. It needs a very large vocabulary, because transformer, transformers, transforms, transformed and so on are all separate entries. And words the tokenizer has never seen become <code>&lt;UNK&gt;</code> (unknown token). This is bad, as it means that word doesn't contribute any information.</li>
</ul>
<p><strong>3. Subword tokenization</strong></p>
<p>Instead of treating each word as a token, we break the words a bit. For example: cat, cats, catlike, catfish, catastrophe.</p>
<p>For a word tokenizer, all these words are separate entries, and if any word is not in its vocabulary, it becomes <code>&lt;UNK&gt;</code> and the information is thrown away. But with a subword tokenizer we find a compromise: we can reuse certain tokens, such as "cat" in the example above. (The tokenizer picks pieces by how often they appear, not by meaning, so it doesn't know catastrophe has nothing to do with cats, and it doesn't need to.)</p>
<h3>Vocabulary</h3>
<p>Vocabulary is a collection of tokens that a tokenizer knows.</p>
<p>For example, a tiny tokenizer:</p>
<pre><code class="language-plaintext">0 → &lt;PAD&gt;
1 → &lt;UNK&gt;
2 → &lt;BOS&gt;
3 → &lt;EOS&gt;
4 → I
5 → love
6 → learn
7 → ing
8 → transformer
9 → s
10 → !
</code></pre>
<p>So the vocabulary size here is 11.</p>
<h3>Token IDs</h3>
<p>The tokenizer needs to convert tokens into integers because neural networks operate on numerical data.</p>
<p>Suppose the vocabulary has "I" → 17, "love" → 928, "transform" → 4521 and "ers" → 812. Then "I love transformers" might become <code>[17, 928, 4521, 812]</code>. These integers are the token IDs.</p>
<p>These token IDs don't contain any semantic information. The model's embedding layer is what converts these token IDs into vectors.</p>
<h3>Special tokens</h3>
<p><strong>1.</strong> <code>&lt;PAD&gt;</code> <strong>(padding)</strong></p>
<p>Suppose we have "I love AI" and "I love machine learning". Their lengths are different, but neural networks often process batches where tensors need consistent dimensions. So we can make them:</p>
<ul>
<li><code>["I", "love", "AI", "&lt;PAD&gt;", "&lt;PAD&gt;"]</code></li>
<li><code>["I", "love", "machine", "learning", "&lt;PAD&gt;"]</code></li>
</ul>
<p>Now both have length 5. The model is generally told: ignore these padding positions. That's where <strong>attention masks</strong> become important.</p>
<p><strong>2.</strong> <code>&lt;UNK&gt;</code> <strong>(unknown)</strong></p>
<p>Suppose the vocabulary doesn't know a particular piece. It might produce <code>["I", "&lt;UNK&gt;", "this"]</code>.</p>
<p>Modern subword and byte-level schemes greatly reduce the need for <code>&lt;UNK&gt;</code>, because almost any text can be decomposed into smaller pieces or bytes. This is one of the reasons modern tokenizers are much more robust than word tokenizers.</p>
<p><strong>3.</strong> <code>&lt;BOS&gt;</code> <strong>(beginning of sequence)</strong></p>
<p>It tells the model: this is where the sequence begins. For example, "<code>&lt;BOS&gt;</code> I love AI" → <code>[2, 17, 928, 431]</code>. Not every modern LLM necessarily uses a BOS token in exactly the same way.</p>
<p><strong>4.</strong> <code>&lt;EOS&gt;</code> <strong>(end of sequence)</strong></p>
<p>It tells the model: the sequence has ended. For example, "I love AI <code>&lt;EOS&gt;</code>". During generation, the model can eventually produce <code>&lt;EOS&gt;</code>, and the generation process can stop. Again, different models use EOS and stopping conventions differently.</p>
<p>In my own tokenizer these became <code>[PAD]</code>, <code>[BOS]</code>, <code>[EOS]</code> and <code>[UNK]</code>, and my training ended up never needing padding at all (more on that in the training post).</p>
<h3>Sequence length</h3>
<p>Suppose "I love machine learning" becomes <code>["I", " love", " machine", " learning"]</code>. That's a sequence length of 4.</p>
<p>If the tokenization instead produces <code>["I", " love", " machine", " learn", "ing"]</code>, then the sequence length is 5.</p>
<p><strong>Same text. Different tokenizer → different sequence length.</strong> This is extremely important.</p>
<h3>Context length</h3>
<p>Context length is the maximum amount of tokenized context the model can consider at once.</p>
<p>Suppose a model has a context length of 8,192 tokens. That means the model can work with roughly 8,192 tokens within its context window. Not 8,192 words, and not 8,192 characters: <strong>tokens.</strong> So if your tokenizer is inefficient for a particular language, you can fit less actual text into the same context window.</p>
<p>My own models have a context length of 1,024 tokens, which is roughly 4,700 characters of English text.</p>
<p><strong>Important:</strong> imagine two tokenizers processing the same document.</p>
<ul>
<li>Tokenizer A: 1,000 words → 1,300 tokens</li>
<li>Tokenizer B: 1,000 words → 2,000 tokens</li>
</ul>
<p>Suppose the model has a context of 2,000 tokens. Tokenizer A can fit the entire document. Tokenizer B barely fits it.</p>
<p>That's why tokenization isn't just preprocessing. It directly affects the model's usable context.</p>
<h3>Tokenization affects efficiency</h3>
<p>More tokens = more computation.</p>
<p>In transformers, the self-attention mechanism is computationally expensive because it compares tokens with one another. It has a complexity of O(n²). So if one tokenizer turns a text into 1,000 tokens and another into 2,000 tokens:</p>
<ul>
<li>1,000 tokens = 1,000,000 pairwise comparisons</li>
<li>2,000 tokens = 4,000,000 pairwise comparisons</li>
</ul>
<p>That's 4x the pairwise comparisons. (The other parts of the model, like the feed-forward layers, grow linearly with length, so the total cost doesn't quite become 4x, but attention is the part that grows fastest.) So a tokenizer that produces unnecessarily long sequences increases the computation cost.</p>
<p>Longer sequences also mean more memory, for the embeddings, hidden states, KV cache and attention-related tensors.</p>
<h3>Why different models tokenize text differently</h3>
<p>Suppose you give "Transformers are amazing!" to three different models.</p>
<ul>
<li>Model A might produce <code>["Transform", "ers", " are", " amazing", "!"]</code></li>
<li>Model B might produce <code>["Transformers", " are", " amazing", "!"]</code></li>
<li>Model C might produce <code>["Trans", "form", "ers", " are", " amaz", "ing", "!"]</code></li>
</ul>
<p>All are valid. There is no universal tokenization. <strong>The tokenizer is part of the model's system.</strong></p>
<h3>Tokenization → Embedding</h3>
<p>Suppose "I love AI" becomes <code>[15, 827, 492]</code>. These IDs go into an embedding matrix:</p>
<p>$$E \in \mathbb{R}^{V \times d}$$</p>
<p>where V = vocabulary size and d = embedding dimension. Token ID 827 means: take row 827.</p>
<h3>Vocabulary size also affects model parameters</h3>
<p>Suppose V = 50,000 and d = 768. Then the embedding matrix has 50,000 × 768 = 38,400,000 parameters.</p>
<p>If the vocabulary size increases to, let's say, 100,000, then it's 100,000 × 768 = 76,800,000 parameters.</p>
<p>So, large vocabulary = large embedding matrix. So we can't keep on increasing the vocabulary size just to reduce the sequence length. There is a trade-off.</p>
<p>In my models, the 32,768-token vocabulary is about 25 million parameters at d = 768 (the 110M model) and about 34 million at d = 1,024 (the 336M model), shared between the embedding and the output layer.</p>
<p>That's why subword tokenization wins:</p>
<ul>
<li><strong>Reasonable vocabulary</strong> → reusable pieces</li>
<li><strong>Reasonable sequence length</strong> → rare words are still representable</li>
</ul>
<p>But how do we decide which pieces to use? That's where BPE comes in, in the next post.</p>
<hr />
<p><strong>Next:</strong> <a href="https://yashksingh.hashnode.dev/transformer-from-scratch-part-2-bpe">Part 2: Building BPE from Scratch</a></p>
<p><em>This post is part 1 of the series "Building a Transformer from Scratch". Code:</em> <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch"><em>GitHub</em></a> <em>· Results:</em> <a href="https://github.com/Kevhand/decoder-only-transformer-from-scratch/blob/main/docs/technical_report.pdf"><em>technical report (PDF)</em></a></p>
]]></content:encoded></item></channel></rss>