Building Tokenizers — From Words to DNA

Every language model has one. Every DNA model has one. It sits at the very front of the pipeline, before the embeddings, before the attention, before any of the parts people write papers about — and almost nobody talks about it. It is the tokenizer, and its job is deceptively small: turn a continuous string of symbols into a list of integers a neural network can actually consume.

This is the first post in a series where I build tokenizers from the ground up. I want to start where a beginner can follow every line and end somewhere genuinely useful — a tokenizer for DNA. Along the way I will point at the two codebases this post draws from: a small LLM and tokenizer toolkit for text, and a short DNA tokenizer notebook that carries the same ideas over to the genome. The goal is not to be exhaustive. It is to make the stack of ideas visible.

---

What a Tokenizer Actually Has to Do

Strip away the vocabulary and the acronyms and a tokenizer has exactly four responsibilities. Understanding them makes every tokenizer — from the simplest word splitter to whatever GPT uses today — look like a variation on one theme.

That is the whole architecture. Everything else is a choice about how to segment, and every choice trades off three quantities that pull against each other: vocabulary size, sequence length, and how gracefully you deal with symbols you have never seen. Keep those three in mind and the rest of this post is just watching the trade-off get renegotiated.

---

The Basic Tokenizer

The simplest tokenizers do nothing clever. They split on a rule. In the text toolkit, basic.py implements three of them, and comparing them shows the trade-off in its rawest form.

basic.py — three ways to split

def character_tokenize(text):
    return list(text)                    # "cat" -> ['c', 'a', 't']

def space_tokenize(text):
    return text.split(" ")               # "the cat" -> ['the', 'cat']

def word_tokenize(text):
    return re.findall(r"\b\w+\b", text)   # drops punctuation, keeps words

Character-level gives you a tiny vocabulary — a few dozen symbols cover most English — and it never meets a token it cannot represent, because any text is just a sequence of characters it already knows. The price is sequence length: a paragraph becomes hundreds of tokens, and the model has to work hard to learn that c-a-t means something.

Word-level is the opposite bargain. Sequences are short and each token is meaningful, but the vocabulary explodes — English has hundreds of thousands of words — and the moment a user types a word you did not train on, or a typo, or a name, you are stuck emitting <UNK> and throwing information away. This is the out-of-vocabulary problem, and it is the reason nobody trains serious models on pure word tokenization anymore.

Character tokenization never runs out of vocabulary but says almost nothing per token. Word tokenization says a lot per token but constantly runs out of vocabulary. The interesting tokenizers live in between.

---

Adding Complexity: Byte Pair Encoding

Byte Pair Encoding (BPE) is the compromise that won. It starts from characters — so it can never be fully stumped by a new word — and then learns which character sequences are common enough to deserve their own token. Frequent words end up as single tokens; rare words fall back to a handful of subword pieces; genuinely novel input still decomposes into characters. You get short sequences for common text and graceful degradation for everything else.

The algorithm itself is almost anticlimactically simple, which is part of its appeal. It was originally a data compression scheme (Gage, 1994) and was repurposed for neural machine translation by Sennrich, Haddow, and Birch in 2016. Here is the core loop from bpe.py, lightly trimmed:

bpe.py — the merge loop

def learn_bpe(dataset, num_merges):
    corpus, vocab = bpe_initialize(dataset)   # each word -> [chars] + '</w>'
    merges = []
    for _ in range(num_merges):
        pairs = get_pair_frequencies(corpus)  # count adjacent token pairs
        if not pairs:
            break
        best, freq = pairs.most_common(1)[0]  # the most frequent pair
        merges.append(best)
        vocab.add(best[0] + best[1])          # the merged pair is a new token
        corpus = [merge_pair_in_word(w, best) for w in corpus]
    return merges, vocab, corpus

Read it as a recipe. Start with every word broken into characters, with a special </w> marking the end of a word so boundaries can be rebuilt later. Count every adjacent pair of tokens across the whole corpus. Merge the single most frequent pair into one new token. Repeat. After a few thousand merges, the most common substrings of your language — th, ing, tion, whole frequent words — have become tokens, and the vocabulary has settled at whatever size you asked for via num_merges.

Encoding new text then just replays the learned merges in order, and the toolkit wraps all of this in a BPEWordTokenizer class that adds the <PAD> and <UNK> tokens, does the integer mapping, and saves to JSON so a trained tokenizer can be reused across experiments — which matters more than it sounds, because a model and its tokenizer must always be the same pair.

reusing a trained tokenizer

from small_llm.tokenization import BPEWordTokenizer

tok = BPEWordTokenizer(["the quick brown fox", "the lazy dog"], num_merges=50)
ids = tok.encode("the fox")      # -> [12, 7, 33, ...]
tok.decode(ids)                  # -> "the fox"
tok.save("my_bpe.json")          # reuse across experiments

BPE is not the only subword scheme — WordPiece (used by BERT) and the Unigram model inside SentencePiece (Kudo and Richardson, 2018) make different statistical choices — but they all share the same insight: let the data decide the vocabulary instead of a rule. That is the conceptual jump from the basic tokenizers, and it is the one worth internalizing before going further.

---

The Python You Actually Reach For

One reason tokenization is a good place to start learning sequence processing is that the core of it needs almost no dependencies. The two basic tokenizers above use only the standard library. It is worth knowing which tools do what, roughly in order of how often they come up.

The point is not to reach for a framework early. It is that you can build a working, correct tokenizer with re and Counter, understand every line, and only then swap in the fast implementations once the concept is solid.

---

From Text to DNA

Here is where it gets interesting, and where my own work lives. DNA is, for a tokenizer's purposes, just a very long string in a four-letter alphabet: A, C, G, T. Everything from the first half of this post applies — but the trade-offs land in a different place, and that changes the right answer.

For English, character-level tokenization was a poor default because sequences got too long and individual characters meant too little. For DNA, character-level is often the natural choice. The alphabet is four symbols instead of forty thousand words, so vocabulary size is a non-issue. And unlike English characters, a single base genuinely is a meaningful unit — mutations, the thing much of genomics cares about, happen one base at a time. The short DNA tokenizer notebook takes exactly this route:

DNA tokenizer — vocabulary built from the data

chars  = sorted(set(train_source))            # ['A', 'C', 'G', 'T']
stoi   = {ch: i for i, ch in enumerate(chars)}   # string -> integer
itos   = {i: ch for ch, i in stoi.items()}       # integer -> string

encode = lambda s:  [stoi[c] for c in s]      # "ATG" -> [0, 3, 2]
decode = lambda ids: "".join(itos[i] for i in ids)

Notice that the vocabulary is built from the data itself rather than hard-coded — the same discipline BPE uses, just without any merges, because with four symbols there is nothing to compress. This is the same segment-map-reverse skeleton from the top of the post, at its most minimal. The notebook then does one DNA-specific thing worth calling out: because the training objective is to predict the next base, it feeds the model fixed-length windows of sequence and sets each target to be the input shifted by one position. The tokenizer produces the integers; that shift produces the supervision.

The notebook is deliberately end-to-end, so the tokenizer is shown in the context it actually serves. Here is what it walks through:

Character-level is not the only option for DNA, and this is where the series will go next. The more advanced move is k-mer tokenization: instead of one base per token, group every k consecutive bases into a token (so ATGGC with k=3 becomes ATG, TGG, GGC). This is what DNABERT (Ji et al., 2021) does, and it is a direct analogue of subword tokenization — a way to pack more context into each token at the cost of a larger vocabulary and a reintroduced out-of-vocabulary question. Newer genomic models have pushed in both directions: the Nucleotide Transformer leans on k-mers, while HyenaDNA and Evo go back to single-nucleotide resolution and solve the long-sequence problem in the model architecture instead of the tokenizer.

The tokenizer choice is not a preprocessing detail you get to ignore. In genomics it decides whether your model can even see a single-base mutation — which is often the entire point.

---

Where This Series Is Going

This post stayed deliberately at the foundation: the four jobs every tokenizer does, the basic splitters, BPE as the learned compromise, and the character-level DNA tokenizer as the point where the same ideas cross into biology. The thread running through all of it is a single trade-off — vocabulary size against sequence length against handling the unseen — renegotiated for each domain.

Most of what follows I want to explore out of plain intellectual curiosity rather than to serve any single application. Tokenization is where the raw material of a problem meets the machinery that learns from it, and I find that boundary genuinely interesting — especially in biology, where the choice of unit is also a choice about what the model is allowed to notice. These are the threads I want to pull next:

The frontier I am most drawn to is tokenization built specifically for biological sequence, because the field is visibly unsettled about the right unit. Early genomic models used fixed k-mers, but overlapping k-mers leak information across token boundaries and bloat the vocabulary, so DNABERT-2 (Zhou et al., 2023) dropped them for Byte Pair Encoding run directly on nucleotides — the same algorithm from the middle of this post, now compressing genomes instead of English. Others went the opposite way: HyenaDNA and Evo abandon subwords entirely for single-nucleotide resolution and push the burden onto the model architecture, while protein models like ESM-2 mostly tokenize one amino acid at a time. There is no settled answer, which is exactly why it is worth my time. Whether a learned, biology-aware vocabulary can beat both fixed k-mers and raw single-base tokens is the question I want to test next.

The code for both halves lives in the open: the text tokenizers and BPE in the Small_LLM_and_tokenizer project, and the DNA tokenizer in the companion DNA_LargeLanguageModel tutorial. If you want to actually understand a tokenizer, the fastest path is to run the basic one, break it, and watch what the model does when you do.

References

← All writing