A chat template inserts special tokens such as <|User|> between segments of ordinary text. If the encoder recognises those strings wherever they appear, a user can type one and inject a control token. The safe encoder maps specials by position in the template and encodes every text segment as plain text; because the segment boundaries are known, the segments can also be encoded independently and in parallel. Implement both encoders and show the difference.
Toy tokenizer. encode_text(text, merges): scan left to right; if the two characters at the current position form a key of merges, emit merges[pair] and advance by two, else emit ord(char) and advance by one.
naive_encode(prompt, special, merges)
special maps strings to ids. At each position, if any special string starts there (longest first), emit its id and skip it; otherwise collect ordinary characters until the next special occurrence and encode that run with encode_text. This is the unsafe encoder.safe_encode(segments, special, merges)
segments is a list of ("special", name) or ("text", string). Specials are looked up by name; text is encoded with encode_text. Return {"ids": [...], "per_segment": [[...], ...]} with the per-segment encodings in order (the parallel-encoding unit).contains_control(ids, special)
ids.<|User|> becomes the ids of its characters.naive_encode and safe_encode agree only when no segment's text contains a special string.