The model eats tokens, not the characters you counted

Count the same sentence in characters and the API's token count does not match. max_tokens cuts off the second half. The model eats integer ids. Pure Python below trains a teaching BPE that encodes and decodes back. It is not the GPT-2 vocabulary. This toy BPE is not a vendor tokenizer. Do not use it to reconcile an official bill.