Chapter 8 · The Rise of Large Language Models

See What the Model Sees

Type anything and watch the real GPT-2 and GPT-4 tokenizers split it into tokens and IDs: common words, rare words, numbers, code and other scripts.