Abstract

Tokenization defines how data is represented as input and output for machine learning systems, including large language models (LLMs). While tokenization has long been a foundational component of NLP systems, it has historically remained a niche research area, often considered solved. The recent proliferation of LLMs has changed this: as these models are deployed across diverse languages and domains, tokenization-related issues have become increasingly apparent---from cross-lingual representation disparities to efficiency bottlenecks. This tutorial will provide a comprehensive overview of tokenization methods in text, starting from classical word-based and morphological approaches, to the current standard of statistical subword tokenizers, and emerging methods on end-to-end learnable tokenizers. We will also discuss the implications of tokenization choices on model performance, fairness, and efficiency, and present best practices for evaluating and designing tokenizers in multilingual and domain-specific settings.