Tokenization defines how data is represented as input and output for machine learning systems, including large language models (LLMs). While tokenization has long been a foundational component of NLP systems, it has historically remained a niche research area, often considered solved. The recent proliferation of LLMs has changed this: as these models are deployed across diverse languages and domains, tokenization-related issues have become increasingly apparent---from cross-lingual representation disparities to efficiency bottlenecks. This tutorial will provide a comprehensive overview of tokenization methods in text, starting from classical word-based and morphological approaches, to the current standard of statistical subword tokenizers, and emerging methods on end-to-end learnable tokenizers. We will also discuss the implications of tokenization choices on model performance, fairness, and efficiency, and present best practices for evaluating and designing tokenizers in multilingual and domain-specific settings.
If you would like to cite the tutorial, please use the following reference (ACL Anthology).
@inproceedings{hofmann-etal-2026-tokenization,
title = "Tokenization in Large Language Models",
author = "Hofmann, Valentin and
Kallini, Julie and
Kumar, Sachin and
Libovick{\'y}, Jind{\v{r}}ich and
Limisiewicz, Tomasz and
Salesky, Elizabeth",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-tutorials.3/",
doi = "10.18653/2026.emnlp-tutorials.3",
pages = "4--5",
ISBN = "979-8-89176-463-7",
}