A study from the Coalition for Language Equity in AI Research finds that AI models need more tokens to process certain languages, driving higher costs for users of underrepresented languages.

The disparity hits languages with large speaker bases that have less web presence in training data, such as Punjabi, Bengali, Urdu, Burmese, and Amharic, not only small-language communities.

This hidden tokenizer tax arises because tokenization is more efficient for languages with abundant digitized data, causing underrepresented languages to use 1.5 to 5 times more tokens than English for the same prompt, and up to 17 times more across about 200 languages in one model.

Since most providers charge by tokens, higher token counts mean higher costs, longer response times, and smaller context windows for these languages.

Languages with writing systems less represented in AI training data—such as Burmese, Tibetan, Khmer, and Lao—are especially affected, while Latin-script languages often enjoy an advantage.

The study frames this as a digital inequality issue, noting that tokenization costs translate into real financial and access implications for speakers of many non-English languages.