Languages big models miss

Tamil · 80M+ speakers

Spoken Tamil differs sharply from written Tamil, and dialects from Jaffna to Chennai barely appear online.

Sinhala · 17M+ speakers

Written and spoken Sinhala are almost two different registers, and models trained on formal text sound stiff in conversation.

Yoruba · 45M+ speakers

A tonal language where meaning rides on diacritics that most web text leaves out.

Hausa · 80M+ speakers

One of Africa's most spoken languages, yet with little web text, written in both Latin and Arabic (Ajami) script.

Amharic · 55M+ speakers

Its own script (Ge'ez) and rich word forms get split into many pieces by tokenizers built for English.

Igbo · 30M+ speakers

Tonal, with many dialects, and the little text online is often missing its tone marks.

Khmer · 17M+ speakers

Written without spaces between words, so standard tokenizers and search struggle from the first step.

Pashto · 40M+ speakers

Strong regional dialects and very little high-quality digital text.

Haitian Creole · 12M+ speakers

Spoken by nearly everyone in Haiti, while French dominates formal writing, so little text reflects how people talk.

Quechua · 8M+ speakers

A family of mostly oral varieties with little standardized text to learn from.