CommonLingua

CommonLingua is a 2.35 million-parameters language identification model trained on 2,482,568 paragraphs from Structured Wikipedia and Common Corpus trained by Pleias in partnership with the GSMA's "AI Language Models in Africa, by Africa, for Africa" initiative.

Active

Since 2026

France

CommonLingua is based on a byte-level hybrid architecture combining three conv1D layers with an attention layer. It was originally designed for large scale classification of pretraining data and intently trained on diverse data sources, especially realistic documents with OCR errors as well as a particular focus on the long tail — 61 African languages are supported, including languages with almost no coverage. Since CommonLingua is trained exclusively on open data under free license, we release the extent original dataset with detailed licensing contribution. As of 2026, CommonLingua is the best performing model on the CommonLID benchmark with significant gains over the previous baseline.