LLM Benchmarks
December 2023

Multilingual TIMETOACT LLM Benchmarks

Have you heard that ChatGPT works better with English and may be less effective with other languages?

This is well known among users. It's not just ChatGPT that's affected, but other large language models as well. The main reason is that the training dataset contains far more English than other languages.

The consequence is that ChatGPT can suddenly become less capable when it has to handle text in other languages. Results become less accurate, and the hallucination rate increases.

This could come as an unpleasant surprise for international companies looking to build AI capabilities for multiple languages.

So the key question is: how well can ChatGPT speak my company's languages?

To help our clients better navigate the world of multilingual AI, TIMETOACT has created the Multilingual Benchmark for large language models.

This first release compares the performance of LLM models across 16 different languages, ranging from English and German to Ukrainian and Swedish.

This benchmark was created with the help of linguistics experts and our TIMETOACT colleagues (who live in different countries). Each language was contributed by a native speaker.

The TIMETOACT Multilingual LLM Benchmark doesn't go as deep into model capabilities as our product benchmark. It focuses on fundamental linguistic skills:

• Understanding and executing instructions in a given language.
• Understanding words in a language, reasoning about them, and deriving meaning.

This is already enough to compare the performance of different models within a language, or to find out which languages are well supported by a model of your choice.

An accuracy of 100 means the model was able to execute all instructions. An accuracy of 50 means it got half the tasks wrong.

The abbreviations in the table stand for:

BAK: Bashkir, CHV: Chuvash, DEU: German, ENG: English, FRA: French, HUN: Hungarian, KPV: Komi-Zyrian, KRC: Karachay-Balkar, KUM: Kumyk, MHR: Eastern Mari, POL: Polish, POR: Portuguese, RUS: Russian, SAH: Yakut, SWE: Swedish, TAT: Tatar, UDM: Udmurt, UKR: Ukrainian

My language isn't supported by LLMs. What now?

If some languages aren't well supported by the generative AI model of your choice, there are still several options for using AI effectively.

As part of TIMETOACT's research and development, we work with low-resource languages (languages spoken by 0.01% of the world's population or even less). Some of that expertise translates into insights for AI-driven business systems as well:

1. Use automated translation to convert requests into English and then translate the results back. It's easier to train an excellent translation model than to build a good foundation model.
2. Submit both the translated text and the original text to LLMs together with the request.
3. Non-English languages can use more tokens per request. This leads to smaller effective context windows and more expensive requests. So keep an eye on token usage in your requests.
4. Make sure to prepend the request with "The language [name the desired language] is your native language" to ensure the language model switches to the right context from the outset.
5. Use Retrieval-Augmented Generation (RAG) to include relevant domain-specific snippets of information in AI requests. Use language-specific embeddings for this. Enrich the context with GPT-generated hints, even if you perform the final inference on local models.
6. If you're building a voice-based interaction, try to use multi-modal models (e.g., mapping audio requests directly to English via the shared vector space). This reduces the number of semantic "hops" that affect accuracy.

How to write a bad multilingual benchmark

A good benchmark requires effort, collaboration with linguists, and GPU time. Despite all that investment, we found it trivial to write a benchmark that even ChatGPT 4 would fail.

Here are two simple questions in English: the first answer is clearly wrong because there are three rhyming words: "mouse," "house," and "spouse." The second answer is wrong because the stress falls on the third syllable, not the second.

Does that mean ChatGPT 4 is bad at English? No, it just means this is a poor benchmark for assessing linguistic ability. Rhyming or syllable-stress tasks are intuitive for people who learn through listening and speaking. But text-based large language models (like ChatGPT or Llama) only learn through reading and writing.

In addition, this kind of benchmark isn't useful because it prevents meaningful comparisons of models across different languages. After all, rhyming works differently in different languages. Tonal languages, such as Chinese, can incorporate tonal patterns (changes in pitch while speaking) into rhyme schemes. Each language's particular features shape how rhyme and rhythm are constructed within its own linguistic structures.

In creating the Multilingual TIMETOACT LLM Benchmark, we tried to take these and many other linguistic subtleties into account.

New models

We've added new models to the benchmark:

• Starling 7B-alpha - a further fine-tune built on OpenChat 3.5
• Mistral 7B Notus-v1 - another promising Mistral fine-tune (LoRA)
• Microsoft Orca 2 13B and 7B - a new foundation model from Microsoft.

These models are included in both the classic TIMETOACT LLM Benchmark and the multilingual benchmark.

Stats

Did you know that producing these two tables required 12,150 chat completions (prompts)? Or even more, if you count the optimization from beam search, which is now enabled for all available models.