LLM Benchmarks
September 2023

At the strategic level, the GPT/LLM landscape is improving month by month. We expect to be able to move some projects from cloud services to local models starting in 2024.

Notable improvements include:

• Meta's release of Llama 2, which is spurring another round of innovation thanks to its generous license.
• The releases of Code Llama and the language-agnostic model "Sonar." These are not included in this benchmark, but they complement generic LLMs. They work particularly well with enterprise-specific assistant systems for particular domains.

Surprising results when benchmarking Llama2 and Nous Hermes 70B

When benchmarking promising new models, we were surprised to find that Llama2 itself is not particularly well suited for building LLM-powered products. It is too chatty. However, there are open fine-tunes of Llama2 that perform particularly well, especially Nous Hermes 70B. This is the first open model to beat ChatGPT 3.5 in our benchmarks.

Cost and quality: the 70B model compared

This 70B model can be somewhat expensive to run. Running a workload equivalent to this benchmark costs about ~5.75 EUR on a GPU machine rented from a major cloud provider. However, this is only the beginning of a larger trend toward improving quality at affordable cost. The 13B fine-tunes of Llama 2 are catching up, while the new Falcon 180B model is putting pressure on the top spot.

Beyond that, less powerful models are still worth a look, especially as we discover and apply new forms of Guidance and domain-specific beam search.