LLM Benchmarks
February 2024

Improvements in ChatGPT-4 - new recommendations

The latest update in the ChatGPT-v4 series finally breaks the trend of releasing cheaper models with lower accuracy. In our benchmarks, GPT-4 0125 (or v4) finally beats the GPT-4 0613 (or v2) model.

This model also includes the most recent training data (up to December 2023) and runs at a fraction of the cost of the v1 and v2 models. This makes GPT-4 Turbo v4/0125-preview a new safe default model that we can recommend.

The trend for the GPT-3.5 models continues to follow the same pattern. New models are getting cheaper and less capable.

Mistral and Claude API - verbosity problem

This benchmark finally includes benchmarks for the Mistral AI and Anthropic Claude models:

• Anthropic Claude Instant v1.2 - Anthropic's smaller LLM - it's anthropic.claude-instant-v1 on AWS Bedrock.
• Anthropic Claude v2.0 and v2.1 - Larger Anthropic LLMs that introduced large context sizes - the anthropic.claude-v2 series on AWS Bedrock.
• Mistral Large Model - Recently released LLM from Mistral, positioned between GPT4 and GPT3.5 in internal benchmarks. It's mistral-large-2402 on La Plateforme.
• Mistral Medium - Another proprietary model from Mistral, roughly comparable to Llama 70B according to the Miqu leak. We're testing mistral-medium-2312.
• Mistral Small - This model used to be the very popular Mixtral 8x7B, though the second version doesn't confirm whether that's still the case. We test both versions: mistral-small-2402 and mistral-small-2312.
• Mistral Tiny - This model corresponds to Mistral 7B Instruct v0.2. Or mistral-tiny-2312 on Mistral AI.

All of these models can be good for creating content and communicating with people. But that's not the point of our benchmark. We rank models on the leaderboard based on their ability to deliver precise answers for tasks like information retrieval, document ranking, or classification.

All of these models are too wordy for that. They also don't follow instructions precisely. Even small local Mistral 7B fine-tunes are better at this. ChatGPT-4 remains at the top. It seems OpenAI understands the needs of enterprise customers better than the rest.

Enterprise AI Leaderboard

We've been tracking the performance of LLM models for many months now; this is our eighth report.

This process has helped us gain firsthand experience working with several different models simultaneously. Unlike typical academic benchmarks, we draw our data from real projects and enterprise-specific tasks.

By the way, we're no longer alone in this space. Another company has recently started working on a similar set of enterprise benchmarks. We invite you to take a look at PatronusAI's Enterprise Scenarios Leaderboard on Hugging Face.

That's all well and good, but it's time to address the real elephant in the room. The truth is:

Large language models are just an implementation detail.

Yes, it's true that a lot depends on their performance and capabilities. That's why, for example, we generally recommend GPT-4 Turbo v4/0125-preview as a short-term default model to start with.

Ultimately, though, we believe large language models are replaceable and interchangeable. In fact, the entire LLM leaderboard was started because of a recurring customer question: "When can I replace ChatGPT-4 in my projects with a local model?"

If you look at Y Combinator's "Request for Startups," one specific request focuses exactly on the topic of replacement: small fine-tuned models as an alternative to giant generic ones. Y Combinator helped incubate companies like Stripe, Dropbox, Twitch, and Cruise. They know a thing or two about market and industry trends.

Huge generic models with many parameters are very impressive. But they're also very costly and often come with latency and privacy challenges. Fortunately, smaller open-source models like Llama2 and Mistral have already shown that, when fine-tuned with the right data, they can deliver comparable results at a fraction of the cost.

Taking this idea further, we believe that local large models will be the path to improving overall system accuracy beyond ChatGPT's capabilities, while significantly reducing operating costs.

Since individual LLMs are an implementation detail, what should the metric be for measuring the state of the art when AI is applied to enterprise workloads?

Here's a hint, in the form of some questions we get asked:

• Which RAG architecture is best for legal workloads?
• Which vector database should we use to build an internal support bot?
• What's the best approach to automatically handle enterprise questionnaires with 1,000 questions in B2B sales?

The metric should target and compare complete enterprise and business AI solutions. End to end.

Building a complete enterprise AI leaderboard will take time and effort. We're starting with the fundamental capability — the ability of the AI system to find relevant information within enterprise-specific documentation. This is the basic building block of RAG systems.

Here's an example: we took a public annual report from the Christian Dior Group. We then asked the AI system 10 specific questions about that report. For example:

• What was the company's revenue in 2022?
• How much liquidity did the company have at the end of 2021?
• What was the gross margin in 2023?
• How many employees did the company have in 2022?

As you can see, each question has exactly one correct answer. No calculations or advanced reasoning are required.

How well do you think different systems would handle these specific questions?

Not very well!

To start, we tested a few common systems:

• ChatGPT-4
• OpenAI Assistant API with document retrieval and the gpt-4-0125 model
• Two popular services for asking questions about a specific PDF: ChatPDF and AskYourPDF.

Each test involved uploading the annual report and asking the question with a very specific instruction:

This instruction mattered because:

• we wanted to encourage models to use the chain-of-thought process (CoT) when it improves accuracy
• we still needed the number to be readable in a specific locale, hence the strict requirement to use a decimal comma and no thousands separators (just like in the original report).

Presumably, RAG systems, as end-to-end solutions, would already have CoT built into their pipelines behind the scenes. Even so, when we added the instruction to the overall request, overall accuracy still improved.

Below are the final scores for several RAG systems in a single test. We gave each system 1 point for a correct and readable answer, and 0.5 points for an answer that pulled out the right information but made an error in magnitude.

So far, OpenAI's RAG systems are the best on the market for this particular task. However, we expect that not to remain the case for long.

Specialized solutions are capable of achieving higher scores even without using state-of-the-art LLMs. We know this for certain, because we've built such systems. One of them even uses Mistral-7B-OpenChat-3.5 to extract information from tens of thousands of PDF documents.

As we expand this enterprise AI benchmark and enrich it with more cases and solutions, we expect ChatGPT to eventually be dethroned.