The TIMETOACT LLM Benchmarks provide an up-to-date comparison of various Large Language Models to assess their suitability for use in product development.
LLM Benchmarks
March 2024
LLM Benchmarks March 2024
Anthropic Claude 3 models
Anthropic recently released the third generation of its models:
- Haiku
- Sonnet
- Opus
All models show enormous improvements over the previous versions in our product-oriented LLM benchmarks.
It seems Anthropic has finally started listening to customers who use language models to build real enterprise products.
Claude 3 Opus overtakes the GPT-4 models
Claude 3 Opus made the biggest leap forward and has caught up to the GPT-4 models.
In the "Documents" category, Opus achieved a perfect score of 100, meaning it performs very well on tasks involving reading documents, extracting and transforming information. These tasks are heavily used in our products and prototypes that leverage Domain-Driven Design and knowledge maps to work with complex business domains.
💡 This news is great for our clients, but not so great for us, since we now have to overhaul our entire benchmark suite to add even more challenging edge cases to the "Documents" category.
Claude 3 Sonnet: the mid-tier model
Claude 3 Sonnet is the mid-tier model. It has also improved over the Claude 2 models, although the jump isn't as significant at first glance.
However, the cost of running the model has dropped by more than half, which points to substantial improvements. Another important point — all models in the Claude 3 lineup offer better multilingual support (important for international companies) and a huge context window of 200K tokens. These are big improvements worth celebrating!
Claude 3 Haiku
Anthropic's Claude 3 Haiku deserves special praise. It's the smallest model, and it even managed to outperform Claude 3 Sonnet in our benchmarks.
Focus on "enterprise workloads"
Anthropic mentions "enterprise workloads" repeatedly when talking about this model. Perhaps that focus was the key to performing so well on such tasks.
The model itself doesn't perform as well in our benchmarks as it's described in their PR. It beats neither GPT-3.5 nor Gemini Pro. But that's not really the point. Given its huge 200k context window and attractive 1:5 pricing model (input tokens cost 5x less than output tokens), it could become the default model for working with large enterprise documents at low cost.
The model itself is 12x cheaper than Claude 3 Sonnet and 60x cheaper than Claude 3 Opus.
Gemini Pro 1.0 - comparable to Claude 3 Haiku
Gemini Pro 1.0 is a new mid-tier model from Google. The full product line includes Nano, Pro, and Ultra.
This model beats the first version of GPT-3.5 (released almost a year ago) and performs at the level of good Mistral 7B fine-tunes on our tasks.
The model is also quite affordable, roughly comparable to Claude 3 Haiku.
However, as is typical for Google, integrating with Gemini Pro 1.0 is a bit more difficult, especially when operating within the EU. The context size is also smaller.
If you have the choice, we recommend using GPT-3.5 (v4/0125), Claude 3 Haiku, or even a good Mistral 7B fine-tune running locally with vLLM.
The challenge of understanding tables
Here's an example benchmark from our enterprise AI benchmark suite. It's no longer just testing text manipulation, but a more complex task.
Given a PDF document with technical specifications for LEDs, the following question is asked: what is the typical peak forward voltage for the GL538?
As you can see below, the number is right there. If you upload the PDF into an AI system of your choice, the system will fail to provide a correct answer. Even ChatGPT-4 with Vision won't be able to handle this task.
Try it yourself!
You can test this yourself by downloading the PDF here and uploading it to an AI system of your choice.
The document is challenging for AI systems for two reasons:
- There is no text layer in this PDF, only the image
- The table itself has gaps and is irregular
While real-world enterprise documents can be in even worse condition (not to mention ☕️ coffee stains on incoming scans), the current setup already makes it very hard for LLM-powered systems to read the document.
What could a good AI-driven system do in this case?
- Isolate the relevant product page
- Select the relevant table on the product page
- Perform table understanding on the selected table
This makes the task achievable for GPT-4 Vision.
Outlook for upcoming benchmarks
This is one of the tasks that will be featured in upcoming enterprise AI benchmarks. Tables and spreadsheets are the backbone of business processes, and any good LLM-powered assistant needs to understand them to be useful. We'll find out which models are particularly good at this!
Let’s turn benchmarks into your competitive advantage and build a custom AI solution tailored to your business.
Discover the transformative power of leading language models and revolutionize your digital products with AI. Stay ahead of the curve, boost efficiency, and gain a clear competitive advantage. We help you take your business value to the next level.
