LLM Benchmarks April 2025

New models from OpenAI landed nicely in our reasoning leaderboard. o3-mini and o4-mini scored at the top, while less expensive 4.1 (base and mini) took 8th and 13th places. There are not that many surprises here, except for the cost of running these models.

Qwen models are a well-known “secret” of teams that need to run LLM on their premises. We heard about successful uses of Qwen 2.5 in many projects. This was represented by relatively high scores on our reasoning benchmark.

However the latest release of Qwen3 models has pushed state-of-the-art even further. Qwen3 models come in a variety of plain models and Mixture-of-Expert models. They may take longer to process problems, but their accuracy is comparable to the best cloud models. This makes them suitable for enterprise jobs that require high accuracy and can run over night.

Qwen3 release pushes state-of-the-art in local language models and creates more pressure for the cloud providers. Things don’t stop there, though.

We all remember how Google was late in the game in the past. OpenAI was much better than first Gemini models, while praised Gemini Ultra - never came out.

Google has managed to gradually shift the tides since then. While it still doesn’t beat the top OpenAI models, it consistently delivers new models within the TOP-10 range, while beating OpenAI on price and stability.

Gemini-2.5-Flash Preview in thinking mode is currently the best Google model on our benchmark. Gemini 2.5 Pro Preview scores 4th and 10th places. Gemini 2.5 Flash in non-thinking mode is at 16th place.

However, benchmarks are only an approximation of the real world. 
So let us give you two additional data points:

First of all, experienced engineers (with access to Mistral / Anthropic / ChatGPT Pro and Gemini) are starting to gradually rely more on Gemini 2.5 Pro as their tool of choice in AI Coding. They praise quality of the answers and reliable work with larger context. For example, it is common to copy essential chunk of the source code into the prompt (up to 50k tokens) and then work interactively with that chat until the overall context reaches 200k-500k.

Another example is about complex document understanding. Within the last months we’ve heard from our peers in AI Research that “if you need to handle enterprise documents reliably - use Google Gemini LLM”.

We went on to verify that. In one of the evaluations, we load a compliance PDF into a Graph database for further analysis. The tricky part is that such documents can span hundreds of pages, so they have to be loaded page by page. Another tricky part is that the documents are deeply nested, and it takes extra care to properly attribute clauses across the page breaks.

Models from Anthropic, Mistral, and even OpenAI fail to handle the task of “stitching” content across pages. They get confused easily even within a small context, misrepresenting the original text.

Google Gemini 2.5 Pro, on the other hand, handles the same task (given the same inputs) reliably. This makes it a good contender to OpenAI models when implementing document-heavy AI workflows.

As a part of the development program at TIMETOACT GROUP Austria we are running Embrace AI initiative - an experiment in training senior-level software engineers to better understand and use modern AI coding tools. This program is also integrated into our AI Research process, helping to strengthen it with insights from the wider community of experts.

As we’ve mentioned earlier, the first insight from this process is that practitioners are gradually shifting from OpenAI and Sonnet 3.7 models to Google Gemini 2.5 Pro in their everyday tasks. This affects even those developers that used to stand by Sonnet 3.5 for a long time.

The reason for this shift is the effective combination of context size (the amount of context the model can reliably handle), speed, quality, and cost offered by Gemini models.

This shift is not mainstream - it affects only experienced engineers that are not bound to a specific AI provider and are free to choose the best tools for the job.

Second insight involves tooling - chats vs complex multi-agent IDEs and environments. To put things into the context, let us provide you with an example of one exercise from our Embrace AI program:

Software engineers were allowed to use any combination of tooling to accomplish this, starting from chats to the most advanced IDEs and Coding agents. We allowed them to experiment, only asking to measure the time.

The fastest solution involved using Anthropic Claude with tools to access powershell and directory. It took 30 minutes and used quite a lot of prompts and tokens under the hood.

Other solutions took 1.5h to 2h to implement the code, they involved human control and a bit of prodding the agents.

Writing the same code from the scratch without AI would probably take a few hours, so the best solution with AI provided a noticeable productivity boost. Median solution with AI took the same time it would take an experienced developer to code this app. 

However, then we flipped the question and pushed the challenge further:

The purpose of this exercise was to demonstrate that modern AI tooling gets in the way more often than not, and that understanding the fundamentals of coding with AI can let engineers get better results much faster.

Our own solution to the challenge took 15 minutes and required 2 steps:

1. Copy-paste kata to ChatGPT o1 pro, request to ask questions clarifying the task, then capture in succinct prompt. This pattern is called Prompt Distillation in our books.

2. The resulting prompt was 432 tokens (or 1833 characters) long. It produced working solution from the first try, when copied to GPT-o1 pro, GPT 4.5 and Claude 3.7.

Another engineer used Gemini 2.5 Pro for Prompt Distillation, arriving to the similar result in 15 minutes (although his final Gemini-generated prompt was 1191 tokens or 4598 characters).

This looks pretty impressive, right? Our AI Research community took the challenge to heart and managed to push it even further.

One engineer manually wrote a prompt that is just 70 tokens (313 characters) long and worked on Gemini 2.5 Pro and Sonnet 3.7. It also worked almost out-of-the-box (required one follow-up question) on GPT-4o.

The prompt looked like this

Another engineer wrote in 7 minutes a prompt (194 tokens+298 custom instructions) that produced valid result on Claude 3.7.