LLM Benchmarks Summer 2025

We finally have a term for the Custom Chain-of-Thought (or SO CoT) approach that we’ve been using heavily in various projects.

This approach was initially extracted from the successful cases in our AI portfolio and further refined by AI R&D work in the community (including successful submissions in Enterprise RAG Challenges).

In fact, all evals from our Reasoning LLM Benchmark v2 (starting from January 2025) leverage specialised SGR schemas to drive the reasoning.

You can read more about SGR here or check out the publicly shared demo. This demo demonstrates how to use SGR to build a business assistant capable of:

• planning and reasoning while using an inexpensive non-reasoning model

• calling tools to help manage customers in a fictional company that sells AGI courses (in the demo we simulate tools to create invoices, send emails and pull customer data)

• creating additional rules and memories for itself

All that is done in 160 lines of Python code without the use of any AI frameworks or tool calling. Just OpenAI SDK and Pydantic.

This topic of creating and orchestrating agents for business tasks is something that we are quite interested in and keen to push state-of-the-art in the field further. We’ll have one more interesting announcement along these lines in this report later, but for now let’s check out which models are the best for working with business tasks given Schema-Guided Reasoning.

Let’s start with the obvious big wins. OpenAI has released a range of models recently:

• available via API: gpt-5, gpt-5-mini and gpt-5-nano (announcement)

• downloadable: gpt-oss-20b and gpt-oss-120b (announcement / download)

gpt-5 from OpenAI is currently the TOP-1 model on our leaderboard!

gpt-5 is the smartest model, also it is quite large, slow and expensive. It is also an overkill for day-to-day business automation tasks at scale. For that we have smaller models like GPT-5-mini. And this is where the interesting things start.

gpt-5-mini is currently on the 5th place of the leaderboard, when running under Schema-Guided Reasoning. It is a reasonably priced and capable model - very well balanced.

However, in addition to API models, OpenAI has also released two open-weights models that you can freely download and run on your hardware. For example:

• Download from HuggingFace

• Run on your local hardware with ollama

• Or experiment with via OpenRouter

The most curious part is that gpt-oss-120b model looks very similar to gpt-5-mini model on our benchmark. It is as if two models were almost the same.

Either way, this is the first time in forever to see a model from the TOP-5 that is shared publicly for free use.

Likewise, gpt-5-nano model scored 16th place on our leaderboard. gpt-oss-20b model had very similar results, taking 18th place.

ℹ️ While the models are called gpt-oss, they are not exactly Open Source models, but rather Open Weights models. This means that one can download and use these models freely, but the original training data and pipelines are not shared.

gpt-oss models leverage Mixture-of-Experts (MoE) architecture, where only a small part of the model is used to generate each new token. This makes these models really fast.

You can run gpt-oss-120B model on a single H100 GPU (it requires a modern GPU with 80 GB VRAM), while gpt-oss-20B requires a modern GPU with 16GB of VRAM like RTX 5090).

Now, the 'modern GPU' requirement normally means such models would not be natively supported by older GPUs like the 4090 or A100. However, there's another unusual catch.

Thanks to the MoE architecture of gpt-oss, you can also run these local models at lower speeds with surprisingly little VRAM as well.

For example, 120B can run fairly well (10-30 tokens per second) on older cards with just 5-8GB of VRAM. In that case we are keeping attention part of the model in GPU, while all the experts reside in usual RAM (you would need 64GB for that). You can read more about configuring llama.cpp to run these models on Reddit (discussion).

ℹ️ TL;DR: there is a new --cpu-moe switch that supports MoE offloading to CPU (not yet supported in Ollama).

There is one problem with the GPT-5 release. It uses a completely new response format for defining conversations, called: OpenAI Harmony. This format doesn’t play well with Structured Outputs as of yet. This means that OpenAI API occasionally fails to fullfill its promise to always return JSON that complies with the provided schema.

We encountered failures with OpenAI SDK, when gpt-5, gpt-5-mini and gpt-5-nano returned responses that were incompatible with the provided schema. Here is the Gist that reliably reproduces the problem with all GPT-5 models: SGR triggers Harmony parsing bug with GPT-5 models. We reported it to OpenAI directly, and also shared it with the OpenAI Community with a repro, so you can check full details yourself.

Not only does the problem go away when switching from gpt-5 models back to gpt-4o, the responses also get much faster. There are two possible reasons for that:

• gpt-4o doesn’t use more complicated Harmony response format

• gpt-4o is not a reasoning model. GPT-5 models like to think under the hood before responding, while gpt-4o just answers.

Issues with the new gpt-5 models don’t stop at the API, they also affect gpt-oss-120B and gpt-oss-20B models. None of the public LLM providers that offer APIs with Structured Outputs yet provide access to these that works with StructuredOutputs. Even Ollama struggles with this new format (see ticket)

ℹ️ How did we get GPT-5 models to work reliably in our benchmark? We didn’t. We simulated a working constrained decoding by discarding all responses with invalid schema until a valid one was produced.

We are pretty sure that the integration issues will be resolved soon enough. Then we would get a great local model that is smart, fast and can be led to reason within SGR, further boosting its capabilities.

Grok models historically scored low on our benchmarks. However Grok-4 suddenly jumped to the top of the leaderboard, getting scores similar to GPT-5 (medium reasoning effort).

The primary gotcha with Grok-4 is that it can get quite expensive and slow. Here is an example of one request from our benchmark, where it took 50 seconds just to start getting response.

Gemini 2.5 Pro is currently one of the best general-purpose models to use in business automation tasks. It features a large context (that it can actually work with), can handle multiple modalities and is quite cheap.

The only problem is that Google LLMs still don’t have a proper Structured Output (similar to the capabilities of Mistral, OpenAI, Fireworks, Cerberas, Grok, or any local deployment). They feature only a limited subset that can be a pain to work with.

Anthropic models have been mediocre at best in the past months. The highest they reached was TOP-12 with a rather expensive claude-3.7-sonnet in thinking mode. Anthropic, however, doesn’t support Structured Outputs via constrained decoding, which makes integration with their LLMs rather unreliable.

New Qwen-3 models gained popularity immediately after the initial release at the end of April and are still praised for their quality. In fact, qwen-3-32B holds 11th place on our leaderboard, right above the Claude-3.7-sonnet:thinking

In fact, when one of the community members decided to check out what is the smallest possible LLM that can be used to drive AI agents in business scenarios, they managed to port SGR Demo to run on top of Qwen-3-4B via a local llama.cpp deployment.

ℹ️ The concrete version being used in this sample is quantised version of Qwen3-4B-Instruct-2507: Qwen3-4B-Instruct-2507-Q8_0

This is worthy of two comments:

• Teams currently choose Qwen-3 models when they need the smallest capable model.

• Obviously, in real-world business cases running something larger would give some safety margin (e.g. gpt-oss-20B or qwen-3-32B). However, it is just crazy that such a small model can make some sense in fairly complex scenarios.

Here is the source code that shows how to upgrade a classical SGR demo to work with Qwen-3-4B. It includes three major changes:

• Removes OpenAI SDK and composes raw requests to the API exposed by llama.cpp

• Extends the prompt by spelling out business rules in more detail

• Adds one more reasoning field in the beginning of the SGR Cascade in NextStep

These changes are enough to make even Qwen-3-4B start making sense in a task that requires a multi-step reasoning agentic behaviour.

DeepSeek models were a big hit when they first came out. However multiple better models came out since then, pushing them down the benchmark ladder.

At the time of writing, the initial deepseek-r1 version has fallen to 17th place, getting the same score in SGR-driven mode as gpt-oss-20B model. deepseek-r1-0528 was just an incremental improvement, getting it to a current 14th place.

However, not that many people would be interested in running 671B model, when there is a much better 120B model available. Qwen3-32B is smaller and also better.

Recently released Deepseek Chat v3.1 didn’t perform much better on our SGR benchmark either: