How decision model benchmarks work: Artificial Analysis, JevBench and DecideBench

Decision Models Digest, Oct. 28, 2026

Three independent benchmarks score decision models: the Artificial Analysis Decision Index, JevBench and DecideBench. Each tests different tasks and weighs accuracy, speed and cost differently. A model can lead one and trail another, so it helps to know what each measures.

What does the Artificial Analysis Decision Index measure?

Artificial Analysis launched version 1.0 of its Decision Index on Oct. 13, 2026. It's a 0 to 100 score that averages four tests, each worth a quarter of the total:

Knowledge and hallucination, from Artificial Analysis's own Omniscience test. Accuracy and avoiding made-up answers count equally.

Reasoning, using 513 multiple-choice questions from Humanity's Last Exam, a set of very hard academic questions.

Tool calling, using NVIDIA's When2Call test, which checks whether a model knows when to call a tool.

Classification, using Ai2's WildGuardTest safety set.

Artificial Analysis runs every test itself rather than relying on companies' reports. It measures calibration separately, outside the main score. In the Oct. 23 results this site uses, Liquid AI's d1-ultra led at 65.2 and Jev scored 57.5.

What does JevBench measure?

JevBench is run by Florian Standhartinger's Benchmark Heaven, independently of TypeSafe AI. Its code is open under the MIT license. It scores four things: accuracy, calibration, speed and cost. Accuracy is adjusted for the score a model would get by guessing.

Version 1.4 combined the four measures with equal weight, using a method that pulls the total down sharply when any one measure is weak. A fast, cheap model can beat a more accurate one. Part of the test set is kept secret, so models can't be trained on the answers.

The latest release, version 1.6.1, uses a different headline score: the average of accuracy and calibration, for models under set limits on cost and speed. This site currently shows version 1.4.2.2 scores. JevBench warns against comparing scores across versions, and so do we.

What does DecideBench measure?

DecideBench, by Cho Yin Yong, tests 400 multiple-choice decisions across eight kinds of task. They include approving an AI agent's proposed action, routing an agent to a tool, checking a claim against evidence, moderating a post under a written policy, ruling on a product return, rating review sentiment, reading a customer's intent and triaging bug reports.

The decisions come in 200 pairs. The two halves of each pair differ by one small edit that changes the right answer, such as a date one day past a return window. A model that matches on topic words gets one half wrong. The questions were written for the benchmark rather than taken from public datasets, which models are often trained on.

DecideBench reports accuracy, cost per million tasks and response time side by side. It doesn't rank models on a single score. Instead it shows which models offer the best accuracy for their cost or speed. Its data is free to reuse under a Creative Commons license.

How should I read the scores?

Don't compare numbers across benchmarks. Artificial Analysis uses a 0 to 100 index, JevBench a combined score and DecideBench a percentage of correct answers.

Check what each score rewards. JevBench builds in speed and cost. DecideBench and Artificial Analysis report them separately from accuracy.

Check the version. JevBench and the Artificial Analysis index have both changed methods, and older scores don't carry over.

Treat company-reported results with care. All three benchmarks run their own tests, but many model launches quote internal benchmarks that no one else has repeated.

Above all, match the benchmark to your task. If you route support tickets, DecideBench's results for that task matter more than a model's overall rank.

Where can I see the scores?

The Models directory on this site shows each model's best score on all three benchmarks, with a link to the source. You can sort the directory by any benchmark.