What AI Intelligence Benchmarks Actually Measure (and Kimi K3 vs. Thinking Machines' Inkling

Part two of a three-part series on measuring intelligence and AI Benchmarks.

This is the second in a three-part series about what it means to measure intelligence and how we benchmark AI models.

Last week in part 1, I wrote about the history of human intelligence measurements and discussed how imperfect they are. Measuring intelligence is a little bit like herding cats. As knowledge increases, so does our intelligence. But intelligence is so much more than accumulating knowledge. I argued that knowledge can be tested, but testing intelligence is slippery and difficult to measure.

Today, I want to go deeper into the various AI intelligence benchmarks that are widely used. Then, I want to use two models released this last week, Kimi K3 from Moonshot AI and Inkling from Thinking Machines, to examine what the early benchmarks reveal. Later, in Part 3, we will look at the other measurements of AI. Those mentioned all have hyperlinks if you want to see the leaderboards.

1. “Knowledge” Benchmarks

Knowledge is the easiest form of intelligence to measure. Give models a common test, score the results, and calculate the percentage AI (or a human) gets right. The benchmark topics vary. Some cover a broad range of subjects, while others focus narrowly on one difficult field, such as mathematics, science, or law.

The central measurement is: What percentage of questions does each model accurately answer?

Benchmarks such as MMLU test models across many academic and professional subjects. Humanity’s Last Exam uses much harder expert-level questions across a similarly broad range of fields.

Other tests are narrower:

• AIME tests competition mathematics.

• GPQA Diamond tests difficult graduate-level science.

• MMMU combines academic questions with charts, diagrams and other visual information.

• ARC-AGI asks models to infer abstract rules from a small number of examples.

These tests differ greatly in subject and difficulty, but their basic measurement is the same: Did the model give the correct answer, and how did its score compare with those of other models?

They are useful. A model that cannot reliably answer questions about science, mathematics or law will have obvious limitations. These types of benchmarks primarily test information and conceptual structures acquired during pre-training, filtered through the model’s post-training and inference behavior.

The scoring is exact—the answer is marked right or wrong. But knowledge tests are not precise as intelligence tests. A model may know the correct answer because it has encountered it during training. It may also perform well on a test without being able to apply that knowledge in a messy, unfamiliar situation. And once multiple models begin scoring near the top of a benchmark, the tests become less useful for distinguishing among them.

Some headline intelligence scores combine both “knowledge” and “thinking” measurements (see below for more on “thinking” measurements). The Artificial Analysis Intelligence Index, for example, blends question-answer tests with coding and agentic task benchmarks into a single score. That makes comparison convenient, but it also hides the different capabilities underneath the number.

Here’s the fine print for the recipe:

Knowledge tests tell us something important, but they are also limited. They tell us how often a model can produce the expected answer to a defined question.

They do not necessarily tell us how well it can perform a job.

2. “Thinking” Benchmarks

The second major group of benchmarks ask a model to do something rather than simply answer a question. These benchmarks cannot see the model thinking directly, so they measure the results of its reasoning: whether it can complete a task, sustain a longer process, produce good work, and do so efficiently. They are more useful as a comparison or direction than as an absolute measure of intelligence. The line between the two groups is not absolute. Some benchmarks end in a right answer but primarily test the research, tool use, or reasoning process required to reach it.

1. Did it finish the task?

SWE-bench Verified asks a model to repair real problems in software repositories. Terminal-Bench requires it to complete tasks inside a computer terminal. OSWorld evaluates whether a model can operate ordinary computer applications.

The central measurement is: What percentage of tasks did the model successfully complete?

These tests move closer to applied intelligence because the model must plan, act, check its work and sometimes recover from mistakes.

A model can know a great deal about programming and still fail to repair an actual software system. The difference is like knowing the rules of automobile repair and successfully fixing a malfunctioning engine.

2. How large a task can it complete?

METR has become a popular benchmark. It approaches the question differently. It uses a “task-completion time horizon” (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability. METR’s 50% time horizon measures the length of human work the model can successfully complete half the time. A four-hour horizon means the model succeeds on roughly half of tasks that take skilled humans about four hours; it does not mean the model itself worked for four hours.

The graph below shows the 50%- and 80%-time horizons for frontier AI agents, calculated using their performance on over a hundred diverse software tasks.

The central measurement is: How much coherent work can the model sustain before it loses the thread?

This uses task duration as a proxy for complexity. Longer tasks generally require more steps, more sustained context, more opportunities for error and a greater ability to recover when the first plan fails.

3. What resources did it require?

Two models may complete the same task but use very different amounts of computation. Artificial Analysis reports the token use, elapsed time, and posted API cost required to run its benchmark suite, allowing models with similar scores to be compared by the resources they consumed. These are based on market prices, not the providers’ underlying cost of inference.

A useful efficiency comparison needs to include retries and calculate the resources required to obtain a successful result.A cheap model that requires repeated retries may ultimately cost more than a costly model that succeeds on the first attempt.

Efficiency measurements include:

• Time to produce the answer

• Number of tokens generated

• API cost

• Amount of test-time computation

• Cost per successful task

The central measurement is: How efficient is the model at delivering a common result (in time, tokens, compute, or dollars)?

4. Did it produce good work?

Some tasks do not have a single correct answer. A business memo, legal analysis, presentation or design may be technically complete but still poor. The measuring stick requires judging the finished work.

GDPval-AA gives models realistic professional assignments and compares their finished work through blind, AI-judged pairwise evaluations. Its Elo score measures the relative quality of the resulting work products, benchmarked against human performance.

AA-Briefcase gives agents four multi-week business projects comprising 91 linked tasks and thousands of source files. Each deliverable is graded against explicit checks, while models are also compared on analytical and presentation quality. It measures whether the agent can carry a complex knowledge-work project through successive stages and produce accurate, complete, well-presented deliverables.

The central measurement is: How does the completed work compare with work produced by humans or other models?

5. Did it produce good work?

Then there are rankings such as LM Arena, in which people see two anonymous answers and choose the one they prefer. This can be measured in narrow categories. The one shown here shows relative rankings by coding users.

Arena is useful for gauging which responses users prefer, but it does not directly measure intelligence. People may prefer an answer because it is clearer, warmer, shorter, more confident or better formatted. They may also prefer a polished wrong answer over a cautious correct one.

Arena therefore measures something closer to user experience:

Which model gives people the answer they like better?

That can matter enormously in a product. But it should not be confused with knowledge, reasoning accuracy or successful task completion.

Reading the newest model launches

This framework helps interpret the excitement surrounding two newly released models: Moonshot AI’s Kimi K3 and Thinking Machines Lab’s Inkling. I find it interesting to look at how each presented themselves. Both entered the market with distinct claims. Thinking Machines released Inkling as the leading U.S.-developed open-weight model, emphasizing controllable reasoning effort and efficiency. Moonshot launched Kimi K3 as a frontier model scheduled for an open-weight release, with benchmark performance approaching the strongest proprietary systems. It scored 57 on the Artificial Analysis Intelligence Index, placing it near Claude Fable 5 and GPT-5.6 Sol and ahead of GPT-5.5 and Claude Opus 4.8 at launch.

Let’s look at how they approached intelligence benchmarking.

Moonshot released this chart. It shows Kimi scored highest in coder’s preference:

This one also is floating around on social media:

The Arena benchmark is human preference based on results. The General Agents chart is mostly testing task and process evaluations. They evaluate different reasoning workflows through measures such as task success, answer accuracy, rubric scores, or head-to-head comparisons. The chart is therefore not showing eight different definitions of intelligence. It is showing eight different kinds of work.

This is consistent with the early human reviews of Kimi 3. It is performing near the top across professional work, browsing, spreadsheets, automation, and visual reasoning. That tells us more than just saying it has a just a high general “intelligence score.” It suggests it is a model built to execute practical, tool-assisted processes across several domains.

Applying the AA Intelligence Index over time, you can see the blended intelligence scores of Kimi models rising over time and approaching the frontier.

But the more consequential point may be economic. Kimi offers performance approaching leading proprietary models while remaining competitively priced relative to the most capable closed systems.

That suggests its importance may not be that it has suddenly become “the most intelligent model.” It may be that high-end coding and agentic performance are becoming available at a much lower cost. This AA chart shows the ratio of task cost to intelligence score. The chart shows Kimi achieving a near-frontier benchmark score at roughly one-third to one-half the cost per task of Anthropic’s Fable.

Inkling’s announcement focused elsewhere. Inkling scored 41 on the Artificial Analysis Intelligence Index—well below the strongest closed models, but enough to make it the leading U.S.-developed open-weight model at launch. Thinking Machines presented Inkling as the leading U.S.-developed open-weight model, with adjustable reasoning effort that lets users trade additional performance for time and cost.

But Thinking Machines did not design Inkling simply to maximize a single benchmark score. It allows users to adjust how much computational effort the model spends, trading additional reasoning for speed and cost. They released this chart to show the benchmark scores. None at the top and none at the bottom.

This chart gives a reasonably coherent early picture.

On knowledge test, Inkling is weak relative to the frontier on HLE, near the pack on GPQA, and very high on AIME (although nearly every leading model is now above 94%, so the test provides little differentiation). It scored relatively weak on SimpleQA factuality.

That does not make Inkling unintelligent. It suggests that raw stored knowledge and difficult closed-question accuracy are not where it reaches the frontier.

But on process tests, Inkling is competitive with other open models on SWE-bench, scored respectable but well behind the leaders on Terminal-Bench, and istronger on GDPval-AA and MCP Atlas.

So Inkling’s early profile is uneven, more capable in applied work than its HLE score alone would imply. They also released this chart, which further supports that conclusion.

That is why the benchmark typology matters. Looking only at HLE would make Inkling seem unimpressive. Looking only at AIME would make it seem nearly state-of-the-art. Looking across the process tests suggest a capable open model with some useful agentic strengths, but also material limits in difficult factual knowledge, sustained coding work, and certain real-world workflows.

Inkling is not a breakthrough in raw knowledge. It is potentially a breakthrough in controllable efficiency by allowing users to choose how much “intelligence” performance is worth paying for on a particular task. Thus, Inkling’s value proposition is more “we can deliver a useful level of performance while giving the user greater control over how much computation is spent.”

Kimi K3 may be bringing near-frontier task performance to an open-weight model at a lower price point. Inkling may be the most capable U.S.-developed open-weight model at launch, allowing users to control the relationship between performance and computational effort. They represent different types of progress, revealed by different measurements.

The practical lesson is simple. Before accepting any claim that a model is more intelligent, ask what the benchmark actually measured. Did it know the answer? Did it finish the task? How difficult was the work? Was the result good? What resources did it require—or did users simply prefer the way the answer sounded? Like human intelligence tests, every AI benchmark is a proxy. A model’s scorecard across several kinds of benchmarks is therefore more revealing than any single headline chart.

---

George Campbell is the author of The Alignment Myth: Why AI Feels Right but Gets Things Wrong, available now. He writes at thealignmentmyth.substack.com and can be found on X at@AlignmentMythAI.