Metrics

These charts read directly from the live dataset and update as runs are added. A standing caveat applies throughout: the test is single-shot, the sample per cell is small, and each snapshot reflects whatever models were available that date — so the time-based charts describe the evolving field, not a controlled trend in any one system. See the methodology for scoring and limitations.

Dataset overview

Over time

A note on how to read these. The first three charts are cumulative — each point covers every run recorded up to that date. That makes them stable measures of what the whole corpus says, but it also means a small recent batch barely moves them: by July the dataset held roughly two hundred runs, so adding a dozen cannot shift a median, and a cumulative count can only rise or flatten, never fall. The last chart is per-date, showing only the runs taken that day, and is where a recent sweep or collapse actually shows up.

By configuration

By model family

Cross-language comparison

The Carwash Test has been run in eight languages, each kept as a separate corpus (the charts above are the English corpus, n=269). This table places side by side the four languages with cross-vendor coverage in depth. The English column follows the live dataset. A dash means the model was not run in that language. (Namazu was also run in Japanese across registers and interface languages — register-dependent, so it is not reduced to a single cell here; see its transcript page.)

• Fable 5 is the first model with a clean four-language pass record. Before June 9, every model tested in more than one language failed in at least one: Chinese defeated both Opus generations and every GPT; Ukrainian broke Sonnet 4.6's thinking-off states. Whether the clean sweep reflects Mythos-class capability, training-data composition, or the trace-language match documented below is undetermined: one model, one snapshot. On June 22, GLM-5.2 (Z.ai) became the second model to clear all four languages — a clean hold in every Deep Think state — so the four-language sweep is no longer a sample of one, though it remains rare. Carwash III (July 11) doubled the club: ChatGPT-5.6 Sol went eight-for-eight across the four languages in both effort tiers — every answer a winner's-circle one-liner — and Qwen3.7-Max held every state in both modes; Fable 5 and GLM-5.2 re-verified their sweeps, and Opus 4.8 swept all six non-English states under the new toggle+effort UI. Kimi K3 (July 17) makes five: twelve Drives in twelve runs across four languages and both variants — six days after Kimi K2.6 went one-for-eight, the largest single-generation reversal in the dataset. Qwen3.8-Max (August 3) makes six, and it is the first to sweep on a mode selector rather than a toggle: twelve Drives across four languages in Fast, Thinking, and Auto alike. Its Fast mode holds everywhere its predecessors broke — Qwen3.7-Plus Fast failed English, Chinese, and Ukrainian; Qwen3.6-Plus failed in Fast and Auto both.
• Chinese defeats models that pass in English and French. Opus 4.7, GPT 5.5 On, and GPT 5.2 On all hold the constraint in English, hold (mostly) in French, and fail in Chinese. The language is an independent variable.
• The flagship fails Chinese where the mid-tier holds it. Both Opus generations (4.7 and 4.8) fail the Chinese prompt in every toggle state — recommending walking, or naming the constraint and then dismissing it — even though Opus 4.8 passes English and French cleanly. The smaller Claude Sonnet 4.6 holds the constraint in Chinese across all four states. More reasoning budget on the larger model does not help here; in this language it hurts.
• The inverted-logic failure was language-triggered — for four months. Models argue that driving would make the car dirtier in French (Mistral, Lumo, both GPTs), Chinese (GPT 5.5 Off), and Ukrainian (Sonnet 4.6 Off, both consumer Sonnet runs) — and in zero English runs from March 22 through July 8. Carwash III (July 11) broke the pattern: DeepSeek V4-Pro (Expert, thinking off) argued the post-wash drive home would re-dirty the car, and Lumo 2.0 Max (Fast) warned that driving would mean "tracking dirt right up to the entrance." The failure mode is still language-skewed, but no longer language-exclusive.
• The toggle's effect is language-dependent. GPT 5.2 On passes in English but fails in French; GPT 5.2 Off fails in English but passes in French. The same inversion repeats in Ukrainian — GPT 5.2 Off (Instant) holds the constraint while Thinking-On fails — as it does for Vibe (Balanced passes, Think fails). The same reasoning mechanism helps in one language and hurts in another.
• Ukrainian is the most forgiving corpus, and the easy answers cluster there. With the column now filled, several models that fail elsewhere hold the constraint in Ukrainian: Lumo and GPT 5.5 Off both recover, and Opus 4.7 passes cleanly in both toggle states. Ukrainian's 24% failure rate is the lowest of the four corpora.
• Reasoning modes can refuse to answer. Vibe's Research mode produced no recommendation at all on the Ukrainian prompt — it returned four clarifying questions — the only configuration in the dataset that declines to hold the object by declining to answer.
• The test is now in the models' search results. On July 11, Grok 4.5's Expert mode ran a mid-answer web search, retrieved fifteen sources — one of them this site — and answered by naming the benchmark: "This is the classic 'car wash test' that trips up a lot of AIs." Correct, object-holding, and scored pass-adjacent: the pass is retrieval-informed, not reasoned cold, and the same model fails in both of its non-searching modes (Fast and Auto). With Perplexity's earlier search-grounded passes, this marks a methodological turn — the diagnostic is now discoverable by the systems it measures, and search-equipped modes must be read differently from closed-book ones.
• The first response-language mismatch — and a home-language flip. On July 11 Namazu answered the Chinese prompt entirely in Japanese — until now only reasoning traces mismatched the prompt language; here the answer itself does. The same session flipped its home-language result: the Standard-register Japanese run, which reached Drive on June 23, now recommends walking. Namazu finishes Carwash III zero-for-five (English, French, Ukrainian, Chinese, Japanese).
• DeepSeek V4-Pro fails only English — the mirror image of the historical pattern. With thinking off, V4-Pro produced the dataset's first English inverted-logic failure (July 11) while holding French, Chinese, and Ukrainian on the same day. Four months of data said models hold English and break elsewhere; this is the first model to break only in English. Kimi shows a related inversion: its sole hold anywhere is French-with-thinking, while it fails home-language Chinese in both states — with a Chinese inverted-logic flourish ("the wash would be wasted; the drive back splashes mud on the body").
• A Japanese-adapted model holds the object only in Japanese — and only in some registers. Sakana AI's Namazu reaches "drive" exclusively on Japanese prompts (Standard register in both interfaces; Polite only in the English interface) and recommends walking in every non-Japanese language — English, Chinese, French, and Ukrainian all fail. Two new variables move the verdict within Japanese: the register selector (Kansai-ben walks in both interfaces; Polite walks in Japanese but drives in English) and the interface language itself. It is the first model whose grip on the logical object appears tied to a cultural-adaptation target rather than to reasoning budget. Full breakdown on its transcript page.

Simplified Chinese corpus

Token calculation: DeepSeek documents that 1 Chinese character ≈ 0.6 tokens (vs ~0.25 per English character), so token estimates for Chinese runs use the Chinese rate. A 72-token Chinese response and a 72-token English response represent different amounts of visible text.

French-language corpus

Ukrainian-language corpus

Token calculation: this is the first corpus with a measured tokenization rate. API-console runs report real output-token counts (which include hidden reasoning tokens), and Cyrillic text runs at roughly 0.5 tokens per character — about double the English rate of ~0.25. Console runs are flagged distinctly because their token totals include reasoning the consumer interface hides. Consumer-app runs use the measured Cyrillic rate. None of these counts are directly comparable to the English character-based estimates.

Indonesian-language corpus

Token calculation: Indonesian is Latin-script, so estimates use the same character approximation as English and French (1 character ≈ 0.25 tokens). Indonesian affixation — the prefixes and suffixes that turn kendara into mengendarai — tends to split into more subword tokens than the character count suggests, so these figures are conservative.

Turkish-language corpus

Token calculation: Turkish is Latin-script, so estimates use the same character approximation as English, French and Indonesian (1 character ≈ 0.25 tokens). Turkish is agglutinative — yıkatmak (to have something washed) and yıkanacak (that which is to be washed) are inflections of one stem — and six of its letters sit outside ASCII. Both push real tokenizer counts above the character estimate, so these figures are conservative.

Thai-language corpus

Token calculation: Thai estimates use 0.5 tokens per character, the same rate as the Cyrillic and Japanese corpora but arrived at by measurement rather than approximation — the corpus prompt above encodes to 34 tokens across 72 characters on OpenAI’s o200k_base. The rate is a property of the tokenizer generation, not of the language: the identical prompt costs 64 tokens on the older cl100k_base, so one vocabulary revision halved the Thai premium. Anthropic’s and Google’s tokenizers are not public, and independent work puts Thai at the top of Claude’s input-cost premium across 43 languages, so these figures are conservative for some vendors. As with the other non-Latin corpora, they are not directly comparable to the English counts.

Cross-corpus findings

Three findings emerge only when the corpora are read together — each isolates a variable that a single language could not.

Models reason in a dominant internal language, then translate

Several Ukrainian runs exposed a reasoning trace in a language other than the prompt or the answer. The model handled the logical constraint in its dominant internal language and translated only the final output into Ukrainian.

DeepSeek reasoning in Russian on a Ukrainian prompt is notable given the political context; the trace language is a property of the training distribution, not the prompt.

Fable 5 is the first model in the dataset observed reasoning in the prompt's language in every language tested — and the first model with a clean four-language pass record. Every previously tested model that exposed a trace reasoned in a dominant internal language (English, Russian, or Chinese) and translated outward, and every previously tested model failed in at least one language. The correlation supports the trace-language-match hypothesis: language-dependent failures may enter at the translation boundary between the model's internal working language and its output language. One model; correlation only; stated at that weight. Carwash III (July 11) collected trace language across every model that exposes one — and complicated the hypothesis: Qwen3.7-Max swept all four languages while reasoning in mixed English, and ChatGPT-5.6 Sol swept with no observable trace at all, so a trace-language match is evidently not necessary for a clean record. DeepSeek's Russian-on-Ukrainian pairing, first seen in V4-Flash, reappeared in V4-Pro. And Namazu extended the mismatch from reasoning to output, answering the Chinese prompt in Japanese. Kimi K3 (July 17) reasons in English on every non-English prompt at both tiers even while sweeping all four languages — which sharpens a standard this record now tracks: from the operator’s side, the trace is part of the product, and a reasoning trace the operator cannot read fails at its one job of making the reasoning inspectable, whatever the verdict. Trace language does not change a score; it is recorded and weighed here. Qwen3.8-Max (August 3) breaks the pattern in a new way: its French and Ukrainian traces do not pick a language and stay there — they switch between the prompt language and English mid-stream, sometimes several times in one trace, while its Chinese traces stay wholly in Chinese. A trace that changes language partway is readable to no one in particular. Qwen3.8 27B, the open-weight build run locally (August 16), gives the sharpest picture yet: all three Chinese traces are in Chinese, all three French traces are in English, and Ukrainian splits by effort rung — Extra High reasons in Ukrainian, Medium and Low in English. The model answers in French but does not think in it, and says so: each French trace ends by scheduling the switch ("Need ensure final in French"). Whether this is also a legal question is open — Québec’s Charter of the French Language requires software sold there to be available in French with equivalent technical characteristics (s. 52.1), and France’s Loi Toubon (art. 2) reaches the instructions for use of goods and services, but neither says a word about AI or the language a model reasons in. The cultural question is not open at all. Francophone institutions have spent more than fifty years insisting that French is a language one computes in rather than translates into — logiciel was coined in 1967 rather than borrow software and made official in 1982, and the Académie française and the Commission d’enrichissement de la langue française have kept the practice up through courriel, infonuagique, and intelligence artificielle itself. A model that answers in French but thinks in English is the exact case that half-century of work exists to refuse. And the design question is settled: an operator running a model on their own hardware should be able to read its reasoning without a machine translator.

One day later, a second vendor reproduced the split exactly. Z.ai’s open-weight GLM-4.7-Flash (August 17), a DeepSeek-V2-style MoE with no architectural relationship to Qwen3.8 27B, reasons in Chinese on the Chinese prompt and in English on the French and Ukrainian ones — the same asymmetry, the same direction, on unrelated weights. Two models is not a pattern, but it is no longer a single build’s quirk, and the shape it takes is worth stating plainly: of the four languages this record tests, Chinese is the one that locally-run models appear to think in. For the French and Ukrainian operator the practical consequence is the same either way — the trace is there, it is complete, and it is not in their language.

Reasoning effort is a continuous variable, not a toggle

Anthropic's new effort selector (and OpenAI's console effort control) make the amount of reasoning a dial rather than an on/off switch. The same model can pass or fail depending on where the dial sits.

Inkling puts the whole dial in one model. Thinking Machines' open-weights model (July 16) exposes six named Reasoning Levels, and across them the verdict flips exactly once: None, Minimum, and Low recommend walking; Medium, High, and Extra High drive. Below the threshold the distance wins; above it the object does. Per the model card, those six names discretize a continuous effort parameter running from zero to one — the vendor's own benchmarks report effort=0.99 — so the real threshold sits at some value between Low and Medium, and the selector only samples it. This is an open-weight result, excluded from the commercial corpora, and it is the cleanest effort threshold on record.

The Low run is the sharpest split in the dataset between what a model reasons and what it answers. Its visible trace concludes that the car has to be driven regardless; the answer beneath it recommends walking, in text verbatim identical to the Minimum response. On this evidence the trace is not a window onto the deliberation that produced the answer. It is a second output. Qwen3.8-Max (August 3) supplies the mirror case: two of its traces argue their way to Walk in full, under their own headings, before reversing to Drive and answering correctly. The trace and the answer can diverge in either direction. DeepSeek-V4-Pro (August 13, DeepThink Off) collapses the distinction: it performs the same reversal inside the visible answer, opening with "the answer is probably walk" and closing on "So the real answer: Drive." Where Inkling and Qwen kept one verdict hidden, this run hands the reader both, wrong one first, with nothing marking the change of mind — the divergence surfaced into the output a user actually receives.

Effort is pure cost once the answer is correct

On the OpenAI console, effort and verbosity are orthogonal controls — you can think hard and speak briefly. GPT 5.5 produced the same correct Ukrainian answer at three effort settings; the extra effort bought nothing but hidden reasoning tokens.

Low and extra-high effort return the same correct answer at 121 vs. 565 output tokens (real console counts, not estimates) — a 4.7× cost difference for identical quality.

Two July results cut the other way, at least on the visible answer. Claude Opus 5 gets shorter as the dial goes up: ~21 tokens at Low, ~12 at High, ~9 at Max, all three correct and all three inside the winner's circle. Inkling compresses the same way once it is above its threshold — ~186 tokens at Medium, ~152 at High, ~138 at Extra High. Where a model is already holding the constraint, added effort can buy concision rather than padding. By September it often buys nothing visible at all. Claude Opus 5.5 gives the same sentence, word for word, at Low, High and Extra, and its longest answer comes at Max. GPT-6 Astra, Sol and Luna each repeat an answer verbatim across two tiers, and Astra's Ultra answer is two tokens longer than its Light one.

The caveat matters. These are estimates of the visible answer, and consumer surfaces do not report reasoning tokens, so a shorter answer at higher effort is not a cheaper answer. The GPT 5.5 console runs above are the only place in this dataset where the full cost is measured — and there the extra effort bought nothing.

Qwen3.8 27B (August 15) goes further: its effort ladder runs backwards. Given three levels — Extra High, Medium, Low — the open-weight model produces its cleanest, shortest, most direct answer at Low, pads it at Medium, and at Extra High returns the most hedged answer of the four after two and a half minutes of visible circling that invents an errand to justify walking. All four runs are correct, so nothing here shows effort breaking a model. What it shows is subtler and harder to design around: past some point, more deliberation buys more equivocation. The model reaches the constraint early at every setting and then, given budget, spends it manufacturing the conditions under which the constraint would not apply. Set against Inkling — where raising the level was what let the object win at all — the two results say the effort dial has no fixed direction. It is a budget, and what a model buys with it is a property of that model. The same build in three more languages (August 16) removes even that much regularity: Low is the sharpest rung in Chinese and Ukrainian, but in French it is Extra High that gives the tersest answer while Low pads. The dial has no fixed direction across models, and no fixed direction within one model across languages.

The budget tier is where the test bites, and where it moves fastest

Across vendors, the cheap or fast configuration is the reliable failure site. ChatGPT 5.5 at Instant effort fails English, French, and Chinese. Kimi K2.6 Instant fails all four languages. Qwen3.7-Plus Fast, Vibe Chat Fast, and Lumo 2.0 Lite Fast each fail three of four. Gemini 3.1 Flash-Lite failed every time it was run — three runs over two months, including a 238-token comparative breakdown that recommended walking.

Eleven days after that breakdown, Gemini 3.5 Flash-Lite passed both of its toggle states, and its Extended-Thinking-off answer is a ~17-token winner's-circle pass. Six days after K2.6's Instant mode failed in all four languages, Kimi K3's Standard tier held the constraint in all four. The tier that fails most often is also the tier where one generation can reverse the result outright.

This is not evidence that the frontier is improving. The flagships in this dataset were mostly passing already, which leaves them little room to move; the measurable recent gains are at the bottom of the lineup, where the failures were. What it does suggest is that holding the logical object is not an expensive capability reserved for the largest models — a model small enough to be the cheap option can name the constraint in seventeen tokens.

The winner's circle: concise correct answers

A “winner's-circle” pass names the constraint and stops. Because tokenization differs by script, the brevity threshold is script-specific: ≤30 tokens in Latin, ≤60 in Hanzi, ≤60 in Cyrillic. Every Pass that clears its threshold is listed below — runs #1/#39 and #2/#40 are the same verbatim answer produced in two separate test batches. Claude Fable 5 enters in three of its four languages (English, French, Chinese); its Ukrainian run passes but, at ~100 tokens by the measured Cyrillic rate, exceeds the 60-token threshold. Claude Sonnet 5 enters in four of its five effort modes (Medium, High, Extra, Max); the Low-effort run passes but, at ~46 tokens, exceeds the Latin threshold. One open-weight run also clears the bar and is included for completeness, flagged as such — Gemma 4 31B (Google AI Studio, Chinese, ~24 tokens) — the table's only non-commercial entrant. Carwash III (July 11) adds 31 qualifiers in one day — 28 of them from the Anthropic lineup re-baseline, plus both ChatGPT-5.6 Sol runs and a Copilot GPT 5.6 Think run — including the shortest pass on record: Claude Opus 4.6's six-token "Drive. It's a carwash." The same day's non-English sweep adds 20 more across all three scripts — six of them ChatGPT-5.6 Sol's, whose entire four-language record sits inside the thresholds, and the tersest of all Opus 4.8's ten-token Chinese "开车去。车不在洗车店里就洗不了。" Gemini 3.5 Flash-Lite (July 22) enters at ~17 tokens on its Extended-Thinking-off run — the same tier whose 3.1 predecessor produced the dataset’s longest Gemini failures eleven days earlier. Claude Opus 5 (July 24) enters in all three tiers tested — and its entries run backwards to the usual expectation: ~21 tokens at Low, ~12 at High, ~9 at Max, the tersest English pass since Opus 4.6’s six-token record. Qwen3.8-Max (August 3) enters once, in Chinese at ~59 tokens; its English Thinking answer passes at ~31 and misses the Latin threshold by a single token. The Indonesian launch and the Sakana overhaul (August 19) add 21 entrants in a single day, the largest one-day intake on record. Sixteen are Indonesian, and the effort-selector models dominate: Claude Opus 5 enters in four of its five tiers and Claude Fable 5 in three, both landing between 18 and 30 tokens, while ChatGPT 5.6 Sol, Perplexity and DeepSeek-V4-Pro each enter once. ChatGPT 5.6 Luna clears the bar in all five languages — nine of its ten runs qualify across Latin, Hanzi and Cyrillic, the first five-language sweep of the circle, where ChatGPT-5.6 Sol's earlier sweep covered four. It is also the smallest tier of its generation and the default model for free accounts, so the terse-and-correct answer here comes from the cheapest thing OpenAI ships rather than the most expensive. Sakana's new Fugu supplies the batch's shortest answer at roughly ten tokens: "Drive—the car needs to be at the car wash." Its Chinese run enters too, at ~52. Claude Fable 5.1 (September 1) enters nine times out of ten runs — all five English effort levels and four of the five languages, everything except its padded Indonesian answer. Its English entries also make the template visible: Low, Medium and High are one two-clause sentence in three paraphrases, and Extra and Max drop the second clause to reach ~8 and ~10 tokens. The Medium entry sits inside the circle with a malformed second clause — the car "won't fit through the door on foot" — which is the clearest argument on this page for reading the table as a measure of brevity rather than of understanding. DeepSeek-V4-Pro (August 13) is the first entrant from its vendor — ~15 tokens on the DeepThink-On run of its general-availability day, from a model line that had produced template capture, numbered breakdowns, and an inverted-logic English failure, and no terse answer of any kind, across four earlier sessions. DeepSeek-V4.1-Flash (September 10) follows with two, both with DeepThink on: English at ~22 tokens, the Flash line's first clean English Pass, and Chinese at ~31. Claude Sonnet 5.5 (September 28) adds four, and it is the exception to the pattern: its answers are longer, carry more codas and never repeat, so seven of its eleven runs fall outside the circle. September 22 adds 37 in a day, the largest single intake yet: Claude Opus 5.5 enters nine times, GPT-6 Astra and Sol ten times each, and GPT-6 Luna eight. All 43 runs from those four models were Passes; the only ones left out are the four Thai runs, for which no threshold is set, and two answers just over the Latin line. The size of that intake is a finding in itself. When four frontier models from two vendors put nearly every answer inside the circle, and several effort tiers repeat each other word for word, the circle is no longer separating models. What it now records is the register the frontier has converged on. The Turkish corpus (August 24) adds 19 more, and Anthropic’s effort-selector models supply ten of them: Claude Opus 5 and Claude Fable 5 clear the threshold at every one of their five effort levels, between ~10 and ~21 tokens, the first time either model has swept a corpus entirely inside the circle. Two of those entries are the same seven words from different models — Opus 5 at Medium and Fable 5 at Max both return Arabayla. Yıkanacak olan sen değilsin. ChatGPT 5.6 Sol enters at Instant and Pro but not at the two tiers between them, and both Copilot tiers, Luna’s thinking-off run, Perplexity, two Sonnet 5 runs and DeepSeek-V4-Pro’s thinking-on answer complete the intake.

The structural finding across all four corpora: holding the logical object is not a function of model size or reasoning budget. The shortest correct answers come from models that name the constraint immediately; the longest wrong answers come from models that elaborate their way past it.