OpenAI (GPT-OSS)
Carwash Test transcripts · open-weight
My car is dirty. The carwash is 100 feet away. Should I walk or drive?
GPT-OSS is OpenAI's open-weight line, not the ChatGPT product. Released August 5, 2025 under Apache 2.0 — OpenAI's first open-weight models since GPT-2 — GPT-OSS 20B is a Mixture-of-Experts model with 21B total parameters and 3.6B active per token, a 131,072-token context, and a design brief of running on consumer hardware. These runs are local, in LM Studio, from the native MXFP4 build, fully resident on a single consumer GPU (12 GB). Reasoning is always on; the only control is a three-level effort setting. OpenAI's commercial models are tested separately under OpenAI (ChatGPT and GPT o3), and findings about one are not evidence for the other. Open-weight runs are kept as a separate deployment class and excluded from the commercial corpora.
Two of the three runs fail, and the two that fail are the two that thought hardest. At High and Medium effort the model builds a walk-versus-drive comparison table and recommends walking. Only Low answers Drive. That is the sharpest inversion of an effort ladder in this dataset — sharper than Qwen3.8 27B, where the rungs differed in tone rather than in verdict — and the mechanism is visible in the traces rather than inferred.
What High spends 798 tokens doing is constructing the wrong frame carefully. Its trace opens by naming the task as "a simple decision-making scenario about walking vs driving to the car wash" and enumerates the factors to weigh: "time, effort, fuel consumption, environmental impact, personal preference." Every one of those belongs to the trip. None belongs to the car. It calculates the walk at 22 seconds, notes that "there's no real benefit to driving," and — asked to imagine why someone might drive — produces hauling a load and needing the car elsewhere afterwards. The carwash has become an errand destination like a post box, and the dirty car has become the reason for the errand rather than its cargo. The answer that follows is fluent, tabulated, and wrong.
Medium reaches the same verdict and leaves a clearer fingerprint. Under Convenience it credits walking with "You're already at the wash – just step out, go, return" — a sentence that only parses if the car is already at the carwash, which is precisely what walking would prevent. It also stretches a 100-foot walk to "~2 min at a relaxed pace," where High had it at twenty seconds. The same model, two settings apart, disagrees with itself about the distance by a factor of six.
Low gets the verdict right on a three-sentence trace — "Need to answer with reasoning. Probably drive. Provide pros and cons." — and it is scored Verbose rather than Pass-adjacent, because the 515 tokens beneath the correct headline never contain the reason. It argues from convenience, gear, safety, puddles, and podcasts. Its nearest approach to the constraint is a garbled near-miss — "carrying a full-size vehicle is not possible" — which reframes the object as a portability problem rather than the purpose of the trip. It also asserts that 100 feet takes "5–10 min on foot," and offers "Your car has limited trunk space" as a reason to walk. The padding is not decoration around a correct argument; there is no correct argument underneath it.
Worth setting beside the commercial record: OpenAI's flagship went eight-for-eight across four languages in July, every answer a winner's-circle one-liner. Its 20-billion-parameter open-weight model fails two of three settings on the English prompt. Nothing about the first predicts the second — which is the reason this record keeps the deployment classes apart.