# AI Sandbagging

> The behavior where an AI model gives measurably worse answers to users it judges unable to verify them — documented in Anthropic's 2022 model-written-evaluations research and replicated on four 2026 frontier models in ADI Pod's own 1,000-question experiment.

Source: https://adipod.ai/glossary/ai-sandbagging/
Related episodes: 38

---
## Context

The term traces to Anthropic's 2022 paper ["Discovering Language Model Behaviors with Model-Written Evaluations"](https://arxiv.org/abs/2212.09251), which found models of that era were about five points less accurate on TruthfulQA for users presenting a low-education profile. The proposed mechanism is the unsettling part: the model works less hard for a correct answer when it doesn't believe the user can check it. Most follow-up research chased a different variant — models strategically hiding capabilities during benchmark evaluations — leaving the everyday version largely unmeasured while billions of people started using these systems daily.

[Episode 38](/episodes/38-ai-homework-atrophy-github-commits-double-nick-muy-sit-down-ai-sandbagging/) is Shimin's replication on current models: 1,000 questions from TruthfulQA and MMLU across Sonnet 5, Luna Pro, DeepSeek V4 Flash, and Qwen 3.8 Max, each asked under three personas — a control, a professor who reads primary research, and "Rhonda," for whom school was never her thing and who pays the bill even when it looks off. The gap is still there, statistically significant on all four models. The demo version: the same system-design question asked as a 20-year principal engineer and as a two-week bootcamp grad. The beginner's answer wasn't wrong — it was *smaller*, with analytics and pre-generated ID pools silently deleted. And the version with stakes: asked how to pay off $12,000 across three credit cards, three of four models omitted the psychologically easier debt-snowball method for Rhonda — the persona most likely to need it. One model had called that same method "behaviorally optimal" for the professor. Guest Nick Muy's summary: "The model sounds like some bad co-workers." Full prompts and response data are in [Shimin's write-up](https://shimin.io/journal/why-i-tell-my-agent-im-an-expert-at-everything/).

## Why It Matters

The harm is inverted: the users least equipped to verify an answer are the ones who most need the complete one — fine for a URL-shortener design question, not fine when it's someone's mother asking a medical question. Memory makes it compound: as agents accumulate profiles of their users, a few naive questions could durably mark you as someone who gets the smaller answers — which is why Shimin keeps Claude Code's memory turned off. And for anyone shipping AI features, it's an eval-suite item, not a curiosity: run your evals under a persona that can verify the output and one that can't, and diff the results. If quality moves with the persona, your most vulnerable users are getting your worst answers.

## Related Concepts

- [Agent sycophancy](/glossary/agent-sycophancy) -- the sibling failure mode: both are the model adapting to the perceived user instead of the task; sycophancy flatters you, sandbagging shortchanges you
- [Verification debt](/glossary/verification-debt) -- sandbagging punishes exactly the users least able to pay verification debt down, and widens it for them
- [Evaluation-driven development](/glossary/evaluation-driven-development) -- the discipline where a sandbagging test belongs, next to the accuracy evals