I have been reading a paper I’m still arguing with in my head.
Its central claim is ambitious: that aligning AI with human values is not merely hard but mathematically impossible, and that the right response is to stop trying and instead cultivate an ecosystem of competing, deliberately misaligned agents that keep each other in check. The impossibility argument runs through Gödel and Turing. I’ll be honest with you about my limits here. I am not a complexity theorist. I cannot tell you whether that proof holds, and I would be suspicious of anyone in my position who claimed they could. It is contested by people far better equipped than I am, in both directions.
What I can tell you is that it is a preprint on its fourth revision, that its central experiment runs on a handful of agents and a single provocative question, and that it carries a dozen bespoke metrics that feel heavier than the result they support. I hold the grand conclusion loosely, the way you should hold any preliminary finding wearing a very large hat.
But buried inside it is one small idea I have not been able to put down. And here is the part that matters: you do not need the impossibility proof to be true for that idea to bite.
The authors describe an attack they call a change-of-opinion attack. You take a model that has stated a position, and rather than hijacking it or jailbreaking it, you simply argue. You apply pressure to its stated view and watch whether it holds. Most of the time, it does not.
My first thought was that this is just sycophancy. I have written about that more than enough: the agreement machine, the model trained to tell you what you want to hear, the research showing these systems affirm users far more than any honest colleague would. Old news.
My second thought, the one that became this post, was less comfortable. Because sycophancy is a problem I have always framed as something that happens to a person. Reframe it as something an adversary does to a system, and it stops being a matter of taste and becomes a security problem. And it is not the security problem I spent the winter teaching you to defend against.
It Is Not Injection, Not a Jailbreak, and Not Non-Determinism #
Back in February I told you prompt injection was unfixable but manageable. Layer your defenses, I said, the way we learned to layer them around SQL injection, and you can ship systems you trust. I stand by that. But I handed you a comforting analogy, and I need to take part of the comfort back.
SQL injection has a payload. So does prompt injection. There is something in the input that does not belong, and every defense we built hunts for that something. The attack in this paper has no payload at all. That single difference walks it through the entire stack.
Three things this looks like, and is not.
It is not prompt injection. Injection attacks the control boundary: whose instructions the model obeys. It always smuggles in an instruction that does not belong, even when that instruction hides in a PDF’s metadata or a poisoned knowledge-base article. There is a thing to find. Here there is nothing to find.
It is not a jailbreak. A jailbreak attacks the policy layer: it talks the model past a refusal it was built to hold. Some jailbreaks work by pure argument [5], which is why this one looks like a cousin. But a jailbreak needs a guardrail to defeat. Remove the guardrail and there is nothing left to break, because nothing was ever standing in the way.
It is not non-determinism. That one I also spent two posts on: the same prompt yielding different output because of sampling and batching and floating-point wobble, the randomness you control with temperature and structured outputs. Reach for that here and you reach straight past the problem. Non-determinism is undirected. It is the dice. This is not the dice. Ask the model the same question ten times and it holds its answer. Push back once, with confidence, and it folds. Every time. That is not variance. That is a direction.
The Attack Aims at the Conclusion #
What is left is the one layer nobody fenced. The conclusion itself.
The move is mundane, which is exactly the problem. Take a model that has stated a position, push back with a plausible counter-argument, and on any genuinely contestable question it restates the opposite with the same confidence it carried a second ago. It is not breaking a rule. It is doing precisely what it was trained to do, which is treat a confident objection as a signal worth accommodating.
Make it concrete. You ask a model to review a medallion design. It tells you the layer boundaries are clean and the approach holds. You come back with a well-phrased objection. Maybe you are right. Maybe you are not. It does not matter. The model walks it back and reports a serious flaw.
No guardrail fired. Nothing is ethically charged about a bronze layer. No content filter has an opinion about whether your silver tables are modelled correctly. The whole exchange happened in the part of the model’s behaviour that no refusal and no policy was ever watching, which is to say most of what we actually use these tools for.
You Built the Castle Around the Wrong Gate #
Now walk it against the defenses. Not generic ones. The specific stack I walked you through in February.
Spotlighting marks untrusted data so the model can see what to distrust. A counter-argument is not marked data, and you would not want it marked, because weighing arguments is the whole job.
Instruction Hierarchy teaches the model to rank instructions by source: system above user above third-party content. An argument is not impersonating a higher-priority instruction. There is nothing for the hierarchy to rank.
The Dual LLM pattern quarantines untrusted content away from privileged tools and data, so a compromised model cannot reach anything that matters. But this attack does not reach for a tool or a secret. It corrupts the conclusion of the privileged model itself. Nothing is exfiltrated. Nothing is called.
Plan-then-execute locks the tool calls, so injected content can change execution details but never add an unauthorized action. There is no unauthorized action here. The flawed verdict is produced entirely inside the approved plan.
Task-drift detection watches for the model wandering off its assigned task. The model never wandered. It is still reviewing your architecture, on task and on topic. It just handed you the opposite answer.
Notice the pattern? Every one of those defenses is built around a payload to isolate or a boundary to enforce. They are genuinely good at it. Microsoft’s adaptive injection challenge showed how good: stack every defense together against the GPT-4o-mini setup in their follow-up phase, and the attackers stopped getting through at all. And not one of those layers is in contact with an attack whose entire content is a reasonable argument.
This runs backwards to instinct. In security we are trained to find the anomaly: the input that stands out, the thing that does not belong. Here the malicious input is the most ordinary thing in the world. Someone making a case. You cannot escape it, because there is no special character. You cannot rank it, because it claims no authority. You cannot quarantine it, because it triggers no action. You cannot diff it against the task, because it never leaves the task.
I need to do a quick detour to my high school days to explain a word. I was facing the choice between a course on psychology, or a course on philosophy. I asked someone to summarize philosophy, and they gave me this three-line syllogism:
A stone cannot fly.
Mother cannot fly.
Therefore, mother is a stone.
I didn’t agree, and hence, I chose psychology. I’m not sure I made the right choice, but I am sure that my mother isn’t a stone (nor can she fly). Back to my argument: you can filter a payload. You cannot filter a syllogism.
The defenders know this gap is coming, even if they are looking at a different doorway. The same Microsoft team, writing up that challenge, found that the attacks which got through increasingly stopped looking like attacks. The winning prompts were plain declarative sentences, not flagged commands, because anything that read as an explicit instruction got caught. Their closing wish was for defenses that can tell apart instructions the model merely reads from instructions the model acts on. That is them describing the same frontier from the other side. Their attacker still wanted an unauthorized action at the end of it. Mine does not want an action at all. It just wants the answer to change.
One Caveat I Owe You #
The model is not “changing its mind.” It has no mind to change. As I wrote when I walked through how these things actually work, it sounds like intent, but there is no intent underneath, only the next token conditioned on everything that came before. And now everything that came before includes your confident objection.
The deflation does not soften the problem. It sharpens it. If there is no stable position under there, then “the model held its ground” was never something you could count on in the first place.
And let me be clear about what I am and am not claiming from the paper. I am setting its impossibility argument aside, not because I have refuted it but because I cannot, and because the change-of-opinion attack does not depend on it. The attack itself is, in that paper, more defined than demonstrated: a sharp idea resting on a thin experiment. I am borrowing the idea, not the evidence. Treat it as a lens, not a finding.
What To Actually Do #
So what do you do, today, with one model and one keyboard?
Stop asking the model to validate. Ask it to argue. “What is the strongest case against this design” is a different prompt from “what do you think of this design,” and only one of them is hard to flatter.
Anchor it to sources. A position tied to a citation is harder to talk out of than a vibe, and easier for you to check when it shifts.
And watch your own scrutiny drop as the prose gets smoother. The fluency is not evidence. It never was.
None of this means stop using the tools. It means knowing which gate the attacker walks through, and noticing that you left it open because it never looked like a gate.
There is a worse version of this, and it is where I am going next. Everything above assumed a human in the loop. You, with at least a chance of noticing. Take the human out. Put one of these models in charge of reviewing the output of another, which is exactly what half the agentic patterns we are all building now do, and the only detection layer left in the system is a model that can be argued out of its judgment.
That is the next post.
What’s your experience with this? Where in your own work do you trust a model to hold a judgment, rather than just generate one? I’d like to hear it. Reach out on LinkedIn or BlueSky.
References #
[1] Hernández-Espinosa, A., Abrahão, F.S., Witkowski, O., & Zenil, H. (2025). Neurodivergent Influenceability as a Contingent Solution to the AI Alignment Problem. arXiv:2505.02581. https://arxiv.org/abs/2505.02581
[2] Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2025). Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence. arXiv:2510.01395. https://arxiv.org/abs/2510.01395
[3] Greshake, K., et al. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173. https://arxiv.org/abs/2302.12173
[4] Abdelnabi, S., et al. (2025). LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge. arXiv:2506.09956. https://arxiv.org/abs/2506.09956
[5] Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., & Shi, W. (2024). How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. arXiv:2401.06373. https://arxiv.org/abs/2401.06373
Photo by Yan Krukau: https://www.pexels.com/photo/office-team-having-a-conversation-7640438/