Security

I spent the winter teaching people to defend against prompt injection. Layer your defenses, I said, and you can ship systems you trust. I still believe that. But I found an attack that walks through every layer I described, and it has no payload at all. It is just an argument. You push back on the model’s conclusion with confidence, and it folds. Every time. No guardrail fires because no guardrail was watching the conclusion itself. That changes things.

Non-determinism in LLMs isn’t a bug to fix - it’s a parameter to control. But most teams are tuning the wrong things: obsessing over temperature settings while ignoring the architectural choices that actually matter. Structured outputs, deterministic skills for calculations, plan-then-execute agent patterns, retrieval logging - these are what separate systems you can trust from ones that roll dice in production. The goal was never a deterministic LLM. It was knowing exactly where randomness is allowed to live.

Non-determinism in LLMs creates real operational problems: flaky tests, irreproducible bugs, compliance nightmares, and unreliable agents. Most people only know about token sampling, but randomness creeps in across six distinct layers—from floating-point variance to hidden system prompts. Temperature=0 and random seeds help less than you’d hope because they constrain token selection, not reasoning paths. The solution requires structural constraints, not parameter tuning. Until then, you’re rolling dice in production.

Prompt injection is architecturally unfixable in current LLMs, but defense-in-depth works. Training-time defenses like Instruction Hierarchy, inference-time techniques like Spotlighting, and architectural isolation create practical systems. Microsoft’s LLMail-Inject showed thatadaptive attacks succeed at 32% against single defenses, 0% against layered approaches. Real failures like GitHub Actions compromise prove that securing obvious surfaces isn’t enough. Like SQL injection, it’s manageable with layering.

LLMs fundamentally cannot distinguish between instructions and data. Whether you’re building RAG systems, connecting MCP servers to your data platform, or just using AI tools with sensitive information, every retrieved document is a potential instruction override. The Wall Street Journal just proved this by watching Claude lose over $1,000 running a vending machine after journalists convinced it to give everything away for free.