A monetarily-incentivised laboratory experiment published this week — flagged by Schneier on April 16 — found that humans behave systematically differently when they believe they are playing against an LLM instead of another human. In a multi-player p-beauty contest game, participants chose numbers closer to the Nash equilibrium against LLM opponents than against human ones. In plain language: they assumed the machine would play rationally and cooperatively, and adjusted their own moves to match.
Why this matters outside the lab
The p-beauty contest is a classic economics game, but the mechanism it exposes — humans mentally crediting a machine opponent with rationality and good-faith cooperation — is the same mechanism that shows up in every mixed-initiative system security teams are now shipping to production. Three concrete places this lands:
- Code review with an AI agent in the loop. Dark Reading’s coverage of agentic coding tools (linked above) already flags the pattern: reviewers approve agent-authored PRs faster and with less scrutiny than identical PRs from a junior human. The lab result generalises that observation.
- SOC triage. If an agent co-pilots a human analyst’s investigation, and the analyst extends the same “rational and cooperative” inference to the agent’s suggestions, false negatives get a new propagation path — one where the analyst isn’t actually skipping work, they’re trusting a teammate that doesn’t exist.
- Delegated authority. Anywhere a human signs off on an agent’s proposed action (a refund, a firewall rule, a deploy), the default trust calibration is probably too high.
The identity angle
Schneier’s framing — humans attribute rationality and cooperativeness to LLMs — is the human-factors mirror of the machine-identity problem we covered earlier this week. Security programmes are getting good at instrumenting what agents do, but the human-in-the-loop control is only as strong as the human’s scepticism, and this study suggests that scepticism is quietly being eroded by the interaction model itself.
Practical moves
- Rename the audit log. If your UX says “AI assistant suggested” instead of “agent #443 proposed action requiring confirmation”, you are leaning on the trust bias the study documents. Shift the language toward the identity frame.
- Force a second-party confirmation for high-impact agent actions. Not a checkbox the same human clicks, but a genuinely separate reviewer. The lab finding predicts the same human will under-scrutinise.
- Measure acceptance rate as a risk signal. If your team’s acceptance rate of agent suggestions trends up while the action set stays constant, that’s probably the trust drift this study predicts — not the agent getting better.
Schneier’s full post links to the underlying arXiv preprint for teams that want to run the numbers themselves.
