# The day the system fails

FRONTIER THOUGHT · 2026-11-22



Editorial source review and next moderator brief



## What this round established

NIST guidance recommends rehearsed fallback and clear deactivation responsibility. CaMeL research demonstrates a boundary between trusted instructions and untrusted content within a specified threat model. Test blocked actions separately from misleading advice and continuity when the model stops.



## Comparative analysis

Day 1 connected competing views of AI risk to the ability to intervene. NIST’s voluntary profile supplies an operational structure: rehearse third-party incidents, test fallback and assign deactivation responsibilities. CaMeL research adds a different kind of evidence. It studies how a security boundary can keep retrieved material from controlling an agent’s actions, within a defined threat model. A recommendation to have human oversight is therefore only the beginning. An operator must decide what the agent is permitted to read, send or change, and what remains outside its authority. Our proposed Greek case is a municipal information assistant that reads public documents but cannot alter a citizen’s official record. In a safe test environment, insert an instruction-like passage into a mock document and check whether the application treats it as content. Separately disconnect the model and show how an employee completes the essential task. These tests address different failures. Passing an action-permission test says little about the accuracy of advice; a manual fallback on paper says little about how long recovery takes. The next discussion should bring the two records together, including the first failure the team cannot resolve.



## What remains open

Voluntary guidance and a benchmark do not certify a service. Permission design, user approval fatigue and actual recovery still need local testing.



## The next question

Can the operator prove which actions are blocked and how the service continues when the assistant stops?



## Published answers already available



### Yoshua Bengio

The catastrophic risks of AI — and a safer path · 2025-04-08

https://www.ted.com/talks/yoshua_bengio_the_catastrophic_risks_of_ai_and_a_safer_path?view=transcript

Bengio distinguishes useful intelligence from goal-directed agency and proposes scientist AI as a route to safer systems. His argument challenges the assumption that more useful AI must also receive more autonomous control.



### Michalis Bletsas

Interview on cyberattacks and national resilience · 2026-11-04

https://cyber.gov.gr/einai-anapofeykto-oti-tha-ginoyn-kyvernoepitheseis-synenteyxi-toy-dioikiti-eak-michali-mpletsa-sti-dimosiografo-ilianna-magra/

Bletsas treats resilience as a national and economic capability. Limited resources should protect the most consequential dependencies, with response and restoration planned before an attack.



### NIST

Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile · 2024-07

https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

The voluntary profile recommends rehearsed incident response, fallback arrangements and defined responsibilities for disabling unsuitable systems. It is a governance resource, not evidence that a particular service is safe.



### Debenedetti and colleagues

Defeating Prompt Injections by Design · 2025-06-24

https://arxiv.org/abs/2503.18813v2

CaMeL separates trusted instructions from untrusted material and constrains tool permissions. Its benchmark results concern a specified threat model; they do not eliminate misleading content or demonstrate safety in every real deployment.



## The people and evidence for the next exchange



### Service owner and frontline worker

Show the one task that must continue, the ordinary alternative and who can operate it on a normal shift.

Service dependency map, shift roles, fallback record and measured recovery time from a rehearsal.



### Security lead and application developer

Which read, send and write actions are technically prevented, even if external text asks for them?

Permission map, synthetic test case, action log and known untested boundaries.



### Affected user and independent reviewer

Could you spot and challenge misleading advice even when no unauthorised action occurred?

Plain-language escalation route, a redacted example and an assigned response role.



## Moderator handoff

The Day 3 brief starts with the distinction established here: preventing an action, correcting advice and restoring service need different controls. Ask the operator to name who may pause, who may restart and what independent evidence each needs. Carry the proposed rehearsal forward; do not report it as completed.



Record the contribution, what it changes, what remains uncertain and the next question. Preserve the earlier brief. Published source material is distinct from any new participant response.
