From positions to tests
THE QUESTION WE STARTED WITH
The day the system fails
NIST guidance recommends rehearsed fallback and clear deactivation responsibility. CaMeL research demonstrates a boundary between trusted instructions and untrusted content within a specified threat model. Test blocked actions separately from misleading advice and continuity when the model stops.
THE NEXT QUESTION
Can the operator prove which actions are blocked and how the service continues when the assistant stops?
Begin with a municipal information task that does not control an essential service, using synthetic documents. Map read, send and write permissions; then rehearse model unavailability with frontline staff. A hospital case would require its own clinical and continuity approval.
Organizing a discussion on this question?
Use these findings and sources as the starting point for a panel, interview, email exchange or small roundtable.
What this means in practice
Day 1 connected competing views of AI risk to the ability to intervene. NIST’s voluntary profile supplies an operational structure: rehearse incidents involving a supplier or another external party, test fallback and assign deactivation responsibilities. CaMeL research adds a different kind of evidence. It studies how a security boundary can keep retrieved material from controlling an agent’s actions, within a defined threat model. A recommendation to have human oversight is therefore only the beginning. An operator must decide what the agent is permitted to read, send or change, and what remains outside its authority. Our proposed Greek case is a municipal information assistant that reads public documents but cannot alter a citizen’s official record. In a safe test environment, insert a passage written as an instruction into a mock document and check whether the application treats it as content. Separately disconnect the model and show how an employee completes the essential task. These tests address different failures. Passing a test of which actions are allowed says little about the accuracy of advice; a manual fallback on paper says little about how long recovery takes. The next discussion should bring the two records together, including the first failure the team cannot resolve.
What remains open: Voluntary guidance and a benchmark do not certify a service. Permission design, user approval fatigue and actual recovery still need local testing.
What the sources say (4)
Yoshua Bengio
The catastrophic risks of AI and a safer path
Bengio distinguishes useful intelligence from goal directed agency and proposes scientist AI as a route to safer systems. His argument challenges the assumption that more useful AI must also receive more autonomous control.
Position, paraphrased · 8 April 2025Read the original answer ↗Michalis Bletsas
Cyberattacks, critical infrastructure and national resilience
Bletsas treats resilience as a national and economic capability. Limited resources should protect the most consequential dependencies, with response and restoration planned before an attack.
Position, paraphrased · 4 November 2026Read the original answer ↗NIST
Which responsibilities and tests make intervention and recovery practical?
The voluntary profile recommends rehearsed incident response, fallback arrangements and defined responsibilities for disabling unsuitable systems. It is a governance resource, not evidence that a particular service is safe.
Position, paraphrased · July 2024Read the original answer ↗Debenedetti and colleagues
Can security boundaries prevent retrieved text from authorizing an agent’s actions?
CaMeL separates trusted instructions from untrusted material and constrains tool permissions. Its benchmark results concern a specified threat model; they do not eliminate misleading content or demonstrate safety in every real deployment.
Position, paraphrased · 24 June 2025Read the original answer ↗Watch the original published conversations
The catastrophic risks of AI — and a safer path
ted.comOriginal recording and context ↗- 01A concrete case
Begin with a municipal information task that does not control an essential service, using synthetic documents. Map read, send and write permissions; then rehearse model unavailability with frontline staff. A hospital case would require its own clinical and continuity approval.
- 02Evidence that could change the answer
Service dependency map, shift roles, fallback record and measured recovery time from a rehearsal.
- 03A question we can answer
A new example can change the answer. What would it tell us to ask next?
WHO CAN HELP ANSWER THIS?
The people closest to the problem.
Service owner and frontline worker
Show the one task that must continue, the ordinary alternative and who can operate it on a normal shift.
Service dependency map, shift roles, fallback record and measured recovery time from a rehearsal.Security lead and application developer
Which read, send and write actions are technically prevented, even if external text asks for them?
Permission map, synthetic test case, action log and known untested boundaries.Affected user and independent reviewer
Could you spot and challenge misleading advice even when no unauthorised action occurred?
Plain language escalation route, a redacted example and an assigned response role.Each perspective can fill a different gap in what we know.
Follow the question across the network.
ThoughtLeaders.gr The attributed positions ↗AIStrategy.gr The question in this sector ↗EventChanger The conversation ↗Sources and what remains uncertain
Read the original work alongside the analysis. For the local case, we still need the evidence requested above.
- Yoshua BengioThe catastrophic risks of AI — and a safer path ↗8 April 2025 · bengio-ted-20250408
- Michalis BletsasInterview on cyberattacks and national resilience ↗4 November 2026 · bletsas-resilience-2026
- NISTArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile ↗July 2024 · nist-genai-profile-202407
- Debenedetti and colleaguesDefeating Prompt Injections by Design ↗24 June 2025 · camel-prompt-injection-20250624