Skip to the findings

FRONTIER THOUGHT / FINDINGS AND NEXT STEPS

THE AI CHOICES

Better AI does not automatically mean better work or stronger learning. The useful test is what a person can do better, with responsibility and correction still clear.

The discussion so far22 November 2026

From positions to tests

THE QUESTION WE STARTED WITH

The day the system fails

WHAT THE SOURCES TELL US

NIST guidance recommends rehearsed fallback and clear deactivation responsibility. CaMeL research demonstrates a boundary between trusted instructions and untrusted content within a specified threat model. Test blocked actions separately from misleading advice and continuity when the model stops.

THE NEXT QUESTION

Can the operator prove which actions are blocked and how the service continues when the assistant stops?

Begin with a municipal information task that does not control an essential service, using synthetic documents. Map read, send and write permissions; then rehearse model unavailability with frontline staff. A hospital case would require its own clinical and continuity approval.

Organizing a discussion on this question?

Use these findings and sources as the starting point for a panel, interview, email exchange or small roundtable.

Coming next: A real AI gain: who gets the time back? ↓

What this means in practice

Day 1 connected competing views of AI risk to the ability to intervene. NIST’s voluntary profile supplies an operational structure: rehearse incidents involving a supplier or another external party, test fallback and assign deactivation responsibilities. CaMeL research adds a different kind of evidence. It studies how a security boundary can keep retrieved material from controlling an agent’s actions, within a defined threat model. A recommendation to have human oversight is therefore only the beginning. An operator must decide what the agent is permitted to read, send or change, and what remains outside its authority. Our proposed Greek case is a municipal information assistant that reads public documents but cannot alter a citizen’s official record. In a safe test environment, insert a passage written as an instruction into a mock document and check whether the application treats it as content. Separately disconnect the model and show how an employee completes the essential task. These tests address different failures. Passing a test of which actions are allowed says little about the accuracy of advice; a manual fallback on paper says little about how long recovery takes. The next discussion should bring the two records together, including the first failure the team cannot resolve.

What remains open: Voluntary guidance and a benchmark do not certify a service. Permission design, user approval fatigue and actual recovery still need local testing.

What the sources say (4)

Yoshua Bengio

The catastrophic risks of AI and a safer path

Bengio distinguishes useful intelligence from goal directed agency and proposes scientist AI as a route to safer systems. His argument challenges the assumption that more useful AI must also receive more autonomous control.

Position, paraphrased · 8 April 2025Read the original answer ↗

Michalis Bletsas

Cyberattacks, critical infrastructure and national resilience

Bletsas treats resilience as a national and economic capability. Limited resources should protect the most consequential dependencies, with response and restoration planned before an attack.

Position, paraphrased · 4 November 2026Read the original answer ↗

NIST

Which responsibilities and tests make intervention and recovery practical?

The voluntary profile recommends rehearsed incident response, fallback arrangements and defined responsibilities for disabling unsuitable systems. It is a governance resource, not evidence that a particular service is safe.

Position, paraphrased · July 2024Read the original answer ↗

Debenedetti and colleagues

Can security boundaries prevent retrieved text from authorizing an agent’s actions?

CaMeL separates trusted instructions from untrusted material and constrains tool permissions. Its benchmark results concern a specified threat model; they do not eliminate misleading content or demonstrate safety in every real deployment.

Position, paraphrased · 24 June 2025Read the original answer ↗
Download the questions and evidence ↓
Watch the original published conversations
Yoshua Bengio

The catastrophic risks of AI — and a safer path

ted.comOriginal recording and context ↗
HOW THIS QUESTION MOVES FORWARD
  1. 01A concrete case

    Begin with a municipal information task that does not control an essential service, using synthetic documents. Map read, send and write permissions; then rehearse model unavailability with frontline staff. A hospital case would require its own clinical and continuity approval.

  2. 02Evidence that could change the answer

    Service dependency map, shift roles, fallback record and measured recovery time from a rehearsal.

  3. 03A question we can answer

    A new example can change the answer. What would it tell us to ask next?

WHO CAN HELP ANSWER THIS?

The people closest to the problem.

Service owner and frontline worker

Show the one task that must continue, the ordinary alternative and who can operate it on a normal shift.

Service dependency map, shift roles, fallback record and measured recovery time from a rehearsal.

Security lead and application developer

Which read, send and write actions are technically prevented, even if external text asks for them?

Permission map, synthetic test case, action log and known untested boundaries.

Affected user and independent reviewer

Could you spot and challenge misleading advice even when no unauthorised action occurred?

Plain language escalation route, a redacted example and an assigned response role.

Each perspective can fill a different gap in what we know.

Sources and what remains uncertain

Read the original work alongside the analysis. For the local case, we still need the evidence requested above.

  1. Yoshua BengioThe catastrophic risks of AI — and a safer path ↗8 April 2025 · bengio-ted-20250408
  2. Michalis BletsasInterview on cyberattacks and national resilience ↗4 November 2026 · bletsas-resilience-2026
  3. NISTArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile ↗July 2024 · nist-genai-profile-202407
  4. Debenedetti and colleaguesDefeating Prompt Injections by Design ↗24 June 2025 · camel-prompt-injection-20250624

NEXT DISCUSSION

23 November 2026 · 09:00 Athens time

A real AI gain: who gets the time back?

Before expanding one Greek AI service, what evidence from complete cases would show a benefit after checking and correction, and who should receive the saved time?

WHERE WE BEGIN

The model may improve faster than the organisation. Compare Amodei on adoption, Nadella on the operating environment and Mitsotakis on visible benefits to people, using the workplace studies as the evidence test.

THE PRACTICAL OUTCOME

One page comparing the service: complete case time, correction work, user completion, omitted cases, benefit recipient and a clear rule for deciding whether to stop or expand.

The published perspectives behind this question

A WORKING BRIEF, READY TO EDIT

Make this the next discussion.

The finding, original sources and evidence requests come with you. Review the question and choose the people who can clarify it.

Saved briefs are visible to authorized site editors. Use public material or material approved for that workspace.

Organizer sign in to save ↗