# From impressive demos to a service people can rely on

FRONTIER THOUGHT · 2026-11-22



Editorial source review and next moderator brief



## What this round established

A customer-support study found a 15% average productivity gain, while METR’s early-2025 developer trial found a slowdown. METR’s February 2026 update warns that selection and concurrent agents make newer estimates unreliable. The practical comparison must include complete cases, checking and omitted work.



## Comparative analysis

Day 1 asked who benefits when AI changes work. The workplace studies now make that question testable. Brynjolfsson, Li and Raymond find a 15% average increase in resolved customer-support issues per hour, with gains varying by experience. METR’s early-2025 developer trial records a slowdown in a very different task setting. These findings cannot be averaged into a forecast for a Greek organisation. They show why the workflow, worker and quality standard belong in the claim. METR’s February 2026 update adds another complication: the newer study is distorted by who participates and which tasks people submit. Our editorial conclusion is to evaluate the whole queue, not a collection of impressive completions. A municipal service could compare similar requests, record every attempted case and count verification, correction, repeat contact and escalation. A fast first draft may be valuable, but it has not improved the service if the citizen returns twice or the employee spends the saved time repairing it. Start with a bounded, reversible trial. Decide beforehand which failures require a pause and how excluded or abandoned cases will appear in the record.



## What remains open

These settings and tools do not predict a Greek service’s result. No local trial or universal productivity effect is established.



## The next question

Can this service resolve more requests without shifting mistakes or extra work onto the people it serves?



## Published answers already available



### Dario Amodei

We are near the end of the exponential · 2026-02-13

https://www.dwarkesh.com/p/dario-amodei-2

Model capability and deployment move on different curves. Even unusually fast adoption is constrained by enterprise rollouts and organisational decisions.



### Satya Nadella

How Microsoft is preparing for AGI · 2025-11-12

https://www.dwarkesh.com/p/satya-nadella-2

Agents need identities, observability, storage and tools. Infrastructure should support several model families because architecture and supplier advantages can change.



### Brynjolfsson, Li and Raymond

Generative AI at Work · 2025-02-04

https://academic.oup.com/qje/article/140/2/889/7990658

In one customer-support firm, AI assistance increased resolved issues per hour by 15% on average; effects differed by experience. This is evidence about a particular workflow, not a universal productivity forecast.



### METR research team

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity · 2025-07-10

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

A randomized trial found that 16 experienced developers took 19% longer on their own repository tasks with early-2025 AI tools. It is a dated, narrow study; METR subsequently published a methodological update.



### METR research team

We are Changing our Developer Productivity Experiment Design · 2026-02-24

https://metr.org/blog/2026-02-24-uplift-update/

METR reports that selective participation, task selection and concurrent agents make its newer productivity estimates unreliable. A local test should record which work is omitted as well as which work is timed.



## The people and evidence for the next exchange



### Frontline employee and service owner

Show the entire case queue, including cases excluded, abandoned or sent to a colleague. Where did checking and repair time go?

De-identified case log, comparable baseline, total staff minutes, case type and reason for exclusion.



### Service user and accessibility representative

Could you complete the process, understand the reply and get a person to correct an error?

One concrete journey, repeat contacts, unresolved issues and an accessible human alternative.



### Independent evaluator

Would the benefit remain if difficult requests and all review time were included?

Sampling method, pre-agreed quality rubric, missing-case report and adverse-outcome record.



## Moderator handoff

Before Day 3, the moderator carries forward this editorial finding: time saved at one desk is not yet a service benefit. Invite the worker, a user and the service owner to define the same completion standard. Request one redacted case each, then compare who did the work and who received the gain. No results are assumed.



Record the contribution, what it changes, what remains uncertain and the next question. Preserve the earlier brief. Published source material is distinct from any new participant response.
