The PlaybookAI Deployment

The UK Government Gave 1,000 Staff Copilot for Three Months. They used it once a day.

Satisfaction was high. Nothing measurable moved.

John TanNovember 5, 20265 min read

A VP of IT told Andy Berman, chief executive of Runlayer and formerly director of AI at Zapier, this month that his department is still on Copilot, ten months after the board told the CEO to go all in. It landed as a punchline. It is worth knowing what being on Copilot actually produced, because somebody measured it.

Almost everything written about AI at work is a survey. Somebody rolled a tool out, asked people whether they liked it, and published the percentage who said yes.

The UK Department for Business and Trade did something rarer. It gave a thousand civil servants Microsoft 365 Copilot for three months, measured what they actually did with it, and published the result with no commercial reason to flatter anyone.

Seventy-two actions each, in three months

The monitoring dashboard recorded an average of 72 Copilot actions per user across the whole pilot. There were 63 working days in it. That works out at 1.14 actions per person per day.

Around 64% of licence holders used it at least once in a given week. Only about 30% used it at least once per working day.

Now the number that would have been in the press release. 72% of respondents said they were satisfied or very satisfied, with a Net Promoter Score of 31, which is a good score.

72%

Satisfied or very satisfied with Copilot.

1.14

Copilot actions per user per working day.

30%

Used it at least once on a working day.

Those describe the same three months and the same thousand people. If you have ever been handed a satisfaction survey after an AI rollout and read it as evidence, this is the study that tells you what it was evidence of.

Figure 1. Department for Business and Trade, October to December 2024
High satisfaction, once a day.

1,000 licences · 63 working days

Satisfied or very satisfied 72%
Used it weekly 64%
Used it on a working day ~30%

1.14Copilot actions per user, per working day72 recorded actions across 63 working days

Figures from the department's own evaluation, published August 2025.

Their conclusion, with the caveat they attached to it

The report's own wording is worth quoting exactly, including the sentence that follows it, which most summaries drop:

We did not find robust evidence to suggest that time savings are leading to improved productivity. However, this was not a key aim of the evaluation and therefore limited data was collected to identify if time savings have led to productivity gains.

That caveat is real and it belongs in any honest reading. The department did not set out to measure productivity, so the absence of evidence is weaker than it first sounds.

The control group is the harder detail. Colleagues who did not receive Copilot had not observed any productivity improvement in the people who did. Nobody was looking for that result, and it does not depend on the study's instrumentation.

A horizontal bar chart headed Percentage of use cases reported by diary respondents. Transcribing or summarising a meeting leads at 16.43 per cent, then writing an email at 11.73 and summarising written communications at 10.74. Producing or editing slide decks is 5.69, data analysis 3.37, generating images 1.54, reviewing code 1.19 and scheduling last at 0.98 per cent.
What they used it for · Oct to Dec 2024 Summarising and transcribing lead at 16.43%. Scheduling, the one task that took longer with Copilot, is last at 0.98%.

The department's own chart, from its published evaluation (N = 1,424 reported use cases). Open Government Licence v3.0.

The tasks split, and not all one way

Underneath the headline numbers, the department watched people perform specific tasks with and without the tool. The results do not line up behind a single verdict.

Summarising reports was a real win and a large one. 12 minutes 37 against 41 minutes 34 without Copilot, and better accuracy with it, 4 against 2.5 out of 5. Written tasks in general showed the largest time savings and drew the highest satisfaction.

Excel went the wrong way on both axes at once. Copilot users completed data analysis more slowly, 25:01 against 20:33, and scored worse on accuracy and quality. Slides came out faster, 10:47 against 18:30, but scored 1 out of 5 on quality against 2, and 1.5 on accuracy against a clean 5. Writing emails saved thirteen seconds.

Scheduling and image generation took longer with Copilot than without it, either because the tool could not produce a usable output or because the task existed only because the tool did.

Figure 2. Four observed tasks, with and without Copilot
Faster is not the same as better.
TaskTime, with then withoutAcc / Qual
Summarising reports 12:3741:34 4.0 / 4.0
PowerPoint slides 10:4718:30 1.5 / 1.0
Data analysis in Excel 25:0120:33 1.5 / 1.5
Writing emails 07:3007:43 4.3 / 4.7
Observed task sessions, accuracy and quality scored 1 to 5. The department states this sample was small and asks that it be treated as supplementary.
Table 6 from the evaluation, headed results of observed task sessions. Four rows compare M365 Copilot users against non-users on average accuracy score, average quality score and average time taken. Data analysis in Excel reads 1.5 against 2.7, 1.5 against 2.7, 25 minutes 1 second against 20 minutes 33. PowerPoint reads 1.5 against 5, 1 against 2, 10 minutes 47 against 18 minutes 30. Summarising reports reads 4 against 2.5, 4 against 3, 12 minutes 37 against 41 minutes 34. Writing emails reads 4.3 against 4, 4.7 against 3.5, 7 minutes 30 against 7 minutes 43.
Table 6 · the source table Every number in the figure above, as the department published it. Accuracy and quality are scored 1 to 5.

Reproduced so you can check the figure against the source rather than take it on trust. Open Government Licence v3.0.

That caveat is the department's own and it matters. The 1.14 and the 72% come from monitoring data and a full survey. The task scores come from a small number of watched sessions. Treat the first pair as findings and the second as texture.

Two results that cut the other way

Copilot was significantly more satisfying for neurodiverse colleagues, a statistically significant difference rather than an anecdote, and non-native English speakers reported real benefits. Where the tool met an actual need, it met it.

That is the same finding as everything else here, seen from the other side. The gains were individual, they were genuine, and they stayed individual.

Which is why this reads as a verdict on how the rollout was authorised, not on Microsoft. A licence is the one move open to somebody who controls a budget and not the work, and the same pattern shows up across whole economies: 9% of firms with AI in production in the country leading Europe, and 1% of companies carrying 90% of the spend.

The two numbers worth having

If you are running a rollout you already have both of them, and neither is in your satisfaction survey. Actions per user per working day. And whether anyone outside the pilot can tell it happened.

The department collected both. That is what makes this the most useful three months of AI evidence anyone has published, a year after it ended.

Measure the actions.

Portrait of John Tan
John Tan ↗

Founder of nativefirst.ai. Runs everything on agents and writes down what works.

Sources
1Department for Business and Trade, Microsoft 365 Copilot evaluation, published August 2025 on GOV.UK. Pilot ran 1 October to 29 December 2024 with 1,000 licences, allocated roughly 70% to volunteers and 30% at random. Every figure in this post was read from that report directly rather than from a summary of it.
2Usage: an average of 72 Copilot actions per user across 63 working days, giving 1.14 actions per user per working day. The report notes this does not account for annual leave, sickness or part-time staff. 64% of licence holders were active in a given week; around 30% used it at least once per working day.
3Satisfaction: 72% of respondents satisfied or very satisfied, Net Promoter Score 31. Producing PowerPoints and scheduling drew the poorest satisfaction ratings. Neurodiverse colleagues were statistically significantly more satisfied than other users.
4Observed task sessions, scored 1 to 5: summarising reports 12:37 against 41:34, accuracy 4 against 2.5, quality 4 against 3. PowerPoint 10:47 against 18:30, accuracy 1.5 against 5, quality 1 against 2. Excel data analysis 25:01 against 20:33, accuracy and quality 1.5 against 2.7. Writing emails 07:30 against 07:43, accuracy 4.3 against 4, quality 4.7 against 3.5. The report states these sample sizes are small and asks that the results be treated as supplementary.
5Conclusion quoted verbatim including its caveat: no robust evidence that time savings led to improved productivity, but productivity was not a key aim of the evaluation and limited data was collected on it. Control group participants had not observed productivity improvements in colleagues taking part.

Read next

AI is moving really fast. Keep up.

Read the Playbook