Book a call

Most AI time savings are self-reported, and the measured ones tell a different story

Government Copilot trials in Australia and the UK reported time saved by asking people. Where the work was timed against a comparison group, some tasks were faster, some were slower and worse. Without a baseline you cannot tell which you have.

Manish Kumar, founder · · 4 minute read

In 2024 the Australian Treasury gave Microsoft 365 Copilot to 218 staff for 14 weeks and then had the trial evaluated. The evaluation worked out how little the tool had to do to pay for itself.

13 minutes

a week is what a mid-level officer needed to move from low-value to high-value work to cover the licence. The evaluation could not say whether that happened.

Source: Australian Centre for Evaluation, Treasury Copilot trial evaluation, February 2025

Thirteen minutes a week is a low bar. The reason nobody could say whether it was cleared is in the evaluation's own list of limitations: the lack of a robust counterfactual against which changes in work could be assessed and attributed to Copilot, and a reliance on voluntary, self-reported data. Nothing had been timed before, so there was nothing to compare with after.

That is the usual state of AI evidence inside a business. It is not that the tool failed. It is that nobody can tell.

Ask people, and the answer is nearly always yes

The largest public trial so far is the UK Government's: about 20,000 civil servants with Copilot for three months in late 2024.

The headline was a saving of 26 minutes a day per user. The report is careful to say that the time savings are self-reported, and that it was not possible to identify how the time saved was spent.

Australian small business surveys read the same way. In MYOB's survey of 1,019 businesses in early 2026, 54 % of those using AI said it saved them time. The chart is labelled "perceived" time savings, which is the honest word for it.

Self-reports are not worthless. They tell you whether people like the tool and where they reach for it. They do not tell you how long the job takes now.

Time the work, and the answer splits

One UK department went further. The Department for Business and Trade ran the same kind of diary study as everyone else, then also set Copilot users and non-users the same tasks and watched.

What people reported

  • UK cross-government trial: 26 minutes a day saved per user, self-reported.
  • Department for Business and Trade diaries: summarising research saved a mean 0.8 hours per task.
  • Australian Treasury trial: participants answered surveys; time saved on administrative tasks was not measured.

What was observed against a comparison

  • Department for Business and Trade, Excel data analysis: Copilot users were slower, with worse quality and accuracy, than non-users.
  • Same study, PowerPoint slides: over 7 minutes faster on average, but worse quality and accuracy.
  • A customer support firm with 5,179 agents: 14 % more issues resolved per hour, and 34 % more for novice staff.

The department's conclusion was that it did not find evidence that time savings had led to improved productivity. Its observed-task samples were small, and it says so. The point is not that Copilot is bad at spreadsheets. It is that the diary and the stopwatch disagreed, and only one of them was measuring.

The customer support result comes from a study by Brynjolfsson, Li and Raymond, and it is worth noticing why it is so clean. The firm already counted issues resolved per hour for every agent, and the tool was rolled out in stages, so there was a before and an after and a group without it. The gain was real, it was large for the least experienced people, and it could be shown because the measurement existed before the tool did.

The largest self-report gap on record is from a randomised trial of experienced software developers in 2025. They expected AI to make them 24 % faster, took 19 % longer, and afterwards still believed they had been 20 % faster. The researchers have since said that result is out of date and that the tools have improved. The gap between what people felt and what the clock said is the part that travels.

What this looks like in an ordinary firm

The Reserve Bank asked medium and large Australian firms about AI in 2025. Two-thirds had adopted it in some form. Nearly 40 % said their use was minimal. Returns on investment were described as mixed, and few firms reported material productivity gains.

That is not the same as saying AI does not pay. It says most firms are early, use is shallow, and the gains that exist have not been shown. A firm that cannot show a gain cannot defend the spend at renewal, and cannot tell a workflow worth extending from one worth switching off.

A baseline is two weeks of counting

None of this needs a research team. For one repeatable job, before anything changes, write down four numbers.

  1. 01How often the job is doneA count per week, from the system that already records it: quotes sent, tickets closed, invoices matched.
  2. 02How long it takesTimed, not estimated, on a sample of at least twenty. People are poor judges of their own task time in both directions.
  3. 03How often it goes wrongRework, corrections and complaints. A job done faster and worse is not a saving, as the slide-writing result shows.
  4. 04Who does it and what an hour of their time costsSo that minutes turn into dollars, and you can see whether the gain is with senior or junior staff.

Then change one thing, run it for the same period, and count again the same way. If you can hold back a comparison, a team or a week without the tool, the result is stronger.

Two rules keep it honest. Decide before you start what result would make you stop. And report the result when it is small. A measured ten minutes a day is worth more than a reported hour, because you can build the next decision on it.

Sources

If you would rather hand this over: AI Workflow Sprint. Fixed scope · 4 weeks · from AU $6,900 ex GST. See the scope

Think you know the numbers? Guess eight published AI figures, then see each one with its source.

Keep reading

Staff under an AI ban break the rules more often, not less

In the largest study of AI at work, risky use was most common where the employer had banned generative AI. A ban removes the approved route and leaves the work. What holds is a short rule people have read and a tool they are allowed to use.

Read more about Staff under an AI ban break the rules more often, not less

One in four firms has named who is responsible for AI

The Australian Government’s Guidance for AI Adoption sets out six practices, and the first is deciding who is accountable. Most organisations have not done it. Here is what the guidance asks a firm of 10 to 200 people to put in writing.

Read more about One in four firms has named who is responsible for AI

Not sure where to start? Book a call.

Twenty minutes. We confirm fit, propose a fixed scope and a date, and tell you if you do not need us.

Book a call

+61 490 083 850 · Wollongong, NSW