How to tell whether an automation actually worked
By Sam Kim · · 6 min read

How do I know whether an automation is actually saving my business time?
Measure the task before the automation goes live, then measure it the same way afterwards. Track three numbers: how long the task takes your people each week, how often the result needs a person to fix it, and the outcome the task exists to move, such as reply time or invoices paid on time. Decide in advance what result would count as a failure, so the verdict comes from the numbers rather than from whoever is most enthusiastic.
Why does "it feels faster" mislead?
Because the work you stopped doing is invisible and the work the automation creates is not. Nobody notices the forty reminder texts they didn't send this week. Everybody notices the one that went to the wrong client.
Two other things skew the feeling. The first few weeks of anything new are slower, because people are learning it and checking it twice, so early impressions are often worse than the steady state. And whoever pushed for the change has a stake in it working, which colours how they describe it, usually without meaning to.
None of this is a reason to distrust your team. It is a reason to write the numbers down.
What should you measure before it goes live?
The task as it runs today, for long enough to see a normal week. Two weeks is usually plenty for daily work. For anything monthly, use the last two or three cycles from your records.
Keep the baseline simple enough that people actually do it:
- Who does the task, and roughly how many times a week.
- How long each one takes, including the time spent finding things and getting back to what they were doing.
- How often it goes wrong: a missed follow-up, a wrong figure, a client who had to chase.
- The result it exists to produce, measured the way you already would. Reply time to enquiries, days until invoices are paid, no-shows per week, quotes sent within a day.
A tally in a shared note is fine. The point is that the same person counts the same thing the same way before and after. If you have never put a number on the time itself, working out what admin hours cost is a good place to start.
Which three numbers are worth tracking afterwards?
The same three you set up in the baseline. Everything else is detail.
- Time per week on the task. Hours your people spend on it now, including checking and correcting the automation's output. This is the number that most often disappoints, because checking time is real and people forget to count it.
- Rework rate. Out of every ten items the automation handles, how many need a person to fix, redo or apologise for. A low time number with a high rework number means the work has moved, not gone.
- The outcome number. The thing the task is for. An automation that halves admin time but makes reply times slower has not worked, however efficient it looks from the inside.
Read the three together. Time down, rework steady and outcome better is a clear win. Time down and outcome worse is a trade you need to decide on deliberately, not drift into.
What counts as an honest failure?
Whatever you said would count before you started. That's the whole trick. Write two or three failure conditions down when the automation goes live, while nobody has a stake in the answer yet.
Reasonable ones look like this:
- After four weeks, time on the task has not dropped by enough to cover the checking.
- Rework is higher than the error rate you had doing it by hand.
- The outcome number got worse and stayed worse past the learning period.
- People have quietly gone back to the old way. If the team is routing around it, the numbers on paper don't matter. Getting a team to use a new system covers why this happens and what to do about it.
When one of these trips, the usual answer is to fix it rather than scrap it. Most failures trace back to a step that was never written down, a handover the automation can't see, or an edge case that turns up weekly rather than yearly. Sometimes the honest answer is that the task should stay with a person. That is a fine result. It is far cheaper to learn it at four weeks than at twelve months.
Where does a person stay in the loop?
On reading the numbers and making the call. An automation can count what it did, and most tools will report how many items they processed. What they won't tell you is whether the output was any good, whether a client was annoyed, or whether the task was worth doing at all.
So the rework count needs a human. Someone has to look at a sample of what went out and judge it, especially in the first month. Pick ten items a week at random and check them properly. It takes a few minutes and it is the only way to catch the confident, plausible errors that AI tools are prone to.
Keep watching after the verdict, too. Automations degrade quietly when a form changes, a supplier changes their emails or a staff member leaves. A monthly glance at the same three numbers will show the drift before a client does, and what to do when one breaks picks up from there.
How do you work out whether it paid for itself?
With your own figures, not anyone else's averages. Four lines:
- Hours saved per week, from the time number, after subtracting checking time.
- Multiply by what an hour of that person's time costs you, including on-costs.
- Subtract the tool's ongoing cost per week. Check its current pricing rather than relying on what you were quoted at sign-up.
- Divide the one-off setup cost by that weekly figure. That is how many weeks it takes to pay back.
If the answer is a few months, good. If it is several years, or the weekly figure is negative, the automation is costing you money however pleasant it is to use. Keep the setup cost separate throughout, so a strong automation isn't judged harshly in its first month.
What to do next
- Pick the next automation you plan to introduce, or the last one you did.
- Write down the three numbers for the task as it runs now, or reconstruct them from records.
- Agree two or three failure conditions with whoever owns the task, and put a four-week review in the calendar.
- Assign someone to check ten random items a week for the first month.
- At the review, read the three numbers together and make a call: keep, fix or hand the task back to a person.
General information, not legal, tax or financial advice · Drafted with AI tools to my written rules · Terms