DID TRAINING CHANGE HOW PEOPLE ACTUALLY WORK?
A FIVE-STEP CHECKLIST.
Imagine a management meeting; someone asks whether the agents know the new process. The answer: "Well, they saw it on a slide during a presentation." That's the gap this series is about. Being shown something is not the same as doing it differently at work.
Throughout the checklist, we follow one customer service team through all five steps, from that meeting to the final decision.
This is the condensed version for anyone who didn't follow along. It isn't a guide to designing training, and it doesn't tell you how to run your programmes. Every organisation knows its own problems better than anyone outside it. It covers something narrower: what your measurement needs in order to be believable, whatever your training is about. Most of it works with data you already collect.
The content of this page, the checklist and the backing case study was created and developed in collaboration with Laszlo Uray bringing his data analytics experience and expertise.
The sequence is logical:
Click on the steps or scroll down to see the explanations.
1. Define the Change
What should people do differently after the training, and where would that show up?
Start from the change you want to see at work, not from the course content. Then find where that behaviour already leaves a trace: a ticketing system, a CRM, call records, a quality checklist. Ideally it is recorded per person and goes back to before the training, so there is a "before" to compare with.
By the end of this step: find a behaviour, a place where it is recorded, and data from before the training.
Picture this: back in that management meeting, the team lead, Nora, is asked to turn a slide deck into a training programme on the team’s new way of handling cases: check the customer’s history first, work through the problem step by step, and confirm the fix before closing.
The easy option would be to judge the course by its end-of-course quiz. Instead, she asks a different question: if the training works, what will look different on an ordinary Tuesday?
The answer: fewer customers having to get back in touch about the same problem. The ticketing system already records that for every agent, going back months. Nobody had previously considered using this metric to measure training impact. Now it's the measure.
2. Map the Influences
What, apart from the training, could change this number?
Almost nothing at work changes for one reason only. Before rollout, list everything that could move the behaviour from Step 1 during the same period:
-
Inside the organisation: process or system changes, new tools, staffing changes and team moves, new targets or incentives, other training running at the same time.
-
Outside it: seasonal peaks, product launches, a change in the type of customers or requests coming in.
For each item, decide what you will do about it: check for it, leave the affected weeks out, or note it as a limit on what you can claim. This is also the moment to ask honestly whether training is the right fix at all. If the problem is a broken process, no course will solve it.
By the end of this step: you'll have a written list of other influences, and agreement on what you will not credit to the training.
Picture this: before the first session, Nora sits down with operations and the other team leads to list what else could change repeat contacts in the coming months. One item on this list is a ticketing system update which is planned mid-quarter. Another consideration is that the busy season is about to start, bringing more first-time customers. Next, there are three new joiners starting. And the final item on the list is that another team has planned a refresher course for Nora’s customer service agents.
They agree on what to do with each item on the list: watch the weeks around the system update closely, leave the new joiners out of the measurement, and move the refresher until after the measurement period. They also agree to monitor changes in customer demand during the busy season.
While going through the list, they notice something else: many repeat contacts come from customers chasing an approval handled by another department, which takes days. These customers come back regardless of how well the agent handled the call, so no training can change that number. Those cases are left out of the measurement.
3. Set the Rules
How will you compare, and what will count as success or "no effect"?
Now turn Steps 1 and 2 into rules, and write them down before anyone sees a result:
-
The measure: the behaviour from step 1, and exactly how it is calculated.
-
The time windows: how long "before" and "after" are, and when "after" starts.
-
Who is included: the same people in both periods. If two of the slowest performers leave and an experienced colleague joins, the team average improves without anyone getting better.
-
The comparison: if you can train people in waves, the group not yet trained shows what would have happened anyway.
-
Success and "no effect": what size or type of change would be meaningful enough to matter, and what would mean the training had no meaningful effect.
-
Support after the training: if you plan refreshers, checklists or coaching for the weeks after, decide them now. They are part of what you are measuring.
You don't need to analyse every individual to do this well. Keeping the same people in both periods is the minimum. Looking at how each person changed against their own history is the stronger version, and it is what makes some of the checks in Step 4 possible.
By the end of this step: a short, dated plan that anyone can hold the results against later.
Picture this: before rollout, Nora puts the impact measurement plan on one dated page, and her manager signs it off.
The measure: the share of cases where the same customer gets back in touch within a week regarding the same problem, leaving out the approval cases.
The time windows: the same number of weeks before and after.
Who is included: the same agents in both periods.
The comparison: the team is trained in three groups a few weeks apart, so the groups still waiting provide a useful comparison.
Success: a drop big enough that the team would feel it in their queue, agreed with operations.
“No effect”: a change no bigger than the normal week-to-week ups and downs.
Support: a short weekly huddle and a desk checklist for a month after each group’s training.
One more rule: if this measure disappoints, we don’t go looking for a better-looking one.
4. Test the Result
What changed, by how much, and does it survive the other explanations?
Measure the change using the rules from Step 3. Then, before celebrating, test it against your list from Step 2 and four uncomfortable questions:
-
Was performance already improving before the training?
-
Did the type of work change?
-
Is the whole effect coming from a few people?
-
Is the change bigger than normal week-to-week ups and downs?
Then look again later. A change that still holds weeks after the training means more than one that fades as old habits return.
By the end of this step: an honest assessment of how strongly the evidence supports a training effect; from no clear evidence through possible and likely to strong evidence.
Picture this: three months later, repeat contacts are down, and the first instinct is to celebrate. Instead, the team works through the questions.
Were repeat contacts already falling before the training? No, they had been flat for months.
Did the work change? Partly. The system update moved some of the simplest requests to an online form just as the second group was trained, so that group’s drop looks bigger than it really is. After accounting for this change, the second group still shows an improvement, although smaller than the initial figures suggested. The first and third groups weren’t affected, and they improved too.
Is it just a few people? In the third group, two agents account for most of the gain. That makes the third group’s result less convincing: it may say more about those two agents than about the training.
The improvements in the first two groups are also larger than the usual week-to-week fluctuations.
Most telling of all: while the first group improved, the groups not yet trained stayed flat.
Ten weeks later, the first group’s improvement is still holding. Someone notices that customer satisfaction scores look even better and suggests leading with those. The impact measurement plan from Step 3 says otherwise.
5. Make the Decision
What should we reinforce, improve, investigate or do differently next time?
The point of measuring is the next decision, not the report. How strong the evidence is from Step 4 shapes what you can decide:
-
Strong evidence of change: keep the course, reinforce the behaviour, and use what worked as a model for the next one.
-
Mixed results: find out where it worked and where it didn't (which people, which kinds of work), and target the remaining gaps.
-
No effect: improve or redesign the course, or look for an underlying problem that training can't solve.
-
Can't tell yet: say so, and decide what you would need to measure differently next time.
"No effect" is a result, not a failure. Knowing what didn't work can save the time and money of rebuilding or re-running something that was never going to deliver.
By the end of this step: a clear decision, and a better starting point for the next training.
Picture this: the verdict is likely evidence of a training effect for the first two groups (where the checks held up), and possible for the third (where two agents drove most of the gain).
The training taught three habits (Step 1). Looking at case reviews alongside the repeat-contact data, Nora notices that agents have become much better at confirming the fix before closing a case. However, checking the customer's history first has barely changed. When Nora asks the agents why, the answer is simple: the history screen takes too long to load in the middle of a call, so they skip it.
The team keeps the course and the weekly huddles, but takes the history screen problem to IT rather than trying to solve it through more training.
At the next management meeting, someone asks again whether the agents know the new process.
This time the answer isn’t “they saw it on a slide during a presentation”.
The answer is what changed, how sure we are, and what happens next.