Here are five main ways in which you can tell if people are using the new functionality or tool you’ve just given them. There are pros and cons to each method. For websites, analytics are easily available and you should choose the target metrics in advance, in agreement with the client. For M365 adoptions, there are some key measures which can be easily found and reported by anyone with admin access to the relevant place.
| Method | Detail | Pro | Con |
|---|---|---|---|
| 1) Look for yourself | Obtain an admin account and check the logs. | The data is the data. | Sometimes the client won’t give you access to the analytics. |
| 2) Listen for yourself | Non-covid, go walk the floor. Covid, lurk where people meet digitally and listen | Your presence stimulates questions | You’re only seeing a limited sub-set of users |
| 3) Ask a sub-set of the target group (focus group it) what has changed for them since you made the new functionality available | Invite a cross-section of the org to take part in a time-limited discussion on the change | With good sampling, this provides a statistically-valid organisation-wide picture | Needs management support, and organisation, as focus group should be in place before the change to create a good before and after analysis. Sometimes, individuals can dominate the discussion giving a misleading conclusion. |
| 4) Survey the target audience to test recall | Create a short questionnaire with a defined deadline and a reward for participation | It can create a vehicle for engagement, particularly with a reward for participation. If participation high, it’s a good source of numbers for your closure report. | This entirely depends on how you weight the feedback to account for the non-participation of many of your target audience. Remember the only people who respond will be the volunteers, or the ones with agendas and axes to grind which may pollute the feedback you collect. |
| 5) Look for outcomes linked to your outputs | Your business case should have set out the benefits you said would accrue from the change. These should have been set out at higher strategic level (outcome) as well as tactical level (output). The business case should have had target timescales in it. | This is what the textbooks say should happen. Depending on org, Internal Audit staff will run this benefits realisation process and future decisions will be made based on the review findings ie if benefits are over-claimed consistently, future benefit claims can be discounted in the analysis of business case proposals. | Often, you leave as soon as the project has been delivered, and the organisation doesn’t have the capacity to run a 6 month or annual benefit review. It drops off the radar for the next problem. |
