How to evaluate and measure a service
In this workshop, Kate Tarling showed us how to to evaluate and measure a service that addresses user needs, real life experience and business goals, as well as other important aspects such as policy aims…
Kate opened by explaining why we measure and evaluate a service. Performance for a service involves a balance of requirements, but often everyone is focussed on a different ‘bit’ of a service, or might view a particular technology solution as the service, rather than thinking about the user or the ultimate outcome of the service.
She suggested that one way to think about evaluating the performance a service is to ask “how well does it do its job?” and “at what cost?”. This includes opportunity costs, not just financial costs. There are always tradeoffs and it is not always straight forward.
Kate shared a model for evaluating a service:
- The wider goal or impact: The intended result, overall
- The job of the service: What this service helps to do or make happen
- A service outcome: What a successful result would be for this particular service
- Service performance indicators: What would tell you that it worked overall, as well as its different parts. Kate recommended grouping indicators based on factors that would indicate effectiveness and efficiency.
Our first workshop activity looked at an electric bike/scooter rental service, and attendees used Kate’s framework to look at the job of the service, the service outcome and the service performance indicators we could use to assess the performance of the service.
We explored what was difficult about this exercise and what would be challenging about doing this with the organisations we work with.
Feedback included:
- Often organisations don’t have a single purpose, some of which may be in conflict
- The job of a service may depend on who we are thinking about as our user. Focussing on the external user of the service can help to keep us focussed
- It may be harder to do this when you are inside the service, rather than as an external person evaluating the service
- You may need to draw lines around a service that is part of a larger organisation. This can be a massive task in itself.
- It can be difficult to define what success looks like for some services. Ratios and proportions can be helpful ways to see how well something is working
- Knowing when to use this technique. Kate confirmed that you can use this technique to complement theory of change or impact map work.
In her closing remarks, Kate emphasised that some services are easier to measure or evaluate or others. As a rule of thumb, aim for a one sentence summary of what the service is for. It is really valuable to be the person bringing that clarity as it can allow us to take action, whilst focussing on the nuance sitting underneath can make it more difficult to take action.
Finally, she stressed that you need to work with others – don’t do this on your own (or if you do, don’t share it with anyone and then do it again with others). Bringing people together to get some cohesion is really valuable – even if you can’t get everyone to agree, as calling this out can be helpful.
About Kate
Kate works with large organisations to develop the conditions and practices for everyone to deliver better services by default. She does this through consulting and training, and regularly advises boards, executives and teams as founder of a services company. She previously held senior service leadership roles in government as well as the private sector, including design, operations and strategy. Kate is the author of the best-selling book The Service Organization and has spoken on service organisations as guest lecturer at Harvard University, UCL, the University of Toronto, as well as Google, the Estonian Government, and the British Institute for Government.