How to Measure Policy Advice

Finding the right KPIs to measure policy advice is a big challenge. But it’s not because policy is immeasurable. It’s because we don’t define policy success clearly enough.

Every government agency has to provide policy advice of some sort to their Ministers. Government policy, like any, should be well formed and well implemented so it can improve the Ministers’ decision making.

Australia’s Protective Security Policy Framework (PSPF), as an example, guides government agencies in how to protect their people and information and assets. It’s a complex policy to navigate and to implement. But it’s important to make it work. In most cases, if a policy was created, it was created because what it provides guidance on really matters.

Policy advice (PA) therefore matters, so it’s important to measure and important to continually improve. But where do you start?

Start with the customer of the policy advice.

For most government agencies, the most obvious direct customers of policy advice are Ministers. So many policy teams default to a measure of “Minister Satisfaction” to evidence the performance of their advice. But it’s too trivial.

There are other important customers of policy advice, since there are many users of the policy. For the PSPF, the customers include users in a large number of government agencies.

So a great place to start to measure something complex and intangible like policy advice is to list the customers that need that advice. You might even go a step further and find out what the most important attributes are for them, when they seek policy advice. Some will be obvious, some might be surprising:

• Timeliness of the advice
• Ability to understand the advice
• Ease of translating the advice into action
• The advice actually helping

An example of how to do this is a simple two-stage customer survey process that helps you measure what really matters to the customers of your service, in addition to measuring overall satisfaction.Get clear about the policy advice process.

Very few people have a clear flowchart of their business processes. Or if they do, it’s simply too convoluted to be insightful.

We want to understand what aspects of our policy advice should be improved, in order to better service those who need our advice. So we need a flowchart of the policy advice process that is as simple as possible but not trivial.

With the flow of steps as the backbone, we can flesh out a cause-effect process model with:

• the PA outcomes most important to customers (see above)
• the tangible or observable outputs that lead to those outcomes, like a report or a conversation or specific recommendations
• the important in-process results, that currently aren’t performing well enough to deliver the outputs and outcomes that matter, such as understanding the customer’s real challenge, or quickly co-ordinating the experts needed to provide a piece of advice
• and the inputs, or qualities of those inputs, that have a strong bearing on the in-process results that matter, which might include enough policy experts, or information about answers to all previous policy questions

(I do realise this list is back-to-front relative to what most people do, but I like to start with the end in mind. This is so I can isolate what really does matter, rather than lose my way in an overwhelmingly comprehensive map of everything that could matter.)For an example of how to map a process to identify the bits that matter most to customers, see this case study on customer-driven KPIs for a billing process.

Write result statements that describe how it should be.

Good measures are born from clear and measurable results statements. They’re kind of like goals, but we don’t worry about including targets and timeframes. We just want a statement that describes clearly what we want to have, as though we already have it.

We’d create results statements for each of the PA outcomes, outputs, in-process results and inputs that matter most. For example:

• Customers value their overall experience of PA
• Customers get their PA when they need it
• Customers can easily take action from the PA
• PA reports are easy for customers to understand
• Access to policy experts saves us time

The key to writing measurable results statements is language. Avoid weasel words, don’t mistake actions for results, and keep them simple.When you know what matters, and express it as results, you’re ready to measure.

Choosing measures is something people try to do way too soon. They ask “what are the KPIs for policy advice?” But they have no clear idea of what aspects of policy advice are worth measuring. They’re hoping for a quick fix, so they can say they’ve jumped through the KPI hoop.

If you really do want your policy advice to be better, stronger, faster, then you’ll want to follow a more deliberate approach to measuring it. If you take the time to understand the specific results that your customers want and your process needs to perform well at, then you’ll end up with the right measures.

Simply take those specific PA results, and put them through a measure design template:

• PA Overall Value: Average rating customers provide on a 10-point scale of the overall value of their PA experience with us.
• PA Timeliness: The average number of days past the date customers required policy advice that the PA was received by the customer.
• PA Actionability: Average rating customers provide on a 10-point scale of the ease with which they could take action from the PA they received.
• PA Clarity: The average number of minutes spent with customers to help them understand the written PA provided to them.
• PA Availability: The average number of days spent waiting for a policy expert to be available to assist with providing PA to customers.

To meaningfully measure complex, intangible things like PA, we must do the work to define success simply, specifically, and starting with our customers.

DISCUSSION:

Do you notice how the process for making PA easier to measure is also a process for making anything complex or intangible easier to measure?

About

Stacey Barr is a specialist in organisational performance measurement and creator of PuMP, the refreshingly practical, step-by-step performance measurement methodology designed to overcome people’s biggest struggles with KPIs and measures. Learn about the bad habits that cause these struggles, and how to stop them, by taking Stacey’s free online course “The 10 Secrets to KPI Success” at http://www.staceybarr.com/the10secretstokpisuccess.