Who should be involved?
Evaluating compliance effectiveness should be done by people who understand both the risk and the environment affecting that risk. While the risk manager will generally take responsibility for making sense of the story, it's good practice to involve other subject matter experts to help make sense of the information as a whole.
When it comes to the indicator analysis, the people that should be involved in bringing together a defensible story should include the relevant risk and strategy experts, as well as analysts who have an understanding of the environmental variables that can affect your risk area.
Bringing the story together
When the suite of indicators was originally chosen, a number of assumptions were made about what you'd expect to see if your strategies were effective. Now that you're ready to bring the results together, you should revisit these early assumptions to see whether the indicator results are as expected.
If the behaviour is in line with your earlier expectations, then determining the extent of your effectiveness should be relatively easy. However, if any indicators show a different result, you must objectively investigate the result to understand how it impacts on your conclusions about the extent of your effectiveness.
Where an indicator shows an unexpected result, explore whether:
- the result is significant in relation to the overall suite of indicators - if the indicator is weak its impact on the overall outcome may not be significant; if it carries a lot of weight within the suite, you need to understand how it affects the outcome
- any other indicators show similar results - if the result is supported by a range of other indicators, then it would be reasonable to conclude that the strategies have not been effective
- it is an unintended consequence - any unintended consequences must be explored in order to understand their impact on the risk and for planning future compliance strategies
- further analysis is necessary - you may need to do further analysis to understand the reason for the result; consider issues such as a flaw in the data that was not previously identified, or a defect in the original thinking.