3.4 Which indicators are feasible?
The following tests are useful in determining whether the indicators are feasible.
Consider the following data issues and identify any limitations or weaknesses.
- Source - Is the data available internally or externally? Is there an additional cost in setting up systems or purchasing the data?
- Timing - Is there a time lag before the data becomes available? Is the data available on a regular or infrequent basis? Is it available for all of the relevant time period?
- Currency - How current is the data?
- Reliability - Is the source of the data reliable? Is the source able to access or supply the type of data required? Does the source have a history of reliability? What is the level of bias in the source obtaining or supplying the data?
- Consistency - Have the data sources, or methods of data collection or analysis changed over time?
Use the characteristics in the SMART test for formulating and testing your indicators, which should be:
- Specific - clear and concise; not too broad. Indicators should have a clear, unambiguous definition so that people have a common understanding of the indicators. This will also clarify what data needs to be collected and help to ensure that the data collection is done consistently.
- Measurable - quantifiable, where possible. If not quantifiable, consider how else the results can be measured.
- Achievable - practical and reasonable.
- Relevant - ensure that the indicators actually measure the expected or desired change; avoid the temptation of choosing an indicator just because it is measurable and achievable. They must be relevant to users and stakeholders, as well as having a clear relationship to the success goals which define the desired outcomes. Well-aligned indicators will also ensure that meaningful data is collected.
- Timed - provide a realistic timeframe. Indicators need to produce data regularly enough to track progress and quickly enough for it to be useful, with only a short time between the period the data covers and when it becomes available.
Indicators should also have the following characteristics.
- Attributable - performance indicators should measure something that an organisation can reasonably be expected to influence. Changes in compliance behaviours should be attributed to your specific compliance strategies, rather than external factors such as a generic ATO marketing campaign or an increased consumer demand for goods and services.
- Comparable - continuity is necessary in order to make comparisons for trend analysis over time. While it is acceptable and unavoidable for performance information to change over time, changes should not be made to all indicators at the one time, as this prevents comparing performance over a number of years. If performance information is changed, the change should be justified and adequate links established to allow comparisons over time.
- Able to be substantiated - the indicator results and the conclusions drawn should be capable of being substantiated by an independent authority. Good record-keeping practices will facilitate this review.
- Unbiased - the information used to measure effectiveness should be impartially collected, analysed and evaluated. All reporting should provide a balanced, unprejudiced view.
You should also ensure that your selected set of indicators:
- can be supported by our business systems, are obtainable at a reasonable cost and can be implemented
- provide a balanced picture of performance over time (if there are significant concerns, the indicators may need to be discarded).
Last modified: 19 Jan 2015QC 21200
Once you have worked through phase 3, revisit the questions within this phase and in phases 1 and 2 to refine your answers and check their alignment with the success goals, desired outcomes and ATO intent.
End of attention