Indigenous Advancement Strategy 2017-18 Annual Evaluation Work Plan
The Annual Evaluation Work Plan for 2017-18 is a key component of the Indigenous Advancement Strategy (IAS) Evaluation Framework. This is part of a multi-year plan.
The IAS Evaluation Framework guides evaluations conducted by the Indigenous Affairs Group of the Department of the Prime Minister and Cabinet (PMC).
The Work Plan covers evaluations and reviews planned over the current year (including those already commenced). Some projects continue across multiple years, in order to examine significant questions about impact.
Introduction
The Annual Evaluation Work Plan for 2017-18 is a key component of the Indigenous Advancement Strategy (IAS) Evaluation Framework. This is part of a multi-year plan.
The IAS Evaluation Framework guides evaluations conducted by the Indigenous Affairs Group of the Department of the Prime Minister and Cabinet (PMC).
The Work Plan covers evaluations and reviews planned over the current year (including those already commenced). Some projects continue across multiple years, in order to examine significant questions about impact.
The plan also covers future priorities for evaluation activities over four years (2017-18 to 2020-21).
All evaluation reports or summaries (if ethical confidentiality concerns or commercial in confidence requirements trigger a restricted release) will be made available on the PMC website.
Evaluation capability is a prerequisite for conducting robust evaluation. Guided by the IAS Evaluation Framework, some effort under this Work Plan is also directed towards increasing capability.
This Work Plan has been endorsed by the PMC Evaluation Committee and was approved by the PMC Executive Board. [1]
Principles for identifying evaluation priorities
To support strategic planning of evaluation activity across the Indigenous Affairs Group, a list of priority areas for evaluation is identified each year. This list includes anticipated evaluation priorities for the current year and a list of future priority areas (in line with requirements under the Public Governance, Performance and Accountability Act 2013).
Identifying specific evaluation priorities over a short horizon and broad priority areas over the longer term allows the Indigenous Affairs Group to provide greater oversight around the allocation of resources required for evaluation activity across the Group.
The principles for identifying evaluation priorities are outlined in the IAS Evaluation Framework, and cover issues of significance, contribution and risk. Broadly, prioritisation of evaluation effort and resourcing is based on:
- size and reach of the program or activity, emphasising what matters to Indigenous Australians as well as policy makers
- strategic need, potential contribution and importance of evidence in relation to the policy cycle, and
- the level of ‘policy risk’ of the program or activity, indicated by gaps in the evidence base, and assessments from past evaluations or reviews of the program or activity.
The Work Plan is structured as follows:
Group 1. Strategic projects
Potential for substantial impact or cross-cutting evaluations, relatively higher risk and limited evidence base, or timely/time-sensitive (to inform decisions). These are higher priority evaluations, many of which are undertaken by the Indigenous Affairs Group’s Information and Evaluation Branch, (or in some cases, conducted at arms-length).
Group 2. Program evaluations
Activities that are considered relatively lower risk, or less time sensitive. The Indigenous Affairs Group undertakes a number of program evaluations and reviews each year. These projects are typically commissioned and led by program areas, with support or guidance from the Information and Evaluation Branch.
Group 3. Future topics
Potential priorities over 2017-18 to 2020-21. While evaluation priorities are identified over the next four years, priorities remain flexible and subject to updating. This flexibility recognises that circumstances change and assessments can vary as new information becomes available (including, information from ongoing monitoring and review, changes in government policy-settings, and information from audit activity).
Group 1. Strategic Projects
Strategic evaluations that are underway or expected to commence in 2017-18.
Cross-cutting
|
Description |
|---|
|
Evaluation capability development Strengthening Indigenous community, service provider and government capability to effectively conduct and utilise high quality evaluations, and undertake performance management. |
|
Data improvement projects Build the availability of data that can inform evaluations. Significant subjects will be identified, such as, adult literacy and language interpreting. |
|
Mental Health First Aid Training Implementation review and participatory outcome study of training which aims to provide the critical knowledge and skills required to help with mental health issues. |
|
Empowered Communities Early implementation review to be co‑designed with Empowered Communities. |
|
Strengthening Organisational Governance Policy Under the IAS, all organisations receiving grant funding of $500,000 (GST exclusive) or more in any single financial year are required to incorporate under Commonwealth legislation. Evaluation will aim to assess the effectiveness of this policy, including any benefits to organisations and management of Commonwealth funding, and implications of any application beyond the Indigenous affairs portfolio. |
|
Strengthening the system effectiveness of early childhood investments Using an evidence informed evaluation framework to identify priorities, including examining significant impacts and cross-cutting, multi-sectoral elements to support systems’ improvement. |
|
National Partnership Agreement on Remote Indigenous Housing and Remote Housing Strategy Development and implementation of an approach to cross-cutting impact evaluation, with a focus on examining the range of potential effects on communities, overcrowding and non-shelter outcomes in the areas of employment, health and education. |
|
Closing the Gap Qualitative analysis of attitudes and reflections on 10 years of the Closing the Gap initiative.
|
Priority areas by sector
|
Description |
|---|
|
Remote School Attendance Strategy Measuring the impact of the Strategy on school attendance rates through an analysis of quantitative data. |
|
School Enrolment and Attendance Measure A randomised controlled trial to examine the impact of the School Enrolment and Attendance Measure on school attendance rates. |
|
Community Development Program A multi-year impact evaluation using mixed methods to examine outcomes in the first two years, and explore how and why the program works in various contexts. |
Group 2. Program Evaluations and Reviews
Program evaluations and reviews that are underway or expected to commence in 2017-18.
Jobs, Land and Economy Program
|
Topics |
Project and description |
|---|---|
|
Economic |
Indigenous Cadetship Support Program This impact evaluation will examine the effectiveness of the Indigenous Cadetship Support program in supporting Indigenous Australians to complete their degree and move to ongoing employment. |
|
Aboriginal employment and training activity The review will examine employment outcomes for those participating in a tailored Aboriginal employment strategy and how these compare to outcomes for similar job seekers under other employment programs. | |
|
Indigenous Procurement Policy A review of the first year of the implementation of the Indigenous Procurement Policy. | |
|
Review of Regional Employment Targets under the Community Development Program The Regional Employment Target methodology will be reviewed and a tool for setting future targets will be developed. | |
|
Review of Indigenous Entrepreneurs Fund – Business Advisors Undertake an operational review to assess the effectiveness of the business advisor model and consider options beyond the initial 12-month pilot period. | |
|
Environment |
Indigenous Rangers and Protected Areas The development of a shared measurement and evaluation framework for the Indigenous Rangers and Indigenous Protected Areas programs, developed in collaboration with stakeholders, will support regular impact assessment and on-going adaptive management of the programs into the future. |
|
Indigenous Rangers Cost Analysis The cost analysis informs a consistent approach to funding future projects and informs funding guidelines to ensure consistency and fairness. | |
|
Indigenous Rangers and Protected Areas Fee for Service An impact analysis study assesses the impact of the Indigenous Rangers and Protected Areas programs as a catalyst for economic and employment outcomes through | |
|
Land |
Our North, Our Future: White Paper on Developing Northern Australia Development of evaluation strategies for three measures under the White Paper: Township leasing in the Northern Territory, Prescribed Bodies Corporate capacity building, and Pilot land reform projects in the north. This work will guide measurement, data collection and periodic evaluation over the long term, up to 20 years. |
|
Native Title Representative Bodies and Service Providers Review of five Native Title Representative Bodies and Service Providers covering performance, relative efficiency, and recent complaints made in relation to bodies being reviewed. | |
|
Role of Land Councils and Native Title Representative Bodies – post determination Review of the potential future role of land councils and Native Title Representative Bodies following the determination of land claims and native title claims. |
Children and Schooling Program
|
Topics |
Project and description |
|---|---|
|
Early childhood |
Stronger Communities for Children Program Theory-based evaluation and implementation review, assessing progress towards a range of outcomes. |
|
Aboriginal Families as First Educators Program A theory-based evaluation to examine what works, for whom, in what circumstances and why. Includes remote and non-remote communities. | |
|
Schooling |
Girls Academies Impact evaluation to assess what’s working, what isn’t, any unintended consequences and what could be improved. A review of the costs and benefit of the program to participants, schools and communities. |
|
Indigenous Scholarships An initial review of IAS funded secondary school scholarships aims to profile delivery models, demand, costs and outcomes. This will inform data improvement to support ongoing monitoring and impact evaluation. | |
|
Models of Studying Away from Home To assess the costs, impacts and benefits of different models for students studying away from home with a view to understanding what works for different students and whether there are models that, in the longer term, we could have a greater emphasis on supporting. |
Safety and Wellbeing Program
|
Topics |
Project and description |
|---|---|
|
Social and emotional wellbeing |
Link Up Services Program An evaluation will consider need and the impact of healing, wellbeing and resilience-building programs and connections to family, community and culture. The evaluation will also test approaches to assessing the longer-term individual and community impact of healing services and activities. |
|
National Indigenous Critical Response Service An implementation review will assess the effectiveness of the initiative in providing culturally appropriate support to Aboriginal and Torres Strait Islander people affected by suicide-related or other trauma. An efficiency review and impact study will also be considered, including for community development initiatives, which seek to strengthen community capacity to prevent and respond to suicide and other critical incidents. | |
|
Safety |
Testing the impact of different program models for improving community safety and justice outcomes This set of trial design projects includes: design and testing of a program model to enhance existing prisoner through-care services for Indigenous prisoners; and design and trial of a youth through-care program model for young Indigenous people exiting detention. |
|
Testing community safety and justice outcomes using Randomised Controlled Trials The evaluations will test the impact of activities to reducing offending, violence and improving community safety outcomes for Indigenous Australians. Projects include: prisoner through-care services in the NT, WA and Qld for Indigenous prisoners exiting the prison system; and activities aimed at improving the connection to therapeutic supports for children in the child protection system in VIC who have experienced trauma. | |
|
Community night patrols A difference-in-difference analysis is being conducted to test the impact of community night patrols in the Anangu Pitjantjatjara Yankunytjatjara Lands communities of Amata, Pukatja and Mimili on safety outcomes. | |
|
Family Violence Prevention Legal Services This outcome evaluation will use mixed methods and include a robust literature review. Where possible, it will include quantitative analysis of service data and tracking client outcomes. | |
|
Third Action Plan to reduce family violence – Indigenous specific measures An impact evaluation of this significant activity includes a program review and assessment using in-depth interviews and existing service provider data. Evaluation plans will be developed covering a minimum of five sites, using either randomised controlled trials or quasi-experimental evaluation designs. | |
|
Substance
|
Alcohol and Other Drug Treatment Services This impact evaluation is working with treatment services in NSW to prepare for impact evaluation, implement standardised data collection to support outcome measurement, and conduct impact evaluation using adaptive learning. |
|
Petrol Sniffing Longitudinal study examining the prevalence of petrol sniffing and other substance misuse in selected remote communities. |
|
Nutrition |
School Nutrition Program An impact evaluation examining whether the School Nutrition Program provides the dietary requirements of students in remote Indigenous schools, and as the delivery and sustainability of the program. |
|
Community Stores An examination of the capacity of a real-time store food sales monitoring system to provide information to decision-makers at all levels, including participating store boards and other relevant policy makers, covering key food and nutrient performance indicators. | |
|
Sugar Reduction Strategy for Remote Community Stores The evaluation will examine the impact of the Strategy overall and test targeted interventions being trialled in stores which aim to reduce sales of sugared products through remote community stores. | |
|
Disability |
National Disability Insurance Scheme - East Arnhem Land Co-design project A participatory explanatory evaluation will study the first 12 months of the implementation of the East Arnhem Land Co-design project using participatory action research. |
Culture and Capability Program/Remote Australia Strategies
|
Topics |
Project and description |
|---|---|
|
Culture |
Indigenous Broadcasting This evaluation will review Indigenous Broadcasting using Social Return on Investment (SROI) methodology. The SROI will provide insight into the impact Indigenous Broadcasting is having on stakeholders through stakeholder surveys and indicators, and will use financial proxies to value outcomes. |
|
Infrastructure |
Army Aboriginal Community Assistance Program This operational review initially examines effectiveness, sustainability and governance. It will also look at ways to leverage broader benefits to achieve better training, employment and economic development outcomes in communities where the program is implemented. |
|
Leadership |
National Aboriginal and Torres Strait Islander Leaders Program Pilot An operational review of the pilot program covering implementation and delivery and an assessment of outcomes for building leadership aspirations. |
Group 3. Future Topics
To assist with forward planning, possible future topics are identified each year. These future topics, which are outlined below, will be considered for possible evaluation activity over the next four years (forward estimates period).
Cross-cutting
|
Project and description |
|---|
|
Building evaluation capability Improve the capacity across the sector to do and use monitoring and evaluation, and access available evidence, including among service providers, government staff and in communities. |
|
System effectiveness Cross-cutting impact evaluation of the Indigenous Advancement Strategy (multi-year project). |
|
Gender Gender-sensitive monitoring and evaluation, considering economic security, leadership and safety across IAS activities. |
|
Drivers of change Examination of shared outcomes and the underlying drivers of change, for example by evaluating the impact of adult literacy programs in relation to inter-connected outcomes beyond the program (e.g. early childhood). |
|
Youth participation Examining youth engagement and achievement, building on existing evidence. |
|
Prison to work Evaluation of the Commonwealth Prison to Work Action Plan, or components therein, to determine the effectiveness of action to support Aboriginal and Torres Strait Islander people as they transition from incarceration into employment. |
|
Place-based approaches Evaluation of place-based or ‘community designed’ approaches and arrangements to support these approaches. This may include testing approaches to place-based evaluation, as well as conducting process and impact evaluations of the Regional Network led place-based design and practice project. |
|
Regional investment Regional Program Investment Reviews of Safety and Wellbeing, and Children and Schooling Programs. |
Jobs, Land and Economy Program
|
Project and description |
|---|
|
Employment services Evaluation of the new employment and participation model for remote Australia. |
|
Economic participation Evaluation of youth transition to work programs focusing on pre-employment training and mentoring for Indigenous participants and the Transition to Work Program. |
|
Business development Evaluation of the Indigenous Business Sector Strategy. |
Children and Schooling Program
|
Project and description |
|---|
|
Economic and social participation Evaluation of the impact of mentoring on youth engagement in high school education. |
|
Tertiary Evaluation of the Away from Base Program – supporting Indigenous VET and Higher Education distance students. Evaluation of higher education focusing on the Indigenous Student Success Program. |
Safety and Wellbeing Program
|
Project and description |
|---|
|
Community wellbeing Evaluation of the contribution of IAS sport and recreation activities to Indigenous community wellbeing. |
|
Social and emotional wellbeing Evaluation of the impact of social and emotional wellbeing support services and service delivery models to improve the lives of Indigenous people. |
|
Social and cultural determinants of health Investigation of the social and cultural determinants of health outcomes for Indigenous people. |
|
Safety Test and trial innovations in service models to divert high risk, youth and adult offenders from re-offending. Adult corrective services data collection, development and analysis to inform prisoner to work strategies. |
[1] Future Work Plans will be endorsed by an independent Indigenous Evaluation Committee.