Five-Year Departmental Evaluation Plan 2022-2023 to 2026-2027

2.1 Departmental mandate and vision

ISC's mandate is to work with First Nations, Inuit and Métis to:

• improve access to high-quality services
• improve well-being in Indigenous communities across Canada
• support Indigenous peoples in assuming control of the delivery of services at the pace and in the ways they choose

ISC envisions itself as being a trusted partner that respects Indigenous perspectives and supports First Nations, Inuit and Métis in controlling the delivery of high-quality services for healthy, prosperous and thriving communities.

The ISC mandate and vision, inspired by the Department of Indigenous Services Act,^{Footnote 2} and the Department's Strategic Plan, set the foundation for the evaluation function's strategic and operating context and priorities. ISC Evaluation has an important role, to act as a catalyst for meaningful change as the department and partners work together to reform and re-design programs for improved service delivery and gradual transfer of control of services to Indigenous partners. More broadly, the evaluation function aims to work collaboratively with Indigenous peoples and organizations to influence and shift ISC's internal, organizational culture in favour of becoming more centered on Indigenous knowledge systems and worldviews.

2.2 ISC Evaluation - Vision

The role of ISC Evaluation is to offer robust, timely and meaningful evidence to help shape and support the re-design of the Department’s policies and programs, and to inform and influence improved service delivery and the transfer of services to Indigenous partners. It strives to contribute to the Government of Canada’s broader commitments to advance reconciliation with Indigenous peoples.

In this context, the ISC evaluation function is transforming its practice to deliver on the following three primary objectives:

1. Co-develop, adapt, and implement evaluation methods and approaches with Indigenous peoples and partners that are more centered on Indigenous worldviews and knowledge systems; 
2. Explore and implement ways to support Indigenous evaluation capacity and advance Indigenous-led evaluation functions including the generation of tools that are supportive of Indigenous-led evaluations in the context of gradual transfer of services to Indigenous partners and peoples;
3. Deliver and manage core work on program evaluations and activities further to the TBS Policy on Results (2016) and the Financial Administration Act (FAA);

2.3 ISC Evaluation innovation projects and other priorities

ISC Evaluation is increasingly emphasizing models of co-development and co-creation with Indigenous partners in all evaluation projects. In the short- to medium-term, this includes ways to integrate Indigenous evaluation expertise, knowledge, world views and/or Indigenous capacity development at key points in evaluations (planning, methodology, data collections, and development of findings and recommendations). In the long-term, the ultimate goal is to support the establishment of Indigenous evaluation functions outside of government.

In general, the evaluation innovation projects have the following objectives:

• To support Indigenous evaluators, research institutes, think tanks, and/or organizations to advance the implementation of ISC program evaluations with a focus on service transfer.
• To support Indigenous organizations to strengthen their capacity to implement program evaluations, enhance and coordinate Indigenous policy and evaluation networks, and organize the Indigenous policy and evaluation field (e.g., conferences, a center of excellence, etc.)
• To partner with Indigenous evaluators, researchers, think tanks, and/or organizations to advance evaluations tools, guides, policies, strategies and other evaluative practices that integrate Indigenous knowledge and worldviews.

In this context, ISC Evaluation is supporting two multi-year evaluation innovation projects with Indigenous partners in 2022-2023. In addition, ISC Evaluation is hoping to expand its innovation work to include building the capacity to bring a stronger distinctions-based lens to evaluation work as well as expanding the ability to weave the invaluable insights from traditional Indigenous knowledge keepers directly into evaluation processes.

3.1 Status of evaluation completion in 2021-2022

Looking back on the previous year, two (2) evaluations were completed:

1. Evaluation of Individual Affairs Program
2. Evaluation of First Nations Solid Waste Management Initiative

Also, as of the end of the 2021-22 fiscal year, ISC Evaluation published the following:

• Evaluation of First Nations Solid Waste Management Initiative (February 2022)
• Evaluation of the Water and Wastewater On-Reserve Program (August 2021)
• Evaluation of the Elementary & Secondary Education Program (July 2021)
• Evaluation of the Post-Secondary Education Program (July 2021)
• Five-Year Departmental Evaluation Plan 2021-2022 to 2025-2026 (November 2021)

The following evaluation projects were launched in 2021-2022:

• Evaluation of Non-Insured Health Benefits (Supplementary Health Benefits Program)
• Evaluation of the Mental Wellness Program
• Evaluation of the Environmental Public Health Program
• Independent Third-Party Evaluation for ISC Reform
• Synthesis Report of Key Studies and Reviews of the FNCFS Program
• Evaluation of the Miawpukek First Nation Grant Agreement
• Evaluation of Housing On-Reserve
• Evaluation of Other Community Infrastructure and Activities Program
• Evaluation of the Indigenous Entrepreneurship and Business Development Program
• Evaluation of the Lands Management Sub-Programs

ISC Evaluation Reports, and Summary Reports, are available on the departmental website.

4.1 Scope of Work

One of the consequences of the creation of ISC was a broader scope of work and increased workload because of the additional programs for which the Department is now responsible. In 2017, the First Nations and Inuit Health Branch (FNIHB) was transferred from Health Canada to ISC. With this transfer, fourteen programs were moved to ISC. Further, with the ISC and CIRNAC enabling legislation in July 2019, five programs were transferred from CIRNAC to ISC. In total, 19 programs were added and the resulting impact to the ISC Evaluation workload was significant, approximately doubling the number of programs to be evaluated with no additional evaluation resources to carry out the evaluation work.

Additional challenges include the inherited backlog of several evaluations from the previous shared evaluation function between ISC and CIRNAC and the many recent budget announcements and investments that have expanded the scope of existing programs or introduced new funding streams. The COVID-19 pandemic has exacerbated the situation. All of these variables have resulted in ongoing and persistent challenges in completing the planned evaluations and have increased the number of evaluations in the backlog that are beyond the five-year review cycle. With the mandated emphasis on co-development and service transfer noted above, and the expanding scope of evaluations, the backlog can be expected to persist for a few years. This situation limits the flexibility of the Directorate to take on new evaluations that may be of interest to Senior Management as the Directorate is challenged to meet the expectations of co-development and innovation while also meeting the due dates of the mandatory evaluations it must conduct to be aligned with FAA requirements.

4.2 Program structures

Most departmental programs, representing a majority of spending, are delivered through partnerships with Indigenous communities and in very few cases federal-provincial or federal-territorial agreements are in place. However, other funding structures have been established that are expected to become increasingly common as the Department moves to transfer services to Indigenous entities. For example, the British Columbia Tripartite Health Governance program involves a joint agreement between the Province of British Columbia, the First Nations Health Authority and ISC on health governance. Under this agreement, the parties agree to a tripartite evaluation, led by the First Nations Health Authority, with the final report being shared between the parties. In the future, more non-standard funding arrangements can be expected, with ISC Evaluation examining the 'readiness' of Indigenous governments or institutions to assume program delivery.

Currently, ISC has 30 authorities (voted grants and contributions) to support the 33 programs in its program inventory. However, a one-to-one relationship between the authorities listed in the Main Estimates and the Program Inventory of ongoing programs of grants and contributions deemed a 'Program' by the Department exists in very few instances. A number of programs draw on multiple grants and contributions; and conversely, there are grants and contributions that contribute to multiple programs. This Plan has been developed using the Treasury Board's Interim Guide on Results, which was updated most recently in August 2018, where there is the recognition that a 'program' of grants and contributions can cut across two or more programs in the Program Inventory and the relationship is not always one-to-one. This structure has presented the Department with some challenges in ensuring evaluation coverage for all authorities because of the segregation of a single authority over multiple programs with different evaluation timelines.

The program structure has also resulted in the number of evaluations exceeding the number of programs in the Program Inventory because of the distinct sub-programs that are included in the larger Programs. There are some sub-programs that have their own funding authority which alone supports the rationale for a separate evaluation that can identify improvements that might otherwise be lost in a larger evaluation. There are some sub-programs that are generally aligned with their Program, but when evaluators begin scoping the evaluation, the sub-program is quite unique and a single evaluation is not practical. One planned evaluation becomes two actual evaluation projects. This results in challenges meeting TB commitments and FAA requirements as evaluations must sometimes be deferred.

4.3 Evaluation planning methodology

The planning methodology is a multi-step process that begins with a review of the previous year's planning approach to update the steps where necessary and to review all relevant guidance and planning documents, such as the current Departmental Plan and Treasury Board commitments.

For 2022-2023, ISC refreshed its Departmental Results Framework (DRF) and its Program Inventory. Evaluation planning was conducted in alignment with the 2022-2023 framework. While there were no new programs added to the Program Inventory, there was an update to one program title where "Jordan's Principle" is now "Jordan's Principle and the Inuit Child First Initiative". Self-Determined Services has been removed from the Program Inventory and its activities are to be reported under the Education program.

The 2022-2023 Core Responsibilities for ISC are:

• Services and Benefits to Individuals – with 4 supporting programs
• Health and Social Services – with 14 supporting programs
• Governance and Community Development Services – with 13 supporting programs
• Indigenous Self-Determined Services – with 2 supporting programs

Appendix A presents the entire Program Inventory for ISC by core responsibility.

The Plan is also expected to align with the Department's Risk-Based Audit Plan, which is prepared by the Audit and Assurance Services Branch (AASB). The AASB is part of the Audit and Evaluation Sector of CIRNAC and it serves both CIRNAC and ISC. As has been past practice, consideration is given to timing audits and evaluations in a way that does not overburden programs.

Each evaluation included in the Plan has been further assessed as to the type of evaluation required based on the review of Financial Administration Act or Treasury Board requirements and stakeholder consultations, as well as noting whether it is a horizontal or non‑horizontal evaluation. It should be noted that in January 2018, the Treasury Board Secretariat released a new Guide to Departments on the Management and Reporting of Horizontal Initiatives. Under the guidance, ISC has no official Horizontal Initiatives, although it does participate in several led by other departments.

According to Section 42.1 of the Financial Administration Act, government departments and agencies must review, at least once every five years, the relevance and effectiveness of each ongoing grants and contributions program. Credit for evaluation coverage is granted upon deputy head approval of an evaluation report, under the premise that approval allows for the sharing of information in support of management action and the sharing of results with the public. Evaluation planning is conducted with that requirement in mind, reviewing the last fiscal year in which a program was evaluated, and scheduling completion and approval of the program evaluation five years later. Note that programs that are not funded through voted grants and contributions are not subject to the Financial Administration Act requirements.

However, as per the Policy on Results (Section 2.5), programs with five-year average actual expenditures of less than $5 million per year can be exempted from Section 42.1 and do not need to be evaluated every five years. Guidance from Treasury Board Secretariat states the five-year average is to be calculated on three years of actuals (Public Accounts) and two years of planned spending (Main Estimates). These calculations were conducted as part of the planning process and, based on these parameters, the following programs falls below the $5 million threshold:

ISC Evaluation conducted individual consultations in May 2022 with ISC senior management on the draft five-year schedule of evaluations to obtain their input on priorities for their respective sectors and on the timing of evaluations. Their input was synthesized and considered as part of the finalization process for the five-year schedule. The following senior management offices were consulted:

• Senior ADM Regional Operations
• Associate ADM Regional Operations
• Senior ADM First Nations and Inuit Health Branch
• ADM First Nations and Inuit Health Branch Regional Operations
• Chief Finances, Results and Delivery Officer
• ADM Child and Family Services Reform
• ADM Lands and Economic Development
• ADM Education and Social Development Program and Partnership
• ADM Strategic Policy and Partnerships Sector
• ADM Communications

4.4 Planned spending

Based on the Main Estimates, total planned spending for 2022-23 is outlined in the table below according to the Department's core responsibilities.

4.5 COVID-19 emergency funds

The Government of Canada began identifying funds in March 2020 specifically for COVID-19 support to Indigenous and northern communities and organizations. As of spring 2022, approximately $5.9B has been committed through ISC. ISC Evaluation has included, as a discrete line of enquiry, questions in recent evaluations to investigate the early impacts of COVID-19 on programming activities and will continue to canvass for this issue in future evaluations. Part of the March 2020 COVID-19 Economic Response Plan was the creation of an Indigenous Community Support Fund to help communities and organizations prevent, prepare and respond to COVID-19. Over $2.0B in support has been announced through the Indigenous Community Support Fund. New to the Evaluation Plan this year therefore is an evaluation of the ICSF in 2023-2024 to study the impact of the funding and gather lessons learned to inform future practices.

4.6 Coverage

Both of the following tables list 17 evaluations for the fiscal year 2022-23, as that year includes the ongoing Cluster Evaluation of Health Infrastructure Support for First Nations and Inuit. This cluster evaluation covers six of the Department's First Nations and Inuit Health Branch programs that were grouped together to take a holistic view of how effectively programs are working in concert to improve, or put in place, health infrastructure for Indigenous communities. Consultations with senior management indicated that the cluster evaluation should take a broad strategic approach in order to examine whether the mix of programs meets the current needs of Indigenous communities and the alignment of goals of both FNIHB and ISC, that it covers all health programs related to health infrastructure, and that it is in sync with the Department's mandate of reconciliation with Indigenous peoples in Canada. A cluster evaluation is also expected to alleviate the burden on program areas by evaluating their programs only once in this five-year evaluation period and to alleviate the backlog of evaluations the Department currently faces.

ISC covers 100% of its voted grants and contributions amounts. The out-of-scope Lands programming (Vote 10 is $5,178,309) is negligible.

4.7 Resources

The Evaluation Directorate has the following planned expenditures for 2022-2023 to deliver on the commitments made in the Plan.

4.8 Challenges

As noted, ISC Evaluation has an important opportunity to play as a catalyst for meaningful change and to work collaboratively with Indigenous peoples and organizations. This opportunity does not come without challenges and has created a very fluid operating environment for the Department in general and the evaluation function in particular.

Evaluations are becoming increasingly complex and are taking longer to conduct. With the increased focus on co-development and collaboration, time is required to build the necessary relationships with Indigenous partners and departmental program officials. To be effective, partners must be included in the process from scoping, developing Terms of Reference, establishing appropriate methodologies and conducting field work. Consultations with Indigenous partners may result in new approaches that might be incorporated as the evaluation methodology develops and evolves. While the Department has 33 programs in its inventory, there are currently 54 evaluations on the Plan due to new authorities, sub-programs and horizontal evaluations, a figure which has been increasing since the creation of the Department in 2019.

The scope of each evaluation has been expanding and will likely continue to do so. In addition to the standard Financial Administration Act (FAA) and Policy on Results questions regarding relevance, effectiveness and efficiency, evaluations must now also include questions canvassing for program impacts on families and children, the application of GBA Plus, whether services provided are culturally appropriate and of high quality, the early impacts of the COVID-19 pandemic on service delivery and the affects of climate change. In addition, there are four mandatory questions focused on service transfer and readiness of partners to assume control. These additional areas of investigation add to the time taken to develop and refine questions, gather data, analyse results and synthesize findings.

There are internal challenges for the evaluation function. Timelines for evaluations are lengthening due to more transparent and rigorous procurement practices that have added months to the process of putting contracts into place. There continues to be challenges related to the lack of data, the inconsistency of existing data, and gaps in performance information. Negotiations with ISC Program officials involved in program evaluation can take time to develop to the necessary level of co-operation and understanding. Internal and external capacity issues, particularly skilled evaluators, further exacerbate timelines necessary for pursuing collaborative approaches.

The increasing complexity of evaluations, and current internal challenges, has resulted in the Department developing a backlog of evaluations that are overdue with respect to the five-year evaluation cycle articulated in the FAA. In parallel with this, the evaluation function has developed a "backlog" strategy to map out a path towards completing a number of evaluations that are overdue. The strategy is to adopt a risk-based and balanced approach. For example, depending on priorities and risks, some evaluations will be put on an accelerated schedule, or a 'clustering' approach that groups evaluations together to provide a more system-wide analysis. Other evaluations may be more suited to integrated, co-developed and partnership-based approaches. It is recognized that co-developed evaluations will take longer in order to allow time to build trust and relationships and to integrate Indigenous worldviews and perspectives, ensuring engagement at all stages of the evaluation project. With the Department of Indigenous Services Act compelling the Department to focus on co-operation, partnership and readiness for service transfer, it is likely that the backlog will persist for the next several years.

4.9 Implementing the plan

Evaluations conducted by ISC Evaluation are used to inform decision-making and reporting (such as the Departmental Plan, the Departmental Results Report), to assess progress made by programs in achieving expected results and to incorporate lessons learned in order to improve program design and delivery. As noted, ISC Evaluation posts approved evaluation reports on its website in a timely manner, after sharing reports with Treasury Board. 

ISC Evaluation will work collaboratively with internal and external partners to tailor evaluation projects to meet the needs of decision-makers, with a growing emphasis on innovating and implementing with Indigenous partners novel methodologies and approaches. This work will set the foundation for meaningful and substantive improvements to service delivery and service transfer while at the same time advance culturally appropriate evaluation capacities and functions. This work remains within the flexibilities of the Treasury Board Secretariat Policy on Results.  

ISC Evaluation is moving towards developing and implementing protocols for engagement with Indigenous representative organizations so that First Nations, Inuit and Métis peoples can contribute their knowledge and experience throughout the evaluation process. The Department is also working to build internal culturally-responsive evaluation capacity, as well as to support external capacity in collaboration with Indigenous partners with a focus on service delivery.

The Directorate is committed to delivering the evaluation projects that have been identified for 2022-23. As noted, it is addressing its capacity challenges through the ongoing recruitment, development, and retention of talented staff, with an emphasis on Indigenous recruitment, and has developed a strategy to address the backlog of evaluation projects using a risk-based and strategic approach. 

The consequences of the COVID-19 pandemic continue to be felt by the Department as it prioritizes the health and safety of Indigenous communities and peoples. It is expected that the pandemic will continue to have an impact on the implementation of evaluation work in 2022-2023. However, recently evaluation teams are being invited to return to some communities. Should factors arise to compromise the delivery of evaluation projects, it will be brought to the attention of the SMC-PMEC in order that the senior management is fully informed.