Five-Year Departmental Evaluation Plan 2021-2022 to 2025-2026
June 2021
ISC Evaluation
PDF Version (431 KB, 25 Pages)
Table of contents
List of Acronyms and Abbreviations
ADM |
Assistant Deputy Minister |
---|---|
CFRDO |
Chief Finance, Results, and Delivery Officer |
CFSR |
Child and Family Services Reform |
CIRNAC |
Crown-Indigenous Relations and Northern Affairs Canada |
ESDPP |
Education and Social Development Programs and Partnerships |
FAA |
Financial Administration Act |
FNIHB |
First Nations and Inuit Health Branch |
Gs & Cs |
Grants and Contributions |
ISC |
Indigenous Services Canada |
PMEC |
Performance Measurement and Evaluation Committee |
RO |
Regional Operations |
SMC |
Senior Management Committee |
SPP |
Strategic Policy and Partnerships |
TBS |
Treasury Board of Canada Secretariat |
Deputy Head Confirmation Note
I approve the Departmental Evaluation Plan for Indigenous Services Canada for the fiscal years 2021-22 to 2025-26, which I submit to the Treasury Board of Canada Secretariat as required by the Policy on Results.
I confirm that the following evaluation coverage requirements are met and reflected in this five-year plan:
- Plans for evaluation of all ongoing programs of grants and contributions with five‑year average actual expenditures of $5 million or greater per year;
- Meets the requirements of the Mandatory Procedures for Evaluation; and
- Supports the requirements of the expenditure management system, including, as applicable, Memoranda to Cabinet, Treasury Board Submissions, and resource alignment reviews.
I will ensure that this plan is updated annually and will provide information about its implementation to the Treasury Board of Canada Secretariat, as required.
Deputy Minister
Indigenous Services Canada
1. Introduction
This document presents the Five-Year Departmental Evaluation Plan ("the Plan") for Indigenous Services Canada (ISC) covering fiscal years 2021-22 to 2025-26. The development of this Plan adheres to the Treasury Board of Canada Secretariat (TBS) Policy on Results (2016) under which the federal evaluation function is called upon to provide evidence that helps the Government of Canada demonstrate that its spending contributes to results that matter to Canadians. This five-year rolling Plan has been informed by an annual planning exercise that identifies the timing of the individual evaluations.
The primary purpose of the Plan is to help the Deputy Minister ensure that credible, timely and neutral information on the ongoing relevance and performance of planned spending and ongoing programs of grants and contributions is available to support evidence‑based decision-making.
The Plan also:
- Provides an opportunity to align evaluations to ensure that the information needs of the Department and other evaluation users (e.g., Indigenous partners and other government departments) are being met;
- Helps ensure that evaluations supporting program redesign and transformation are planned and completed in advance of shifting priorities and program renewal dates;
- Outlines the evaluation innovation work in the context of service transfer and efforts to build internal and external evaluation capacity;
- Initiates regular communication and consensus-building on evaluation needs and priorities across the Department;
- Provides central agencies with advanced notice of when evaluations will be available to inform their work (e.g., in support of Memoranda to Cabinet, Treasury Board Submissions, etc.);
- Allows departmental units responsible for the development of the Departmental Plan and the Departmental Results Reports, as well as other groups engaged in strategic planning and reporting activities, to identify when evaluations will be available to inform their work; and
- Ensures transparency with partners and the general public regarding availability of evaluation results, planned evaluations and program spending.
Moreover, the Plan serves as an important tool for the Department's Head of Evaluation and ISC Evaluation to manage project workflow and plan the activities of its human and financial resources.
2. Planning Context
2.1 Departmental transition
In August 2017, the Prime Minister announced the dissolution of Indigenous and Northern Affairs Canada (INAC), and the creation of two new departments, ISC and Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC), designed to better meet the needs of the people they serve, to accelerate self-determination and the closing of socio‑economic gaps, and to advance reconciliation. ISC was created to ensure a consistent, distinctions-based approach to the delivery of services to Indigenous peoples. ISC believes that transferring control to Indigenous partners is the best way to improve services and support their inherent right to self-determination with flexible approaches that are inclusive and responsive to the diversity of Indigenous peoples.
2.2 Departmental mandate and vision
ISC's mandate is to work collaboratively with partners to improve access to high quality services for First Nations, Inuit and Métis. Its vision is to support and empower Indigenous peoples to independently deliver services and address the socio-economic conditions in their communities.
2.3 Strategic Policy and Partnerships Sector (SPP)
SPP sector supports the Department's legislative mandate by advancing, in an integrated, co-developed, sustainable and evidence-based way, the improvement and transfer of Indigenous services to Indigenous peoples, for Indigenous peoples, by Indigenous peoples. SPP Sector will serve as an internal driver of change by supporting sectors in improving services and enabling Indigenous control.
2.4 Evaluation and Policy Re-Design Branch
The role of the Branch is to lead horizontal policy integration to advance departmental priorities, identify innovative policy interventions and partnerships to improve and transfer services, evaluate the success of programs in a neutral manner and support the advancement of Indigenous evaluation capacities and functions.
The Branch further articulates its role to identify forward-looking, innovative, cross-cutting, horizontal policy interventions to support ISC sectors, other government departments and Indigenous partners in accelerating the transfer of Indigenous services, and leveraging partnerships to advance and evaluate progress on Indigenous-centric service design and delivery.
2.5 Evaluation Directorate
Aligned with the departmental and sector visions, the role of ISC Evaluation is to offer robust, timely and meaningful evidence to help shape and support the re-design of the Department's policies and programs, and to inform and influence improved service delivery and the transfer of services to Indigenous partners.
In this context, the evaluation function is anchored in three main areas:
- Delivering and managing core work on program evaluations and activities in accordance with the TBS Policy on Results (2016) and the Financial Administration Act (FAA);
- Discovering and implementing innovative evaluation methods and approaches with Indigenous partners that specifically support service transfer; and
- Exploring and implementing ways to support Indigenous evaluation capacity and advance Indigenous-led evaluation functions. This work is crosscutting to (1) and (2) above and underpins all of the evaluation work in this 5-year evaluation plan.
ISC Evaluation has undertaken a great deal of transformational work with Indigenous partners in 2020-2021 on evaluation innovation projects and on other foundational pieces that are priorities in fulfilling its role.
2.6 ISC Evaluation innovation projects and other priorities
ISC Evaluation is increasingly emphasizing models of co-development and co-creation with Indigenous partners in all of its evaluation projects. In the short- to medium-term, this includes ways to integrate Indigenous evaluation expertise, knowledge, world views and/or Indigenous capacity development at key points in evaluations (planning, methodology, data collections, and development of findings and recommendations). In the long-term, the ultimate goal is to support the establishment of Indigenous evaluation functions outside of government.
In general, the evaluation innovation projects have the following objectives:
- To support Indigenous evaluators, research institutes, think tanks, and/or organizations to advance the implementation of ISC program evaluations with a focus on service transfer.
- To support Indigenous organizations to strengthen their capacity to implement program evaluations, enhance and coordinate Indigenous policy and evaluation networks, and organize the Indigenous policy and evaluation field (e.g., conferences, a center of excellence, etc.)
- To partner with Indigenous evaluators, researchers, think tanks, and/or organizations to advance evaluations tools, guides, policies, strategies and other evaluative practices that integrate Indigenous knowledge and worldviews.
In this context, ISC Evaluation began to support two multi-year evaluation innovation projects with Indigenous partners in 2020-2021:
1. Centering Indigenous Worldviews within Evaluation Frameworks
Commonly used methodologies for program evaluations are often rooted in Westernized, positivist perspectives that do not incorporate the diversity and vibrancy of Indigenous cultures, worldviews and knowledge systems. These types of evaluation frameworks have not taken into account Indigenous worldviews when evaluating federal programming, policy and investment. Evaluation frameworks today need to focus on the importance of capacity strengthening, developing relationships based on respect, trust, humility and reciprocity, and the historical, socio-economic conditions and distinction of Nations. The Department is supporting exploratory research for an evaluation project called Centering Indigenous Worldviews within Evaluation Frameworks, which is a partnership between ISC Evaluation and the Indigenomics Institute. Previously focusing more squarely on the value-for-money paradigm, this project has expanded to emphasize overall community-determined well-being and the primacy of relationships, co-designed processes, and multiple mandates. The project is rooted in a dynamic theory of living that considers the complexity of historical, current and evolving relationships and mandates with Indigenous Nations, and aims to generate a set of practical co-designed tools and methods for evaluation processes.
2. Strengthening Indigenous Evaluation Frameworks and Practice
ISC Evaluation has partnered with Johnston Research Inc. on an exploratory project entitled Strengthening Indigenous Evaluation Frameworks and Practice whichaims to build dynamic evaluation tools that are relevant and appropriate for ISC, Indigenous partners and communities. This project will advance a new framework that attempts to overcome the limitations of the colonial underpinnings of logic models and includes the expansive concepts of time. It is rooted in three key pillars (spirit, relationship and evaluation processes) which serve as a lens in which to ground evaluation questions, outcomes and indicators. The project will apply this exploratory framework to a small subset of ISC evaluations as a means of generating learning to improve and further refine the framework. In addition, the project will support the fostering of dialogue and knowledge transfer amongst a core group of Indigenous evaluators.
ISC Evaluation has also been focusing on the following additional priorities:
Results-based innovation and assessing readiness for service transfer
As stated in the legislated mandate, priorities, and corporate vision, the primary end goal of ISC is to transfer the control of services to Indigenous partners, organizations and communities. The overall readiness for transfer both for the Department and Indigenous partners, organizations and communities is a critical component in ensuring that the transfer is properly paced and leads to successful and sustainable outcomes.
This is also tied to the experimentation direction from TBS to Deputy Heads (2016) which requires that departments test new approaches and measure impacts to instill a culture of measurement, evaluation and innovation in program and policy design and delivery.
Work is underway to better understand concepts and measurement frameworks to assess and track readiness and progress towards service transfer. In 2020-2021, and under the Department's Service Transfer Strategy, ISC Evaluation, in collaboration with the Department's Chief Data Officer, is developing a multi-year research plan to develop and make recommendations on approaches to measure outcomes and report on progress, including readiness for transfer – both internally and externally. As a first step, a literature review was launched and will be one of many lines of evidence, including engagement with Indigenous partners, interviews with key informants and group discussions, that aim to identify and describe key measurement dimensions of readiness and progress of programs towards service transfer.
In the meantime, and until a more formalized measurement tool is developed, all evaluations are including high-level lines of enquiry on program readiness, evaluation by evaluation. Some preliminary results from this approach should be available in the coming months.
Evaluation projects related to reform of Child and Family Services
ISC will complete the evaluation of the First Nations Child and Family Services program by 2024-25 as well as undertake a Synthesis Report of studies on the program the help bridge the gap since the last departmental evaluation in 2014-15 (to be completed in 2021-22).
The evaluation work is expected to be both innovative and transformative. The projects are expected to be undertaken with Indigenous partners through co-developed evaluation frameworks that will include Indigenous participatory and culturally responsive research methodologies, as well as Indigenous concepts of child and family well-being.
Given the scope and complexity of the programing and services related to child and family services, including in response to Tribunal orders, the possibility of undertaking other evaluation projects is being explored and will be reflected in subsequent departmental evaluation plans moving forward. Considerable advanced planning to establish innovative co-developed evaluation frameworks with Indigenous partners will also be essential.
Ongoing engagement and Indigenous evaluation capacity
Engagement approaches that are considered and employed can vary by evaluation as some programs have well-established engagement protocols with national Indigenous organizations, regional Indigenous organizations, Indigenous technical/manager working groups, and communities. ISC Evaluation works with individual programs to determine best approaches for engagement while at the same time is exploring opportunities with Indigenous experts in the evaluation community and/or the private, academic and not-for-profit sectors.
ISC Evaluation is also exploring the adoption of distinctions-based evaluation approaches. We will be talking to national Indigenous organizations over the next year to explore potential partnership approaches that will help to increase evaluation capacity within Indigenous organizations.
A recent example of how evaluation projects are shifting is the evaluation of the First Nations Solid Waste Management Initiative. In this evaluation, an Evaluation Advisory Committee was established as a forum for shared decision-making around all major evaluation milestones. The committee included representatives from the major stakeholders, including Indigenous representation at the community level. In addition, an expert Indigenous evaluator was invited to be part of the consultancy firm engaged to carry out the evaluation. The process also included planning data collection early and jointly with representatives from the target communities, allowing community representatives to contribute to shaping the way data collection took place, planning for sufficient time in communities to ensure clear understanding of the purpose of the work being carried out and planning for opportunities to circle evaluation findings back to communities and conduct joint interpretation of evaluation findings. ISC Evaluation intends to take the lessons learned from this evaluation planning process to inform future work.
GBA Plus
GBA Plus provides a rigorous method for developing, implementing, monitoring and evaluating initiatives to provide equal outcomes, with an emphasis on inclusiveness and reaching those marginalized due to a combination of social norms and attitudes; institutional power structures; policies and practices that together can limit access to opportunities.
GBA Plus is an intersectional analysis – a framework for understanding how aspects of a person's identity combine to create different modes of discrimination and privilege. In doing GBA Plus, many factors should be considered. The needs and experiences of different groups of people are influenced by intersecting parts of their identity, the context they are in and their lived experiences. All federal government organizations are directed to apply GBA Plus as part of all decision-making processes.
For ISC, the approach to GBA Plus must also be culturally-competent. This means:
- It must start with a recognition of the unique relationship Indigenous peoples have with the government as the First Peoples of this land, and as rights holders under s.35 of the Constitution.
- Taking a distinctions-based approach where Indigenous people are not considered a homogeneous group. It is critical to acknowledge and work to understand the unique historical, cultural, political and socio-economic realities of First Nations, Inuit and Metis people.
- It must be informed by the perspectives and worldviews of Indigenous people and undertaken in respectful partnership with First Nations, Inuit and Metis partners.
- It must recognize the diverse historical and ongoing impacts of colonialism, patriarchy and other systems of oppression.
ISC Evaluators are actively applying the GBA Plus approach in their work, weaving its processes into evaluations to understand how diverse groups of Indigenous people may be impacted differently by ISC policies and programs. To the extent possible, ISC evaluators are applying an intersectional, distinctions-based lens to the whole evaluation cycle, starting at the design phase, through data collection and analyses, and in the generation of findings and recommendations. For example, in the case in the Indigenous Entrepreneurship and Business Development program evaluation, an intersectional, distinctions-based lens was applied to the designing of the evaluation Technical Advisory Committees to ensure that representation on the committee reflects the diverse knowledge and experience of partners involved in the program.
There is an opportunity to foster ongoing dialogue with partners and communities regarding the application of GBA Plus processes in programs and policies in order to better understand how a GBA Plus lens is being applied at the community level and the differential impacts of programs and policies.
Corporate advice and management practice
In the first six months of the COVID-19 pandemic, ISC Evaluation conducted a number of "Rapid Scanning" projects to provide Senior Management with real-time comparative analysis and advice regarding COVID-19 policy interventions in selected jurisdictions, both within Canada and abroad. The scans covered topics such as testing, containment and communication strategies and other comparative analyses with similar experiences such as H1N1. The scans were critical to consolidating and providing current information on possible mitigation strategies for Indigenous communities across Canada. There were five scans:
- A Rapid Scan on COVID-19 Testing Approaches in Selected Jurisdictions
- A Rapid Scan on COVID-19 Containment Approaches in Selected Jurisdictions
- A Rapid Scan on COVID-19 Communication Approaches in Selected Jurisdictions
- A Rapid Scan on COVID-19 Situation for the Navajo Nation and associated government supports
- A Rapid Scan on a Comparative Analysis on ISC's Response to the 2009 Influenza A (H1N1) and COVID-19 Pandemics
ISC Evaluation provides ongoing advice to support the development of Treasury Board Submissions and Memoranda to Cabinet to ensure evaluation evidence is represented and used appropriately, that plans for future evaluations are properly referenced, and to provide high level commentary and advice on the Department's Performance Information Profiles. ISC Evaluation reviewed a high volume of Treasury Board Submissions, and Memoranda to Cabinet. The Directorate also provides annual input into the Departmental Plan, the Departmental Results Report, as well as to the corporate risk and business planning processes.
ISC Evaluation also redesigned its Management Response and Action Plans (MRAP) process and tracking tools. In 2020-21 ISC Evaluation developed and established an interactive, dynamic database to store and structure, in a standardized manner, detailed data for each MRAP. The database has the ability to provide aggregate information on the status of MRAPs from a departmental, sector, or program level and for different aspects of the MRAP (such as recommendations, action items, and timelines). Led by ISC Evaluation, MRAPs are updated on a quarterly basis and tabled at the ISC Performance Measurement and Evaluation Committee twice annually. This database helps ISC Evaluation monitor progress made in addressing evaluation report recommendations and to ensure commitments are met and program improvements are made. However, the potential of the database goes beyond the tracking, monitoring and reporting of MRAPs. It is presently being mined from an analytical perspective using "text" identification and coding methods to see if the database can provide or uncover hidden insights regarding common or systemic program performance issues and barriers from a cross-sectoral or horizontal perspective. This new MRAP tracker and process has become a foundational tool for the practice management function. It also supports the TBS Capacity Survey. Recently other government evaluation functions indicated it was a best practice across the federal evaluation community.
3. The Year in Review
3.1 Status of evaluation completion in 2020-2021
Looking back on the previous year, four (4) evaluations were completed:
- Evaluation of Elementary and Secondary Education Program
- Evaluation of Post-Secondary Education Program
- Evaluation of the Water and Wastewater On-Reserve Program
- ESDC-led Horizontal Summative Evaluation of the Youth Employment Strategy
Also, as of the end of the 2020-21 fiscal year, ISC Evaluation published the following:
- Evaluation of the First Nations and Inuit Home and Community Care Program (March 2020)
- Evaluation of the Assisted Living Program (March 2020)
- A Review of Jordan's Principle (August 2020)
- Evaluation of the Aboriginal Health Human Resources Initiative 2013-14 to 2017-18 (August 2020)
- Five-Year Departmental Evaluation Plan 2020-2021 to 2024-2025 (January 2021)
The following evaluation projects were launched in 2020-2021:
- Evaluation of Education Facilities
- Evaluation of Healthy Child Development
- Cluster Evaluation of Health Infrastructure Support for First Nations
- Evaluation of Health Planning, Quality Management
- Health Services Integration Fund
- Quality Improvement and Accreditation Program
- Evaluation of Health Facilities
- Evaluation of BC Tripartite Governance
- Evaluation of Land, Natural Resources and Environmental Management
4. The Five-Year Plan
4.1 Scope of Work
One of the consequences of the transition was a broader scope of work and increased workload because of the additional programs for which the Department is now responsible. In 2017, the First Nations and Inuit Health Branch (FNIHB) was transferred from Health Canada to ISC. With this transfer, fourteen programs were moved to ISC. Further to the ISC and CIRNAC enabling legislation in July 2019, five programs were transferred from CIRNAC to ISC: Individual Affairs (includes Treaty Annuities); Indigenous Entrepreneurship and Business Development; Economic Development Capacity and Readiness; Land, Natural Resources and Environmental Management; Statutory, Legislative and Policy Support to First Nations Governance. In total 19 programs were added to ISC - the resulting impact to the ISC Evaluation workload was significant, approximately doubling the number of programs to be evaluated with no additional evaluation resources to carry out the evaluation work. Additional challenges include the inherited backlog of several evaluations from the previous shared evaluation function between ISC and CIRNAC and the many recent budget announcements and investments that have expanded the scope of existing programs or introduced new funding streams. For these reasons, there have been ongoing and persistent challenges in completing the planned evaluations. This also limits the flexibility of the Directorate to take on new evaluations that may be of interest to Senior Management as the Directorate is challenged to meet the due dates of the mandatory evaluations it must conduct to be compliant with the FAA. It will likely take a couple of years to stabilize the evaluation planning cycle.
4.2 Program structures
Most departmental programs, representing a majority of spending, are delivered through partnerships with Indigenous communities and in very few cases federal-provincial or federal-territorial agreements are in place. However, other funding structures have been established that are expected to become increasingly common as the Department moves to transfer services to Indigenous entities. For example, the British Columbia Tripartite Health Governance program involves a joint agreement between the Province of British Columbia, the First Nations Health Authority and ISC on health governance. Under this agreement, the parties agree to a tripartite evaluation, led by the First Nations Health Authority, with the final report being shared between the parties. In the future, more non-standard funding arrangements can be expected, with ISC Evaluation examining the 'readiness' of Indigenous governments or institutions to assume program delivery.
Currently, ISC has 30 authorities (voted grants and contributions) to support the 34 programs in its program inventory. However, a one-to-one relationship between the authorities listed in the Main Estimates and the Program Inventory of ongoing programs of grants and contributions deemed a 'Program' by the Department exists in very few instances. A number of programs draw on multiple grants and contributions; and conversely, there are grants and contributions that contribute to multiple programs. This Plan has been developed using the Treasury Board's Interim Guide on Results, which was updated most recently in August 2018, where there is the recognition that a 'program' of grants and contributions can cut across two or more programs in the Program Inventory and the relationship is not always one-to-one. This structure has presented the Department with some challenges in ensuring evaluation coverage for all authorities because of the segregation of a single authority over multiple programs with different evaluation timelines.
The program structure has also resulted in the number of evaluations exceeding the number of programs in the Program Inventory because of the distinct sub-programs that are included in the larger Programs. There are some sub-programs that have their own funding authority which alone supports the rationale for a separate evaluation that can identify improvements that might otherwise be lost in a larger evaluation. There are some sub-programs that are generally aligned with their Program, but when evaluators begin scoping the evaluation, the sub-program is quite unique and a single evaluation is not practical. One planned evaluation becomes two evaluation projects. This results in challenges in meeting TB commitments and FAA requirements as evaluations must sometimes be deferred.
4.3 Evaluation planning methodology
The planning methodology is a multi-step process that begins with a review of the previous year's planning approach to update the steps where necessary and to review all relevant guidance and planning documents, such as the current Departmental Plan and Treasury Board commitments.
For 2021-2022, ISC refreshed its Departmental Results Framework (DRF) and its Program Inventory. Evaluation planning was conducted in alignment with the 2021-2022 framework. While there were no new programs added to the Program Inventory, there were two new Departmental Results added to Governance and Community Development Services. The 2021-2022 Core Responsibilities for ISC are:
- Services and Benefits to Individuals – with 4 supporting programs
- Health and Social Services – with 14 supporting programs
- Governance and Community Development Services – with 13 supporting programs
- Indigenous Self-Determined Services – with 3 supporting programs
Appendix A presents the entire Program Inventory for ISC by core responsibility.
The Plan is also expected to align with the Department's Risk-Based Audit Plan, which is prepared by the Audit and Assurance Services Branch (AASB). The AASB is part of the Audit and Evaluation Sector of CIRNAC and it serves both CIRNAC and ISC. As has been past practice, consideration is given to timing audits and evaluations in a way that does not overburden programs.
Each evaluation included in the Plan has been further assessed as to the type of evaluation required based on the review of Financial Administration Act or Treasury Board requirements and stakeholder consultations, as well as noting whether it is a horizontal or non‑horizontal evaluation. It should be noted that in January 2018, the Treasury Board Secretariat released a new Guide to Departments on the Management and Reporting of Horizontal Initiatives. This updated guide contains new requirements for horizontal initiatives, including a horizontal framework, specific reporting requirements and the creation of a governance structure comprising senior-level officials. TBS also went through an exercise of identifying mature horizontal initiatives that could submit close-out reporting rather than comply with the new requirements. Under the new guidance, ISC has no official Horizontal Initiatives, although it does participate in several led by other departments.
According to Section 42.1 of the Financial Administration Act, government departments and agencies must review, at least once every five years, the relevance and effectiveness of each ongoing grants and contributions program. Credit for evaluation coverage is granted upon deputy head approval of an evaluation report, under the premise that approval allows for the sharing of information in support of management action and the sharing of results with the public. Evaluation planning is conducted with that requirement in mind, reviewing the last fiscal year in which a program was evaluated, and scheduling completion and approval of the program evaluation five years later. Note that programs that are not funded through voted grants and contributions are not subject to the Financial Administration Act requirements.
However, as per the Policy on Results (Section 2.5), programs with five-year average actual expenditures of less than $5 million per year can be exempted from Section 42.1 and do not need to be evaluated every five years. Guidance from Treasury Board Secretariat states the five-year average is to be calculated on three years of actuals (Public Accounts) and two years of planned spending (Main Estimates). These calculations were conducted as part of the planning process and, based on these parameters, the following program is not scheduled for evaluation:
Program inventory | Rationale |
---|---|
Statutory, Legislative and Policy Support to First Nations Governance | Below $5 million |
The Evaluation and Policy Redesign Branch conducted individual consultations in May 2021 with ISC senior management on the draft five-year schedule of evaluations to obtain their input on priorities for their respective sectors and on the timing of evaluations. Their input was synthesized and considered as part of the finalization process for the five-year schedule. The following senior management offices were consulted:
- Senior ADM Regional Operations
- Associate ADM Regional Operations
- Senior ADM First Nations and Inuit Health Branch
- ADM First Nations and Inuit Health Branch Regional Operations
- Chief Finances, Results and Delivery Officer
- ADM Child and Family Services Reform
- ADM Lands and Economic Development
- ADM Education and Social Development Program and Partnership
- ADM Strategic Policy and Partnerships Sector
- DG Communications
- DG Human Resources and Workplace Services
The planning methodology is largely driven by the requirements of the TBS Policy on Results and the Financial Administration Act. In future planning exercises, ISC Evaluation expects to engage national Indigenous organizations to ensure greater transparency of the planning process and to take into consideration the priorities of Indigenous partners.
4.4 Planned spending
Based on the Main Estimates, total planned spending for 2021-22 is outlined in the table below according to the Department's core responsibilities.
Core responsibility | 2021-2022 planned spending |
---|---|
Services and benefits to individuals | $2,006,677,872 |
Health and social services | $5,837,808,255 |
Governance and community development services | $3,069,685,325 |
Indigenous self-determined services | $2,422,137,941 |
Internal services | $169,788,003 |
Total | $13,506,097,396 |
Source: 2021-22 Main Estimates. Figures include the following expenditures: salary; operations and maintenance; capital; statutory and grants and contributions funding; and the costs of employee benefit plans. |
4.5 COVID-19 emergency funds
As of December 18, 2020, over $4.2 billion has been committed in specific COVID-19 support to Indigenous and northern communities and organizations, the bulk of which will be delivered by ISC. At this time, there is not a specific evaluation project identified to evaluate these emergency funding initiatives. Instead, as part of ongoing and future evaluations, ISC will take into consideration and/or address what impact COVID-19 emergency funds have had on the activities and results of core programming. ISC Evaluation will adjust the Plan to reflect any changes to that approach as the pandemic evolves and as priorities and needs change.
4.6 Coverage
Program evaluations are planned to meet mandatory evaluation requirements, including fulfilling evaluation commitments specified in TB Submissions and addressing the statutory requirements of the FAA. These evaluation projects will cover 99% of planned departmental spending from 2021-22 to 2025-26. Departmental planned spending includes salary, operations and maintenance, capital, statutory payments and the costs of employee benefit plans, as well as grants and contributions funding. The remaining 1% represents Internal Services ($169,788,003) and the Statutory, Legislative and Policy Support to First Nations Governance Program ($3,972,389) which has a five-year annual average below the $5 million threshold that exempts it from Section 42.1 of the FAA.
Horizontal evaluations are on the schedule that will address initiatives across multiple departments and organizations. ISC is a partner department in these horizontal evaluation projects.
Evaluation project connected to First Nations Child and Family Service Reform: a Synthesis Report of Key Studies and Reviews of the FNCFS Program.
Discretionary evaluations are scheduled that have been identified as departmental priorities to support internal decision-making.
Both of the following tables list 14 evaluations for the fiscal year 2022-23, as that year includes the Cluster Evaluation of Health Infrastructure Support for First Nations and Inuit. This cluster evaluation covers seven of the Department's First Nations and Inuit Health Branch programs that were grouped together to take a holistic view of how effectively programs are working in concert to improve, or put in place, health infrastructure for Indigenous communities. Consultations with senior management indicated that the cluster evaluation should take a broad strategic approach in order to examine whether the mix of programs meet the current needs of Indigenous communities and the alignment of goals of both FNIHB and ISC, that it covers all health programs related to health infrastructure, and that is in sync with the Department's mandate of reconciliation with Indigenous peoples in Canada. A cluster evaluation is also expected to alleviate the burden on program areas by evaluating their program only once in this five-year evaluation period and to alleviate the backlog of evaluations the Department currently faces.
Fiscal year |
Departmental spending |
Percentage coverage |
Number of evaluations |
---|---|---|---|
2021-22 | $900,866,098 | 7% | 8 |
2022-23 | $3,637,994,616 | 27% | 14 |
2023-24 | $3,850,772,678 | 29% | 11 |
2024-25 | $1,628,419,380 | 12% | 5 |
2025-26 | $3,314,284,232 | 25% | 3 |
Total | $13,332,337,004 | 99% | 41 |
Source: 2021-2022 Main Estimates |
ISC covers 100% of its voted grants and contributions amounts. The contribution amount for the Statutory, Legislative and Policy Support to First Nations Governance Program ($180,207 for 2021-22) is negligible.
Fiscal year | Voted grants and contributions (Gs&Cs) amount |
Percentage coverage voted Gs&Cs |
Number of evaluations |
---|---|---|---|
2021-22 | $691,764,467 | 6% | 8 |
2022-23 | $2,272,171,517 | 20% | 14 |
2023-24 | $3,464,441,508 | 31% | 11 |
2024-25 | $1,598,876,668 | 14% | 5 |
2025-26 | $3,255,913,478 | 29% | 3 |
Total | $11,283,167,638 | 100% | 41 |
Source: 2021-2022 Main Estimates |
The Department will also be participating in two horizontal evaluations that do not address coverage but will involve ISC Programs. These are evaluations of:
- It's Time: Canada's Strategy to Prevent and Address Gender-Based Violence, a horizontal initiative led by Women and Gender Equality Canada.
- The National Action Plan to Combat Human Trafficking, a horizontal initiative led by Public Safety Canada
Although there is no departmental funding attached to these horizontal initiatives, both initiatives involve the Department's Family Violence Prevention Program in reporting and evaluation, given its prevention programming and operational funding for shelters.
4.7 Resources
The Evaluation Directorate has the following planned expenditures for 2021-22 to deliver on the commitments made in the Plan.
Expenditure type | 2021-22 (Budget) |
---|---|
Salaries | $3,035,461 |
Operations and maintenance | $1,826,738 |
Grants and Contributions* | $261,800 |
Total | $5,123,999 |
Full-time equivalents (FTEs) | 31 |
Source: Strategic Policy and Partnerships Sector, Business Management Unit |
4.8 Challenges
Impact of COVID-19 Pandemic
The consequences of the worldwide COVID-19 pandemic will likely resonate for some time posing risks to the Department's ability to meet its planned evaluation targets. In early 2020-21, evaluation activities were curtailed as travel restrictions were put in place and many Indigenous communities declared a state of emergency, allowing only limited access for essential functions and remaining closed to outside visitors for several months. In addition, regional and community partners faced enormous workload challenges due to the pandemic and the multiple effects it is having across Canada. As such, over the past year it has been very difficult to speak with community health workers who would typically be involved in FNIHB evaluations but faced other urgent priorities. For example, the Directorate's ability to advance the evaluation of the Communicable Disease Control and Management program has faced significant challenges. The increased workload for regional colleagues also made it difficult to schedule activities to support all evaluation projects.
The pandemic has had a significant impact on the work of ISC Evaluation, as evaluators were redirected in early 2020-21 to conduct numerous rapid scans related to COVID-19 policy interventions, including planning for a rapid survey and case studies with community health managers on lessons learned in responding to the pandemic. Responding to urgent assessments and real-time decision making also delayed other work on evaluation projects and affected the planned delivery of final evaluation reports.
To advance evaluations, new approaches for partner engagement and data collection methodologies were implemented (for example, remote data collection, virtual interviews, virtual focus groups, etc.). While some communities were able to use internet tools (such as MS Teams) or engage by telephone, many communities lack access to broadband or other types of technology.
Adhering to physical distancing measures (meaning no in-person interviews) and respecting community emergency response measures will likely remain the practice into 2022. In this environment, there is a risk to the timely co-development of evaluation frameworks with broad or diverse groups of Indigenous partners, with the potential to delay evaluation reports. As the effects of the pandemic continue to resonate, ISC Evaluation will monitor the situation and revise timelines as needed, while keeping programs and the ISC Performance Management and Evaluation Committee informed.
Human resources
The dedicated ISC Evaluation Directorate was established in July 2019 with an allocation of 16 full-time equivalents (FTEs) with, at the time, only half of these positions permanently staffed. Evaluation planning exercises have consistently noted the risks to delivering on the Plan because the FTE complement has not been sufficient given the complexity and volume of the work involved. Beyond the 41 planned evaluations, there are also significant requirements and pressures from a practice management perspective (such as tracking and monitoring of Management Response Action Plans, input into TBS submissions and Memoranda to Cabinet, TBS planning, reporting, and publishing requirements, etc.). Furthermore, ISC Evaluation has a unique and strategic role relative to other evaluation functions across government given its objectives to support and assess 'Service Delivery Transfer' to Indigenous partners and to also advance Indigenous evaluation capacities and evaluation functions outside of government. More generally, the demand for evaluation skills across government presents an additional challenge related to recruitment and retention, most particularly lower and mid-level evaluators. Over the past year, the Directorate has focused on building its capacity and has increased its FTE allocation to 22 for 2021-2022, which can be expected to alleviate some workload pressures. However, the Directorate is planning to continue to grow its organization over 2021-2022.
Continuous improvement of evaluation function. The Directorate is taking steps to support the continuous improvement of the function and to address some of the capacity and workload challenges it faces in order meet its three-part mandate. The focus is on how to establish a function that can act as an effective change catalyst and be a key player in driving the Department's agenda, particularly with respect to improving service delivery and supporting eventual service transfer. These steps are also expected to ensure departmental evaluators have opportunities to develop the knowledge and competencies required for their role in alignment with the Policy on Results.
In parallel with this, the evaluation function is developing a "backlog" strategy to map out a path towards completing a number of evaluations that are overdue. The strategy is to adopt a risk-based and balanced approach. For example, depending on priorities and risks, some evaluations will be put on an accelerated schedule, using methods such as 'Rapid Impact Assessments' or a 'clustering' approach that groups evaluations together to provide a more system-wide analysis. Other evaluations may be more suited to integrated, co-developed and partnership-based approaches. It is recognized that co-developed evaluations will take longer in order to allow time to build trust and relationships and to integrate Indigenous worldviews and perspectives, ensuring the engagement at all stages of the evaluation project.
Performance information
The availability and quality of performance information remains the major challenge for evaluation activities. The Government of Canada introduced the Policy on Results in 2016 to instill a culture of performance measurement and evaluation, taking a results-focused approach that relies on gathering performance information. All government programs are required to develop Performance Information Profiles (PIPs) to serve as a guide to gather performance information. However, collecting data at the front-line delivery level remains a challenge for program managers given the resources required for setting up databases that can be used to gather data and then create reports. Much of the performance information resides with First Nations communities, with little information available at the regional level to inform evaluation. Accessing data can often depend on the level of community engagement that has occurred.
Approvals and posting processes
The Practice Management team of the Evaluation Directorate has improved the approval and posting process significantly over the past year and has dedicated resources to ensure timely and high-quality publications. For example, a style guide has been developed for ISC Evaluation to inform evaluators of the requirements for documents intended for publication on the departmental website. Fewer revisions during the HTML coding stage would ensure fewer delays in the process. An evaluation report template is also in development to create efficiencies prior to even the document approval stage. Finally, the approval process has already benefited from the role the Senior Management Committee (SMC) plays as the Department's Performance Measurement and Evaluation Committee (PMEC) as the SMC meets frequently and can approve evaluation reports as they are completed. ISC Evaluation participates at the SMC-PMEC at least once a month.
4.9 Implementing the plan
Evaluations conducted by ISC Evaluation are used to inform decision-making and reporting (such as the Departmental Plan, the Departmental Results Report) to assess progress made by programs in achieving expected results and to incorporate lessons learned in order to improve program design and delivery. As noted, ISC Evaluation posts approved evaluation reports on its website in a timely manner, after sharing reports with Treasury Board.
ISC Evaluation will work collaboratively with internal and external partners to tailor evaluation projects to meet the needs of decision makers, with a growing emphasis on innovating and implementing with Indigenous partners novel methodologies and approaches. This work will set the foundation for meaningful and substantive improvements to service delivery and service transfer while at the same time advance culturally appropriate evaluation capacities and functions. This work remains within the flexibilities of the Treasury Board Secretariat Policy on Results.
ISC Evaluation is moving towards developing and implementing protocols for engagement with Indigenous representative organizations so that First Nations, Inuit and Métis peoples can contribute their knowledge and experience throughout the evaluation process. The Department is also working to build internal culturally-responsive evaluation capacity, as well as to support external capacity in collaboration with Indigenous partners with a focus on service delivery.
The Directorate is committed to delivering the evaluation projects that have been identified for 2021-22. As noted, it is addressing its capacity challenges though the ongoing recruitment, development, and retention of talented staff, with an emphasis on Indigenous recruitment, and is developing a strategy to address the backlog of evaluation projects using a risk-based and strategic approach.
The consequences of the COVID-19 pandemic continue to be felt by the Department as it prioritizes the health and safety of Indigenous communities and peoples. It is expected that the pandemic will continue to have an impact on the implementation of evaluation work in 2021-22. However, the extent of this is unknown. Should factors arise to compromise the delivery of evaluation projects, it will be brought to the attention of the SMC-PMEC in order that the senior management is fully informed.
5. Five-Year Schedule of Evaluations
Year planned | Evaluation title | Sector | Last evaluated | 2020-2021 Total planned program spending | Status | Launch date | Expected DM approval |
---|---|---|---|---|---|---|---|
2021-22 | Evaluation of First Nations Solid Waste Management Initiative | LED | Never Evaluated | $99,000,000 | On-going | Jan-19 | 31-Oct-21 |
2021-22 | Evaluation of Healthy Living | FNIHB | 2014-2015 | $64,660,535 | On-going | Feb-19 | 31-Dec-21 |
2021-22 | Evaluation of Individual Affairs | RO | 2013-2014 | $41,552,176 | On-going | 31-Dec-21 | |
2021-22 | Evaluation of Indigenous Entrepreneurship and Business Development | LED | 2015-2016 | $63,286,008 | To be Launched | Jun-21 | 31-Mar-22 |
2021-22 | Evaluation of Healthy Child Development | FNIHB | 2014-2015 | $135,303,958 | Launched | Nov-20 | 31-Mar-22 |
2021-22 | Synthesis Report of Key Studies and Reviews of the FNCFS Program | CFSR | n/a | TBD | To be Launched | Sep-21 | 31-Mar-22 |
2021-22 | Evaluation of Other Community Infrastructure and Activities | RO | 2015-2016 | $497,063,421 | To be Launched | Jun-21 | 31-Mar-22 |
2021-22 | Evaluation of Indigenous Employees Initiatives and Development Programs | Internal Services/ FNIHB |
Never Evaluated | To be Launched | 31-Mar-22 |
Year planned | Evaluation title | Sector | Last evaluated | 2020-2021 Total planned program spending | Status | Launch date | Expected DM approval |
---|---|---|---|---|---|---|---|
2022-23 | Evaluation of Housing | RO | 2016-2017 | $138,777,038 | To be Launched | Jun-21 | 30-Jun-22 |
2022-23 | Evaluation of Communicable Disease Control and Management | FNIHB | 2014-2015 | $104,169,210 | On-going | Mar-21 | 31-Mar-23 |
2022-23 | Evaluation of Education Facilities | RO | 2015-2016 | $168,330,012 | On-going | Sep-20 | 31-Aug-22 |
2022-23 | Cluster Evaluation of Health Infrastructure Support for First Nations |
FNIHB | 2016-2017 | $146,811,457 | On-going | Jan-19 | 31-Mar-23 |
2022-23 | Cluster Evaluation of Health Infrastructure Support for First Nations |
FNIHB | 2016-2017 | $123,433,198 | On-going/ Included in Health Cluster |
Jan-19 | 31-Mar-23 |
2022-23 | Cluster Evaluation of Health Infrastructure Support for First Nations |
FNIHB | 2016-2017 | $28,164,452 | On-going/ Included in Health Cluster |
Jan-19 | 31-Mar-23 |
2022-23 | Cluster Evaluation of Health Infrastructure Support for First Nations |
FNIHB | 2019-2020 | $2,522,754 | On-going/ Included in Health Cluster |
Jan-19 | 31-Mar-23 |
2022-23 | Cluster Evaluation of Health Infrastructure Support for First Nations |
FNIHB | 2017-2018 | $578,365,990 | On-going/ Included in Health Cluster |
Jan-19 | 31-Mar-23 |
2022-23 | Evaluation of Environmental Public Health | FNIHB | 2016-2017 | $72,306,072 | To be Launched | Sep-21 | 31-Mar-23 |
2022-23 | Evaluation of Mental Wellness | FNIHB | 2016-2017 | $337,978,649 | To be Launched | Jun-21 | 31-Dec-22 |
2022-23 | Evaluation of Land, Natural Resources and Environmental Management | LED | 2015-2016 | $60,234,485 | Scoping | May-21 | 01-Jun-22 |
2022-23 | Evaluation of Contaminated Sites On-Reserve (South of the 60th Parallel) Program | LED | 2015-2016 | $32,226,284 | To be Launched | TBD | 31-Mar-23 |
2022-23 | Evaluation of the Emergency Management Assistance Program | RO | 2016-2017 | $101,852,955 | To be Launched | TBD | 31-Mar-23 |
2022-23 | Evaluation of Supplementary Health Benefits | FNIHB | 2017-2018 | $1,587,908,515 | To be Launched | Oct-21 | 30-Sep-22 |
2022-23 | Evaluation of Urban Programming for Indigenous Peoples | ESDPP | 2016-2017 | $72,864,037 | Year 2 launch | TBD | 31-Mar-23 |
2022-23 | Evaluation of Economic Development Capacity and Readiness | LED | 2015-2016 | $70,952,910 | Scoping (along with Land, Natural Resources and Environmental Management) | Jul-21 | 31-Jul-22 |
2022-23 | Evaluation of the Impact Assessment and Regulatory Processes | FNIHB LED |
Never Evaluated | In progress | Jan-21 | 30-Jun-22 | |
2022-23 | Evaluation of the Miawpukek Grant | RO | 2010-2011 | $11,974,828 | To be Launched | Sep-21 | 31-Jul-22 |
Year planned | Evaluation title | Sector | Last evaluated | 2020-2021 Total planned program spending | Status | Launch date | Expected DM approval |
---|---|---|---|---|---|---|---|
2023-24 | Evaluation of Family Violence Prevention | ESDPP | 2017-2018 | $68,058,392 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of Income Assistance | ESDPP | 2017-2018 | $810,572,408 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of the Strategic Partnerships Initiative (SPI) | LED | 2014-2015 | $15,950,000 | Year 2 launch | Dec-21 | TBD |
2023-24 | Canadian Drugs & Substances Strategy (Including opioids and drug overdose crisis in Canada) | FNIHB | 2017-18 | Year 2 launch | Dec-21 | 2022-23 | |
2023-24 | Evaluation of Clinical and Client Care | FNIHB | 2018-2019 | $349,713,317 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of Community Oral Health Services | FNIHB | Never evaluated | $27,503,864 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of Jordan's Principle | FNIHB | 2018-2019 | $454,373,737 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of Indigenous Governance and Capacity | RO | 2016-2017 | $280,829,009 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of the Grant to Support the New Fiscal Relationship for First Nations | SPP | Never evaluated | $1,558,727,896 | Year 2 launch | TBD | TBD |
2023-24 | Evaluation of Self Determined Services - Education Agreements | ESDPP/ CFRDO |
Never evaluated | $285,044,055 | Year 2 launch | TBD | TBD |
2023-24 | Horizontal Evaluation of Federal Contaminated Sites Action Plan - Phase IV | LED | 2018-2019 | TBD | To be Launched | TBD | TBD |
Year planned | Evaluation title | Sector | Last evaluated | 2020-2021 Total planned program spending | Status | Launch date | Expected DM approval |
---|---|---|---|---|---|---|---|
2024-25 | Evaluation of First Nations Child and Family Services | CFSR | 2014-2015 | $1,279,788,472 | To be Launched | Apr-22 | 31-Mar-25 |
2024-25 | Evaluation of Assisted Living | ESDPP | 2019-2020 | $137,531,536 | To be Launched | TBD | TBD |
2024-25 | Evaluation of the Youth Employment Strategy | ESDPP | 2019-2020 | $26,175,000 | To be Launched | TBD | TBD |
2024-25 | Evaluation of Home and Community Care | FNIHB | 2019-2020 | $184,924,372 | To be Launched | TBD | TBD |
2024-25 | Evaluation of the Survey of Indigenous Peoples | SPP | 2019-2020 | n/a | To be Launched | TBD | TBD |
Year planned | Evaluation title | Sector | Last evaluated | 2020-2021 Total planned program spending | Status | Launch date | Expected DM approval |
---|---|---|---|---|---|---|---|
2025-26 | Evaluation of Education (Elementary and Secondary Education) | ESDPP | 2020-2021 | $1,736,116,658 | To be Launched | TBD | TBD |
2025-26 | Evaluation of Education (Post-Secondary Education) | ESDPP | 2020-2021 | $350,462,465 | To be Launched | TBD | TBD |
2025-26 | Evaluation of First Nations Water and Wastewater Activities On-Reserve | RO/FNIHB | 2020-2021 | $1,227,705,109 | To be Launched | TBD | TBD |
Appendix A - Departmental Results Framework 2021-2022
Core Responsibility | Departmental Results | Program Inventory 34 Programs |
---|---|---|
Services and Benefits to Individuals | Quality and timely services are delivered directly to Indigenous people | 4 Programs
|
Health services delivered to Indigenous peoples contribute to improved health outcomes | ||
Health and Social Services | Indigenous people and communities are healthier | 14 Programs:
|
Indigenous people receive social services that respond to community needs | ||
Indigenous students receive an inclusive and quality education | ||
Governance and Community Development Services | Indigenous communities advance their governance capacity | 13 Programs:
|
Indigenous people have reliable and sustainable infrastructure |
||
Land and resources in Indigenous communities are sustainably managed |
||
Indigenous communities build economic prosperity |
||
Indigenous Self-Determined Services | Indigenous peoples control the design, delivery and management of services | 3 Programs:
|
Indigenous self-determined services are improving outcomes for communities |