Five-Year Departmental Evaluation Plan 2024-2025 to 2028-2029

Co-development and Co-creation with Indigenous Partners

ISC Evaluation is increasingly emphasizing models of co-development and co-creation with Indigenous partners in all evaluation projects. In the short- to medium-term, this includes ways to integrate Indigenous evaluation expertise, knowledge, world views and/or Indigenous capacity development at key points in evaluations (planning, methodology, data collections, and development of findings and recommendations, and reporting). In the long-term, the ultimate goal is to support the establishment of Indigenous evaluation functions outside of government. In general, the evaluation innovation projects have the following objectives:

• To support Indigenous evaluators and experts, research institutes, think tanks, and/or organizations to advance the implementation of ISC program evaluations with a focus on service transfer.
• To support Indigenous organizations to strengthen their capacity to implement program evaluations, enhance and coordinate Indigenous policy and evaluation networks, and organize the Indigenous policy and evaluation field (e.g., conferences, a center of excellence, etc.)
• To partner with Indigenous evaluators, researchers, think tanks, and/or organizations to advance evaluations tools, guides, policies, strategies and other evaluative practices that integrate Indigenous knowledge and worldviews.

In this context, ISC Evaluation continues to support two multi-year evaluation innovation projects with Indigenous partners.

Clustering Evaluations

ISC Evaluation sees clustering evaluations as an effective method to determine how programs are working together and what the overall impact of multiple programs is within communities. The cluster approach means the implicated programs are evaluated as a system rather than at the detailed and programmatic level which has been past practice. Evaluating programs in a cluster provides a more holistic view of the interrelationships between and among programs and the broader impacts of the clustered programs on communities. Additionally, breaking down silos in evaluation work has been an ongoing and important theme with Indigenous partners. New and emerging Indigenous-led evaluation frameworks emphasize holistic evaluation approaches as an important way to uncover systemic barriers and to have greater impacts on improving overall community well-being.  

The clustering approach also aligns more closely with the structure of the Departmental Results Framework (DRF) that supports a single 'Core Responsibility' focusing "Indigenous Well-being and Self-determination". The recently instituted structure shifts the organization of the department's programs from service delivery types into those that support outcomes for Indigenous Peoples and communities—aligning with the department's mandate and vision to support and empower Indigenous Peoples to independently deliver services and address socio-economic conditions in their communities.

In general, the conduct of cluster evaluations requires unique yet mixed sets of skills, including expertise in systems and complexity theory, evaluation theory, Indigenous cultural competencies, and to some extent, expertise in the program itself and in other administrative/ organizational system-level domains (e.g., HR, Financial, data, etc.). The challenge is that, in many cases, the department's programs were not designed or implemented in a systemic way. Due to this complexity, additional effort is required in the planning and co-development phase. While cluster approaches offer some economies such as common frameworks, a generic theory of change, and the development of common interview guides and data collection tools, this approach does require more time to do as well as substantial financial and human resources.

ISC Evaluation recently applied this method to evaluate six health programs in a single evaluation in order to take a holistic view of how the multiple programs work in concert to improve health infrastructure for First Nations and Inuit communities. The Cluster Evaluation of Health Infrastructure Support for First Nations and Inuit^{Footnote 4} took a broad strategic approach, and sought to assess whether the mix of "health system infrastructure programs" meets the current needs of Indigenous communities, supports the Canadian Patient Safety Framework, and aligns with the Department's mandate of transformation in the short term and the transfer of programs to Indigenous Peoples in the long term. This evaluation was successful in producing a comprehensive understanding of the challenges that health infrastructure services are facing, reinforcing the need for increased program information sharing between the regional and headquarters staff. It also pointed out the impact of high staff turnover on the capacity to sustain the delivery of services to communities. The evaluation reported the need to clarify roles and responsibilities regarding ISC's mandate on service transfer. Overall, the cluster approach was found to provide a higher-level view of program achievement but it also confirmed the expectation that the cluster approach requires experienced evaluators with strong management and analytical skills to deal with the high volume of information. 

ISC Evaluation is confident that clustering evaluations is an important approach going forward as it is better aligned with Indigenous partners' feedback on evaluation approaches as well as the closer alignment with the new departmental DRF. It is recognized that integrating this new approach will take a couple of years as methodologies are refined and as the planning process evolves year over year, but clustering will be increasingly integrated into the ISC Evaluation practice.

Finally, while cluster evaluations focus on a system, not on individual programs, they still provide a "light touch" evaluation on each program in the cluster and provide high level details on individual program relevance and performance. It is important to also note that this approach does not preclude the conduct of individual program evaluations at the sub- and sub-sub levels as may be determined by the priorities and needs of the department and partners. It is expected that many sub-programs will continue to be individually evaluated.

In the immediate term, and with the scope still under development, there will be the following:

Crosscutting themes of enquiry in strategic/priority areas

The Government of Canada's commitment to advance reconciliation with Indigenous Peoples underpins all of ISC Evaluation's work and activities as reflected in the ISC Evaluation's vison and key objectives. However, there are a number of other lenses which are added to evaluation frameworks that are also important to inform and influence current and future service delivery and the gradual transfer of services to Indigenous partners. These other cross-cutting themes include: Service Transfer; Gender-Based Analysis (GBA) Plus; Indigenous Children and Families; Impacts of COVID-19; and Climate Change.

Service Transfer: One of the primary goals of ISC is to support Indigenous Peoples in assuming control of the delivery of services. ISC Evaluation has an important role to play in terms of informing program change and re-design that supports the transfer of control. As such, ISC evaluations are including standard lines of enquiry in all program evaluations that assess progress towards service transfer; barriers and opportunities to service transfer; and perspectives on service transfer readiness.

GBA Plus: GBA Plus provides a rigorous method for developing, implementing, monitoring and evaluating initiatives to provide equal outcomes, with an emphasis on inclusiveness and reaching those marginalized due to a combination of social norms and attitudes; institutional power structures; policies and practices that together can limit access to opportunities.

GBA Plus is an intersectional analysis – a framework for understanding how aspects of a person's identity combine to create different modes of discrimination and privilege. In doing GBA Plus, many factors should be considered. The needs and experiences of different groups of people are influenced by intersecting parts of their identity, the context they are in and their lived experiences. All federal government organizations are directed to apply GBA Plus as part of all decision-making processes.

ISC Evaluators are actively applying the GBA Plus approach in their work, weaving its processes into evaluations to understand how diverse groups of people may be affected differently by ISC policies and programs. To the extent possible, ISC evaluators are applying an intersectional, distinctions-based lens to the whole evaluation cycle, starting at the design phase, through data collection and analyses, and in the generation of findings and recommendations. For example, in the case of the Indigenous Entrepreneurship and Business Development program evaluation, a distinctions-based lens was applied to the designing of the evaluation Technical Advisory Committee to ensure that representation on the committee reflects the diverse knowledge and experience of partners involved in the program. Indigenous perspectives in research and evaluation typically involve an acknowledgment of intersectionality and differential impacts, and further communications between the Government of Canada and Indigenous partners and communities could illuminate new ways to work these perspectives into departmental GBA Plus processes.

Indigenous Children and Families: Most programming at ISC touch the well-being of Indigenous children and families either directly or indirectly. In the effort to emphasize and adopt more child- and family-centered evaluation approaches, lines of enquiry on the impacts of the programming on Indigenous children and families are systematically added to all ISC program evaluations.

Impacts of COVID-19: The consequences of the worldwide COVID-19 pandemic had significant impacts on the delivery of ISC programming as travel restrictions were put in place and many Indigenous communities declared a state of emergency, allowing only limited access for essential functions and remaining closed to outside visitors for several months. In addition, regional and community partners faced enormous workload challenges due to the pandemic and the multiple effects across Canada. ISC Evaluation is looking at impacts in each evaluation as a discrete line of enquiry, to investigate lessons learned on programming activities.

Climate Change: It is well understood that Canada's changing climate exacerbates existing challenges and health stressors for Indigenous Peoples in Canada, including wildfires, flooding, permafrost thaw, changing wildlife patterns, and diminishing access to traditional food sources. ISC Evaluation seeks to better understand how Inuit, First Nations and Métis people are experiencing climate change by collecting preliminary and/or cursory information on climate change impacts in Indigenous communities as a cross-cutting theme in all of its evaluations.  

Independent Third-Party Evaluation for ISC Reform and Child and Family Services Suite of Evaluations

In the context of the First Nations child and family services reform agenda, evaluation work has become increasingly more elaborate.  In 2016, the Canadian Human Rights Tribunal order of January 26, 2016, found Canada's First Nations Child and Family Services Program to be discriminatory in its funding, and was ordered to cease its discriminatory practices, reform the Program, cease applying a narrow definition of Jordan's Principle, and to take measures to immediately implement the full meaning and scope of Jordan's Principle.

This led to the 2021 Agreement in Principle on long-term reform signed between Canada and the Parties, along with Consent Order 2022 CHRT 8 that requires Canada to "work with the Parties to establish an Expert Advisory Committee within sixty (60) days of … [the] order to develop and oversee the implementation of an evidence-informed work plan to prevent the recurrence of discrimination … [and to] take reasonable measures to begin implementing the work plan."

As such, Indigenous Services Canada's Evaluation Branch is supporting an Expert Advisory Committee to advise on the development and implementation of an independent third-party evaluation. This is the first time that such an internal review has been conducted at Indigenous Services Canada and is distinct from our ongoing program evaluation work. The evaluation will identify and provide recommendations to redress departmental processes, procedures and practices that contributed to the discrimination identified by the Tribunal to transform what has been described by the Tribunal as the "old mindset."

Ongoing engagement and Indigenous evaluation capacity

Engagement approaches can be expected to vary by evaluation project as some programs have well-established engagement protocols with national Indigenous organizations, regional Indigenous organizations, Indigenous technical/manager working groups, and communities while others do not. ISC Evaluation works with individual programs to determine best approaches for engagement while at the same time is exploring opportunities with Indigenous experts in the evaluation community and/or the private, academic and not-for-profit sectors. ISC Evaluation is also exploring the adoption of distinctions-based evaluation approaches, which would require tailoring evaluations to explore the unique operating environments for some Programs (e.g. urban versus rural differences) and discussions with national Indigenous organizations are ongoing.

In general, ISC Evaluation has a number of engagement practices that enable and advance Indigenous evaluation involvement:

• Establishing working groups or advisory committees that include internal and external subject matter experts and community members. The main purpose of these working groups is to ensure evaluations are undertaken in a culturally responsive and appropriate manner, and they produce meaningful results that are oriented on Indigenous worldviews and perspectives;
• Developing meaningful relationships with Indigenous professional organizations to leverage their sectoral expertise in specific evaluations;
• Travelling to Indigenous communities for engagement and data collection rooted in ceremony, respect, and reciprocity, as defined by each participating community;
• Hiring Indigenous-led evaluation consulting firms;
• Working with partners to achieve high Indigenous/community representation in data collection activities (focus groups, surveys, informant interviews) while taking into consideration Indigenous principles of data sovereignty;
• Seeking guidance from Indigenous Elders and Knowledge Keepers throughout the evaluation journey, where possible; and
• Bringing Indigenous partners to co-present to Senior Management at critical stages of the evaluation.

As an example, for the Urban Programming for Indigenous Peoples (UPIP) program evaluation, the evaluation team determined that a Technical Advisory Committee would be created to advise on the design of the evaluation and to provide feedback on the evaluation findings. Committee members included representatives from ISC Evaluation, UPIP Program staff, the Congress of Aboriginal Peoples, the National Association of Friendship Centres, the National Urban Indigenous Coalitions Council, the Native Women's Association of Canada, the Ontario Federation of Indigenous Friendship Centres, and rotating knowledge keepers representing First Nations, Métis and Inuit communities who opened and closed the meetings, as well as provided their wisdom throughout.

The UPIP Technical Advisory Committee provided guidance on the evaluation terms of reference, methodology, partners to be engaged in the evaluation, preliminary evaluation findings, the final evaluation report, and the final community report. Guidance took the form of two formal group meetings and more than 40 individual consultations, including initial consultations with each Committee member before the development of the evaluation Terms of Reference in order to identify and discuss their specific needs and priorities for the evaluation. Committee members were invited to provide feedback through meetings and/or written correspondence and, if desired, could have their feedback remain anonymous. Notes of all consultations were circulated back to Committee members to provide an opportunity for members to edit and/or add to their feedback. Throughout the evaluation, the evaluation team additionally regularly consulted other Indigenous partners informally to ensure that evaluation activities and deliverables were culturally relevant.

In support of Indigenous evaluation capacity, ISC Evaluation supported the inaugural Indigenous Evaluation Summit, held by the Indigenous Evaluation Network. The Indigenous Evaluation Summit was held virtually over two days in February 2024 and fostered knowledge transfer, networking, and creative discourse related to Indigenous evaluation practices and methodologies. The Summit marked a significant milestone in advancing Indigenous evaluation practices, signaling a renewed commitment to enhancing Indigenous research methodologies and evaluation frameworks.

The Summit brought together 220 evaluators and program staff to share their experiences with Indigenous evaluation practices, with a particular focus on Indigenous youth, whose attendance was sponsored by the Summit. There were several networking sessions to build connections among those working with Indigenous evaluation approaches, and thematic discussions were facilitated by leading Indigenous evaluators, with topics including self-determination in Indigenous evaluation, Indigenous evaluation in non-Indigenous environments, youth inclusion, and Indigenous evaluation in the federal government. The Summit included various cultural components and featured Elders, drum groups, and singers from the four directions in its opening and closing ceremonies and blessings. Elders and Indigenous knowledge holders actively participated in the Summit and supported breakout sessions and discourse involving Indigenous cultural knowledge about evaluation. The Summit highlighted the contributions that evaluation can make to programs and community decision-making by sharing knowledge and planning for the future.

As ISC Evaluation continues to explore and test approaches to transforming its practice, it will begin consolidating lessons learned into the groundwork for its own service transfer strategy. This strategy will include the development of a suite of potential approaches, which can be selected for and tailored to individual programs, to gradually and purposively shift evaluation responsibilities to Indigenous governments and organizations. This suite of approaches will build on successful models that have been implemented in different contexts, including:

• The British Columbia Tripartite Framework Agreement on First Nations Health Governance.

The agreement requires a tripartite evaluation to be conducted every five years, allowing the parties to participate in and guide the evaluation as equals. Accordingly, the scope ISC Evaluation's role in assess the program under the Financial Administration Act and Policy on Results limited to look exclusively at ISC's support role in the Tripartite Relationship.

• The Miawpukek Grant Agreement.

The agreement placed responsibility for leading the evaluation in the hands of Miawpukek First Nation. Beyond the collaborative development of the evaluation Terms of Reference with Miawpukek First Nation, ISC Evaluation's accountability under Financial Administration Act and Policy on Results was limited to tracking the progress of the evaluation.

• The Evaluation of the Mental Health Program.

The Evaluation Working Group highlighted the need for an Indigenous-guided approach focused on community voices, while acknowledging that realizing such an approach within the Financial Administration Act and Policy on Results timelines would be a challenge. As such, the working group agreed that the initial evaluation would have a Governance-level focus limited to national and regional-level perspectives on key issues, but that it would issue an early recommendation regarding the need for an additional phase of work that should be Indigenous-led and community-focused. This two-phase approach is one of the avenues ISC is exploring to simultaneously fulfil federal evaluation obligations and its commitment.

Corporate advice and management practice

The ISC Evaluation Branch provides ongoing advice to support the development of Treasury Board Submissions and Memoranda to Cabinet. This ensures that evaluation evidence is represented and used appropriately and that plans for future evaluations are properly referenced. The Branch also provides high level commentary and advice on the Department's Performance Information Profiles. The Branch provides annual input into the Departmental Plan, the Departmental Results Report, as well as to the corporate risk and business planning processes. ISC Evaluation participates in the regular Neutral Assessment exercise and the annual TBS Capacity Survey Report, while ensuring the quality of published documents.

ISC Evaluation's Management Response and Action Plan (MRAP) process and tracking tools continue to evolve. The interactive database ISC Evaluation developed to store and structure detailed data for each MRAP continues to provide aggregate information on the status of MRAPs from a departmental, sector, or program level and for different aspects of the MRAP (such as recommendations, action items, and timelines). ISC Evaluation works closely with Programs to develop sound and robust measurable MRAPs which are updated on a quarterly basis and tabled at the ISC Performance Measurement and Evaluation Committee at least once each fiscal year. This database helps ISC Evaluation monitor progress made in addressing evaluation report recommendations and to ensure commitments are met and program improvements are made. Further, the Department's Results Delivery Unit (Risk and Management Practices) has found this tool excellent for its own tracking and reporting needs (tracking MAF, Risk mitigation action plans, Management Practices activities, etc.) within the Unit.

The potential of the database goes beyond the tracking, monitoring and reporting of MRAPs. For example, from an analytical perspective, it can be mined by using "text" identification and coding methods to uncover hidden insights regarding common or systemic program performance issues and barriers from a cross-sectoral or horizontal perspective. The MRAP tracker and process has become a foundational tool for the Branch's practice management function, including the recently launched interdepartmental discussions on best practices for monitoring the implementation and completion of MRAPs. With other government evaluation functions indicating that it was a best practice across the federal evaluation community, this has led to the creation of an interdepartmental Evaluation MRAP working group that meets on an ad-hoc basis.

In terms of publishing evaluation reports, ISC Evaluation's Practice Management team has improved the approval and posting process significantly in recent years and has dedicated resources to ensure timely and high-quality publications. For example, a style guide has been developed for ISC Evaluation to inform evaluators early in the writing process of the requirements for documents intended for publication on the departmental website. Templates were developed for evaluation reports and other evaluation-related documents to create efficiencies prior to the approval stage of evaluation products.

Finally, the approval processes are quite efficient as the Evaluation function benefits from the Department's new, stand-alone PMEC meetings, enabling ISC Evaluation to seek timely approvals on a vast array of documents pertaining to the extensive, evaluation-based work undertaken by the Branch. The PMEC meetings are planned to be held every two months, or more frequently if needed.

Human resources

The dedicated ISC Evaluation Directorate was established in July 2019 with an allocation of 16 full-time equivalents (FTEs) with, at the time, only half of these positions permanently staffed. Evaluation planning exercises have consistently noted the risks to delivering on the Plan because the FTE complement has not been sufficient given the complexity and volume of the work involved. Beyond the 46 planned evaluations, there are also significant requirements and pressures from a practice management perspective (such as tracking and monitoring of Management Response Action Plans, input into TBS Submissions and Memoranda to Cabinet, TBS planning, reporting, and publishing requirements, etc.). Furthermore, ISC Evaluation has a unique and strategic role relative to other evaluation functions across government given its objectives to support and assess 'Service Delivery Transfer' to Indigenous partners and to also advance Indigenous evaluation capacities and evaluation functions outside of government. More generally, the demand for evaluation skills across government presents an additional challenge related to recruitment and retention, most particularly lower and mid-level evaluators. To address the volume and increasing complexity of the work, the Directorate transitioned to a Branch in the Fall of 2024, and has been gradually able to increase its capacity and its FTE allocation up to 47 2024-2025, which can be expected to alleviate some workload pressures.

Continuous improvement of evaluation function. The Branch is taking steps to support the continuous improvement of the function and to address some of the capacity and workload challenges it faces in order meet its three-part mandate. The focus is on how to establish a function that can act as an effective change catalyst and be a key player in driving the Department's agenda, particularly with respect to building the relationships necessary for co-development work, improving service delivery and supporting eventual service transfer. These steps are also expected to ensure departmental evaluators have opportunities to develop the knowledge and cultural competencies required for their role and also in alignment with the Policy on Results.

Performance information

The availability and quality of performance information remains the major challenge for evaluation activities. The Government of Canada introduced the Policy on Results in 2016 to instill a culture of performance measurement and evaluation, taking a results-focused approach that relies on gathering performance information. All government programs are required to develop Performance Information Profiles (PIPs) to serve as a guide to gather performance information. However, collecting data at the front-line delivery level remains a challenge for program managers given the resources required for setting up databases that can be used to gather data and then create reports. Much of the performance information resides with First Nations communities, with little information available at the regional level to inform evaluations. Accessing data can often depend on the level of community engagement that has occurred.

The Evaluation Team faced challenges collecting internal data, which was not always aggregated or standardized. This is in line with what has been flagged as an issue in ISC's Management Accountability Framework. In order to address this, ISC Evaluation is employing Indigenous evaluation frameworks and participatory methodologies that involve a broader vision of what data is (e.g., creative products, traditional knowledge, on-the-land learning, etc.) and how to collect, store, and disseminate data in a reciprocal way.