Five-Year Evaluation Plan 2019-2020 to 2023-2024

2.1 Roles and Responsibilities

Across the Government of Canada, evaluation is understood as the systematic and neutral collection and analysis of evidence to judge merit, worth or value. This is done to provide an assessment of, and allow for the judgement of, the relevance, performance, and efficiency with which resources are utilized. It helps to identify possible alternative ways to contribute to or achieve the same results in a more effective and efficient manner. The information evaluation provides is used to inform decision making and make improvements.

The purpose of the evaluation function is to ensure that ISC has a comprehensive, timely and reliable base of evidence to support policy and program improvement, expenditure management exercises, senior management and Cabinet decision making and public reporting. Evaluations are playing an increasingly important role in decision making, improvements, innovation and accountability across government. They are intended to provide transparent, clear and useful information on the results that departments have achieved and the resources used to do so, in order to manage and improve programs, policies and services as well as to allocate resources based on performance to optimize results.

2.2 Purpose of the Evaluation Plan

The primary purpose of the Evaluation Plan is to help the Deputy Head ensure that credible, timely and neutral information on the ongoing relevance and performance of direct program spending and ongoing programs of Grants and Contributions is available to support evidence-based decision making. The Plan also:

• Provides an opportunity to align evaluations to ensure that the information needs of the department and other evaluation users (e.g., other government departments) are being met;
• Helps ensure that evaluations supporting program redesign are planned and completed in advance of program renewal;
• Allows departmental units responsible for the development of the Departmental Plan and the Departmental Results Reports, as well as other groups engaged in strategic planning and reporting activities, to identify when evaluations will be available to inform their work;
• Initiates regular communication and consensus-building on evaluation needs and priorities across the Department; and
• Provides central agencies with advanced notice of when evaluations will be available to inform their work (e.g., in support of Memoranda to Cabinet, Treasury Board Submissions, etc.).

Moreover, the Plan serves as an important tool for the Department's Head of Evaluation and EPMRB to manage project workflow and plan the activities of its human resources. ISC has chosen to utilize its Senior Management Committee as its Performance Measurement and Evaluation Committee (PMEC), where evaluation reports are presented. The PMEC is a requirement as per the TB Policy on Results (2016).

2.3 Policy Framework and Guidance

ISC is subject to the TB Policy on Results (2016) and Directive on Results (2016). The Policy on Results allows departments to establish a results framework that identifies core responsibilities, associated results and appropriate indicators. It also limits requirements so that departments will collect performance information while remitting only some of it to Treasury Board. It allows departments to prioritize evaluation coverage and issues, while professionalizing the evaluation function.

It will continue to ask departments to report on both direct program spending and ongoing programs of grants and contributions evaluation coverage, but permits greater flexibility by allowing for risk-based planning for direct program spending coverage, and the option to defer evaluations beyond five years where average yearly expenditures are less than $5 million (over the preceding five year period). The Policy on Results further recognizes that various types of evaluation are possible and essential to meeting the information needs of decision makers.

The Directive on Results requires evaluations to focus on relevance, effectiveness and efficiency. It also clarifies many of the requirements in support of the Policy on Results, including roles and responsibilities of individuals and the PMEC as well as the expectations around planning departmental evaluation coverage.

2.4 Evaluation Planning Methodology

ISC updated its Departmental Results Framework – used to report its high level, core responsibilities and outcomes – and a program inventory for 2019-2020 and evaluation planning was conducted in alignment with that framework. Appendix A presents the Program Inventory for ISC by core responsibility. EPMRB has taken the approach to plan for evaluations based on these core responsibilities and program inventory. This has resulted in the amalgamation of some previously distinct programs into larger programs, reflecting the streamlining efforts that have been undertaken in response to the Policy on Results. There have also been three additional programs added to the Program Inventory since the 2018-19 Evaluation Plan was approved.

In addition, there is the requirement under the Financial Administration Act (Section 42.1) to evaluate all ongoing programs of Grants and Contributions every five years. Credit for evaluation coverage is granted upon deputy head approval of an evaluation report, under the premise that approval allows for the sharing of information in support of management action and the sharing of results with the public. Evaluation planning is conducted with those requirements in mind, reviewing the last fiscal year in which a program was evaluated, and scheduling completion and approval of the program evaluation five years later. 

The planning timeframe for the Department's evaluations is typically 12 months (e.g. the date of approval for the Terms of Reference – the start of the evaluation – is generally 12 months before the planned approval date).

This evaluation plan is also aligned with the 2019-20 to 2020-21 Risk-Based Audit Plan prepared by the Audit and Assurance Services Branch (AASB). Both the AASB and EPMRB are part of the Audit and Evaluation Sector, serving both CIRNAC and ISC. Consideration has been given to timing audits and evaluations in a way that does not over burden programs.

Since the EPMRB supports both departments, consultations with both CIRNAC and ISC leadership/senior management and with Treasury Board officials have also influenced which programs will be evaluated during which fiscal year.

Each evaluation on the Plan has been further assessed as to the type of evaluation required (e.g., impact, formative) based on the review of Financial Administration Act or Treasury Board requirements and stakeholder consultations, as well as noting whether it is a horizontal or non-horizontal evaluation.

3.1 Departmental Core Responsibilities

3.2 Program Structures

Departmental activities are largely structured by funding arrangements or formal agreements with First Nations and/or provincial or territorial governments. Most departmental programs, representing a majority of spending, are delivered through partnerships with Indigenous communities and federal-provincial or federal-territorial agreements.

Currently, ISC has 22 authorities (voted Grants and Contributions) to support the 29 programs in its program inventory. A one-to-one relationship between the authorities listed in the Main Estimates and the Program Inventory of ongoing programs of Grants and Contributions deemed a 'program' by the Department exists in the minority of instances.

A number of programs draw on multiple grants and contributions; and conversely, there are grants and contributions that contribute to multiple programs. This plan has been developed using the Treasury Board's Interim Guide on Results, which was updated most recently in August 2018, where there is the recognition that a 'program' of Grants and Contributions can cut across two or more programs in the Program Inventory and the relationship is not always one-to-one.

3.3 Exemptions from Section 42.1 of the Financial Administration Act

According to Section 42.1 of the Financial Administration Act, government departments and agencies must review, at least once every five years, the relevance and effectiveness of each ongoing Grants and Contributions program. Note that programs that are not funded through voted Grants and Contributions are not subject to the Financial Administration Act requirements. However, as per the Policy on Results (Section 2.5), programs with five-year average annual expenditures of less than $5 million per year can be exempted from Section 42.1 and do not need to be evaluated every five years. Guidance from Treasury Board Secretariat states the five-year average is to be calculated on three years of actuals and two years of planned spending (Main Estimates). Based on these parameters, there are no ISC programs that fall into this category.

3.4 Planned Expenditures and Transfer Payments

According to the 2019–2020 Main Estimates, total planned spending for 2019-20 is outlined in the table below according to the Department's core responsibilities.

4.1 Status of Evaluation Completion in 2018-19

Looking back on the previous year, nine evaluations were completed with three of those evaluation reports awaiting formal approval. These evaluations were:

• Evaluation of Income Assistance
• Evaluation of Clinical and Client Care (First Nations and Inuit Health)
• Horizontal Evaluation of the Federal Initiative to Address HIV/AIDS in Canada (Health Canada-led)
• Review of Jordan's Principle
• Evaluation of the West Coast Energy Infrastructure Initiative (Horizontal; NRCan-led)
• Evaluation of the Shelter Enhancement Program On-Reserve (Horizontal; CMHC-led)
• Evaluation of Assisted Living (awaiting final approval)
• Evaluation of First Nations and Inuit Home and Community Care (awaiting formal approval)
• Evaluation of the Aboriginal Health Human Resources Initiative (awaiting formal approval)

Looking forward, the following evaluations, which were on the plan for completion in 2018-19, are now expected to be completed and approved in 2019-20:

• Evaluation of Elementary and Secondary Education
• Evaluation of Post-Secondary Education
• Evaluation of Nutrition North Canada (Horizontal; CIRNAC-led)

EPMRB presents evaluation reports during regularly scheduled meetings of the Senior Management Committee. The Committee reviews evaluation reports and recommends the approval of each report to the Deputy Head. Formal Deputy Head approval is necessary in order for Treasury Board to consider an evaluation report approved.

4.2 Implementing the Plan

Evaluations conducted by EPMRB are used to inform decision making and reporting (such as the Departmental Plan, Departmental Results Reports) to assess progress made by programs in achieving expected results and to incorporate lessons learned in order to improve program design and delivery. The Branch also posts approved evaluation reports on its website, after sharing reports with Treasury Board.

EPMRB will work collaboratively with partners and stakeholders to tailor evaluation projects to meet the needs of decision makers, as well as perhaps experimenting with new and novel methodologies and approaches for use in the next generation of federal evaluation. The Policy on Results provides for more flexibility in the various types of evaluations and the identification of issues, allowing EPMRB more flexibility in scoping its work.

EPMRB plans to move towards developing and implementing protocols for engagement with Indigenous representative organizations so that First Nations, Inuit and Métis peoples can contribute their knowledge and experience throughout the evaluation process. The Department is working to build capacity in collaboration with Indigenous partners with the ultimate objective of improved service delivery. The evaluation function can play an important role in this transformation. With this objective in mind, a decision has been made by the Department to move evaluation resources from the Audit and Evaluation Sector to the new Strategic Policy and Partnerships Sector, thereby imbedding the function within ISC's Strategic Policy function. This organizational change will be planned and undertaken this fiscal year.

4.3 Planned Evaluation Coverage

ISC plans for 29 evaluations covering 99 percent (99%) of direct program spending from 2019-20 to 2023-24. The remaining one percent (1%) represents Internal Services ($110,917,383). There are no programs below the $5 million threshold that would exempt the program from Section 42.1 of the Financial Administration Act. ISC covers 100 percent of its voted Grants and Contributions amounts. 

4.4 Other EPMRB Activities

In addition to conducting evaluations, EPMRB undertakes a wide range of activities to support and strengthen evaluation and performance measurement work. These include providing advice and support on performance measurement, corporate planning and reporting at the departmental level, supporting capacity building and outreach activities, and developing research and special studies on evaluation and performance measurement issues.

4.5 Challenges