The demand for evidence in the Canadian public service

Written by Louis Melançon, Research assistant at Research Impact Canada

Bédard, Pierre-Olivier, et Alexandra Mallett. “La demande pour des données probantes au sein de la fonction publique canadienne.” Démocratie et politiques publiques (2019): 229.

How does evidence influence policy decisions in Canada? In a chapter of Démocratie et politiques publiques, published in 2019 under the direction of Jérôme Couture and Steve Jacob, researchers Pierre-Olivier Bédard and Alexandra Mallett explore the mechanisms and policies in place within the Canadian public service that impose the use of evidence.

In this context, evidence can be defined as “all sources of data, research results and knowledge resulting from systematic investigations and methodical analyses, regardless of their origin or disciplinary field.” We owe it to Carol Weiss for the best-known typology for the use of evidence. The first form of evidence use, called instrumental concerns cases where the conclusions of research have directly influenced a political decision. Conceptual use concerns cases where the data has indirectly influenced a decision, by modifying the understanding of the decision maker or analyst faced with an issue. Symbolic use consists of using evidence to justify or legitimize a decision on scientific grounds once this decision has already been made. Finally, compulsory use concerns cases where the consideration of evidence is required within the decision-making process itself.

Compulsory use is described by Weiss as a partial solution to the problems associated with the use of evidence in policy, such as the lack of capacity or motivation of public decision-makers, as well as the inadequate dissemination of knowledge. It is a sort of mandated, obligatory instrumental use. Compulsory use can help bring transparency to the decision-making process, as well as embed learning in the direction of public policy. In a context where evidence of results and performance is required, the imposed use of evidence allows the establishment of a certain accountability.

Type of use	Definition
Instrumental	Evidence guides the decision. Direct and linear contribution to the decision.
Conceptual	Evidence helps to modify the understanding of a phenomenon. Indirect and non-linear contribution to the decision.
Symbolic/political	Evidence is used to legitimize and justify a decision that has already been made. No contribution (symbolic) to the decision.
Imposed	Evidence is used because of a clear mandate (legal or otherwise) requiring its analysis and consideration. Variable contribution depending on the nature of the mechanism (accountability versus learning).

In Canada, much of the decision-making by the executive branch of government is orchestrated by the Cabinet of Ministers, supported by central agencies such as the Department of Finance, the Treasury Board, the Privy Council Office and the Office of the Prime Minister. The Cabinet is intended as a decision-making forum for proposing new policies, or even bringing about changes to an existing policy. Sectoral ministries (health, environment, etc.) are linked to the Cabinet through various processes such as memoranda to the Cabinet, presentations to the Treasury Board, and presentations to the Governor in Council. The analysis and information used to develop a policy or program proposal must be included in the documents sent to Cabinet. The authors of these documents must not only present these data, but also specify and cite the documents from which they are drawn. The specialized tool DEBAT (due diligence and evidence-based analysis tool) was made available to these editors in 2016 in order to ensure a more robust analysis of the data used.

The regulation of various sectors and activities is another important task of public decision-makers. The Cabinet Directive on Streamlining Regulation (CDSR) emphasizes the importance of “making decisions based on evidence and the most advanced scientific and empirical knowledge available.” A key step in the process is the preparation of a Regulatory Impact Analysis Statement.

The Policy on Results, implemented in the summer of 2016, established “the fundamental requirements of Canadian federal ministerial responsibility for performance and evaluation information.” Each department must therefore develop a departmental results framework that reports on these results on a regular basis. The policy also mandates evaluations for any program with five-year average expenditures above $5 million. The Performance Measurement and Evaluation Committee (PMEC) directs and plans these evaluation activities. In practice, the methods used and the quality of evaluations vary greatly from one ministry to another, and even within a ministry. Also, contrary to what one might think, the achievement of results as determined by all these evaluations is not necessarily a formal condition for obtaining funding.

Time horizon of demand	Stage in the public policy process	Objective	Example
Ex ante	Formulation and implementation	Establish the parameters of a policy and determine its feasibility, relevance and justification.	Memorandum to Cabinet – Due diligence and evidence-based analysis tool for Memoranda to Cabinet.
	Regulatory	Determine the anticipated results of a policy with a view to mitigating them. Regulatory Impact Analysis Statement (RIAS).	Summary of the Regulatory Impact Study [Résumé de l’étude d’impact de la réglementation (REIR)].
Ex post	Evaluation	Determining the results achieved by a policy and informing decision-making.	Outcome-based policies.

Bédard and Mallett underline some limitations to the imposed use of evidence. First, it is always difficult to determine to what extent these data influence decisions. Also, just because the use of evidence is institutionalized does not necessarily mean that it is accomplished effectively or substantially. Finally, in the absence of a clear definition of what is meant by “evidence data,” quality standards for this data or official guidance on their selection, public administration actors have such a latitude that it leads to a significant risk of selection bias. The use of systematic literature reviews, a practice still little known to analysts and public decision-makers, could be an interesting tool for dealing with this issue.