Amo, C. (2007). Conceptualizing research impact: The case of education research. The Canadian Journal of Program Evaluation, 22(1), 75-98. http://cjpe.ca/secure/22-1-075.pdf
Abstract
This qualitative study aims at conceptualizing research impact generally by studying the specific case of research impact in the field of education. An analysis process akin to grounded theory was applied to the analysis of sections of reports provided by educational researchers. Literature on the subject of research impact was used to substantiate and complete the portrait of educational research impact that emerged from the data. The resulting conceptual framework proposes five interdependent stages, each one characteristic of certain categories of research impact that are typically interrelated in time and in terms of researcher control. It is hoped that this conceptual framework will help program evaluators and researchers tackle the larger task of uncovering and arguing the meaningfulness of alternative ways of measuring the impacts of research in the social sciences and humanities.
Data source: Because the research worked at SSHRC at the time of writing the paper the author was able to obtain 51 end of grant reports from the 1993 education panel of the Standard Research Grant competition. These reports are due 6 months after end of the funding period and are researcher self-reports of the significance of the research results and the additional (i.e. extra academic) impacts of the research. Here are my challenges with the primary data source:
a) Researcher self-disclosure: impact of research is mediated through our partners who use research to inform policy and practice decisions. How reliable are researchers who self- report without necessarily confirming with their research partners (if there were any)?
b) 6 months? We know that it can take years for research to inform decisions about practice and policy. End of grant reports do not effectively capture these extra academic impacts (see comment below on SSHRC’s Connections Evaluation).
c) 1993? Even if the data were collected a couple of years before publication in 2007 why pick end of grant reports that would have been submitted in 1996-1997? Reports from more recent grants would provide more current information.
Connections evaluation: SSHRC recently (summer 2013) completed a deep evaluation of funding programs in their Connections Theme. This theme encompasses funding programs that support knowledge mobilization including Public Outreach, Knowledge Impact in Society, Strategic Knowledge Clusters and Aid to Workshops. This evaluation started with end of grant reports as one data source but recognized that this was not sufficient to capture impacts that are mediated through partners. The 2013 evaluation interviewed the partners of these research and knowledge mobilization grants and it was through those interviews that impressive impacts on policy and practice were identified (full disclosure, I was the lead investigator on a project that was studied in the evaluation and I was on the external advisory committee for this evaluation). The report of that evaluation will be posted by SSHRC shortly.
Results: The author developed a conceptual framework to illustrate five interdependent stages in the process of research impact with each stage illustrating certain categories of research impact that were illustrated in the data collected.
Conducting the research – the benefits arising from the conduct of research « refers to the increase in knowledge and research skills acquired by those participating in the research process« . Benefits also accrue to participants in the research such as an increased awareness of research « increased capacity to receive and respond to research results and benefits that result from participating in an intervention« . The conduct of research can also produce new tools and methods that can help other researchers in the conduct of their own research.
Sharing the research findings – Sharing with the scholarly community (conferences, workshops) can validate and confirm preliminary results. Sharing with the non-scholarly community can generate « buy in or interest in the future results of the research« .
Disseminating the knowledge – Dissemination means releasing the results through traditional academic means such as peer reviewed articles, book chapters and student theses but can also mean through non-traditional routes such as media and through the internet (and since 2007 through the increasing use of social media). The author indicates the increasing reach of the research if it is disseminated through the internet and/or through the media.
Short term research impact – Short term research impacts appear to accrue to the researcher and his/her research enterprise. These include contributing to the field of educational research, scholarly recognition and contribution generally to the social sciences and humanities.
Long term research impact – Long term research impacts are those impacts on teaching policy and practice and on curriculum development – « impacts that tend to occur over time« . The author cites one example of long term research impact as « proposing concrete solutions to teachers…the results of our research should also influence the training of future teachers« . Giving solutions to teachers sounds like disseminating to a non-academic audience (2 steps back). There is nothing in this statement that speaks to any change occurring in teacher practice or training. Prophesizing something might happen is not the same thing as demonstrating that it has. Her second example about incorporating children’s rights education into curricula is a better example of a long term impact of research.
The conceptual framework: The author draws a spiral diagram depicting these five stages starting at the centre and proceeding to the outer reaches of the spiral. It’s a fairly clever model since there are double headed arrows connecting between different stages of the spiral illustrating how there is cross talk between these stages. This is how the author addresses the concerns of linearity because, essentially, this is a logic model that progresses from conducting the research to long term impacts of research. The diagram also illustrates that as short term and long term impacts accrue over time the influence of the research on impact decreases (see discussion of attribution on Mobilize This!, our knowledge mobilization blog).
Without going into any detail, the author references the context and the facilitation of research. Please see a journal club on a paper by Ben Levin and one on the PARIHS framework that discusses these mediators of impact in more detail.
Questions for brokers
- The primary data source was end of grant reports from grants awarded in 1993. With the more recent rise of the research impact « agenda » would end of grant reports from the Standard Research Grant competition in 2007 provide any further consideration of extra academic impacts of research?
- What do you think of the spiral depiction of an otherwise linear model? Does this adequately address the iterative and non-linear nature or research use?
- By being unable to demonstrate much long term, extra-academic impact from end of grant reports this paper indirectly supports the observation that « impact is measured at the level of the user » (one of my basic principles of knowledge mobilization). If we were to actually interview the non-academic research partners do you think we could finally move beyond all these conceptual frameworks (see here and here) which is a tool to help us think about the impacts of research but doesn’t actually demonstrate impact?
- If you’re interested in scholarship on the impacts of researcher check out the academic journal, Research Evaluation that had a special issue on the state of the art in assessing research impact in 2011.
ResearchImpact-RéseauImpactRecherche is producing this journal club series as a way to make the evidence and research on knowledge mobilization more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read the articles. Then come back to this post and join the journal club by posting your comments.