York responds to the knowledge mobilization evaluation

Like most service units York’s Knowledge Mobilization (KMb) Unit counts lots of stuff. We count every presentation we make according to the audience, we count numbers of requests for service, we count students, web hits, tweets, students engaged with non-academic research stakeholders and, of course, we count money.
What has this shown? We’ve brokered over 155 collaborations between researchers and non-academic research stakeholders. We’ve worked with over 200 York faculty. We’ve had over 2M web hits on our web site. We have helped faculty raise over $15M in funding for engaged research, we have placed 29 graduate student KMb Interns with partners (4 of whom were subsequently hired by their internship partner organization) and we have attracted over $700K for funding from KMb partners. But the question remains:
So What?
We have been busy but what impact has this activity had on faculty, students and partners? Most of what we count are inputs into a system of institutional KMb support. What are the outputs, outcomes or impacts? We wish to better articulate the downstream impacts of our efforts so in May 2009 we began a formal evaluation of the first 2.5 years of KMb. York’s KMb Unit contracted the Program Evaluation Unit of the York Institute for Health Research to develop and undertake an evaluation of the KMb Units at York, at UVic and at their collaboration called ResearchImpact. Under the supervision of Michaela Hynie, Director of the Program Evaluation Unit, the evaluation ran from June – September 2009 and a final report which will be released at Congress 2010 was tabled in November 2009.
On February 19, 2010 the report was reviewed by a committee of stakeholders representing local and national perspectives on KMb. Chaired by Stan Shapson (Vice-President Research & Innovation, York University), two York faculty met with external stakeholders (see below for Committee membership) and were asked three questions:
1. How should we continue to develop institutional support services at York University?
2. How should we develop ResearchImpact (Canada’s national knowledge mobilization network)?
3. How can we apply the principles of KMb to industry liaison?
The review committee was positive about York’s institutional KMb capacity and recognized that KMb has certainly created benefits for York, for researchers, for graduate students, for research partners. However, they also provided helpful critical suggestions for the future. The reviewers felt the evaluation fell short of identifying the social, economic, cultural or environmental impacts of research. One reviewer commented that the report did not capture the intangible benefits of KMb at the community level. In a presentation on measuring the outcomes and benefits of university research at the Canadian Association of University Research Administrators (May 9-12, 2010 ) David Phipps recognized that York invested a lot of time to get to a point where we were less than satisfied with our ability to demonstrate the impact of KMb. This is not a criticism of the evaluation report or of the methodology but an observation about the state of evaluation of the non-academic impacts of research, especially in the social sciences and humanities. We also recognize that 2.5 years is insufficient time for many impacts to be realized. Outputs (such as graduate student interns trained) and outcomes (such as graduate students hired by their intern host organizations) were identified but impacts and a social return on investment were generally lacking.
Nonetheless we are pleased that the evaluation reported many successes and achievements and the Committee left the KMb Unit with a number of areas on which we can focus for continued growth. Since the Evaluation Committee meeting on February 19, 2010, Stan Shapson has met three times with David Dewitt (Associate VP Research, Social Sciences & Humanities), David Phipps (Director, Research Services & Knowledge Exchange) and Michael Johnny (Manager, Knowledge Mobilization) to consider the outcomes of the evaluation report and the Committee’s input to inform decisions about future KMb developments at York and ResearchImpact. See the table below for recommendations and action items arising from York’s response to the review committee’s comments on the evaluation report. See also the following documents for more information on York’s evaluation of the KMb Unit and ResearchImpact.
• ResearchSnapshot summary of evaluation report Hynie Evaluation of KMb ResearchSnapshot
• Presentation made by Michael Hynie to Evaluation Committee Evaluation Report Feb 19, 2010
• Final Evaluation report KMb Evaluation Final Report
If you have any feedback please comment on this article using the Leave a Comment feature above.
We wish to thank Michaela Hynie and her team for undertaking the evaluation. We also wish to thank the members of the Evaluation Review Committee:
Lisa Drouillard, Director NSERC Liaison,
Science and Innovation Sector, Industry Canada
Pierre-Gerlier Forest, President
Trudeau Foundation
Janet Halliwell
Former Executive VP, Social Sciences and Humanities Research Council
Bill Hogarth
Former Director, York Region District School Board
Valerie Preston
Professor, Department of Geography, York University
Director, Centre of Excellence for Research on Immigration and Settlement (CERIS)
Stephen Gaetz,
Associate Professor, Faculty of Education
Director, Canadian Homelessness Research Network