York responds to the knowledge mobilization evaluation

Like most service units York’s Knowledge Mobilization (KMb) Unit counts lots of stuff. We count every presentation we make according to the audience, we count numbers of requests for service, we count students, web hits, tweets, students engaged with non-academic research stakeholders and, of course, we count money.
What has this shown? We’ve brokered over 155 collaborations between researchers and non-academic research stakeholders. We’ve worked with over 200 York faculty. We’ve had over 2M web hits on our web site. We have helped faculty raise over $15M in funding for engaged research, we have placed 29 graduate student KMb Interns with partners (4 of whom were subsequently hired by their internship partner organization) and we have attracted over $700K for funding from KMb partners. But the question remains:
So What?
We have been busy but what impact has this activity had on faculty, students and partners? Most of what we count are inputs into a system of institutional KMb support. What are the outputs, outcomes or impacts? We wish to better articulate the downstream impacts of our efforts so in May 2009 we began a formal evaluation of the first 2.5 years of KMb. York’s KMb Unit contracted the Program Evaluation Unit of the York Institute for Health Research to develop and undertake an evaluation of the KMb Units at York, at UVic and at their collaboration called ResearchImpact. Under the supervision of Michaela Hynie, Director of the Program Evaluation Unit, the evaluation ran from June – September 2009 and a final report which will be released at Congress 2010 was tabled in November 2009.
On February 19, 2010 the report was reviewed by a committee of stakeholders representing local and national perspectives on KMb. Chaired by Stan Shapson (Vice-President Research & Innovation, York University), two York faculty met with external stakeholders (see below for Committee membership) and were asked three questions:
1. How should we continue to develop institutional support services at York University?
2. How should we develop ResearchImpact (Canada’s national knowledge mobilization network)?
3. How can we apply the principles of KMb to industry liaison?
The review committee was positive about York’s institutional KMb capacity and recognized that KMb has certainly created benefits for York, for researchers, for graduate students, for research partners. However, they also provided helpful critical suggestions for the future. The reviewers felt the evaluation fell short of identifying the social, economic, cultural or environmental impacts of research. One reviewer commented that the report did not capture the intangible benefits of KMb at the community level. In a presentation on measuring the outcomes and benefits of university research at the Canadian Association of University Research Administrators (May 9-12, 2010 ) David Phipps recognized that York invested a lot of time to get to a point where we were less than satisfied with our ability to demonstrate the impact of KMb. This is not a criticism of the evaluation report or of the methodology but an observation about the state of evaluation of the non-academic impacts of research, especially in the social sciences and humanities. We also recognize that 2.5 years is insufficient time for many impacts to be realized. Outputs (such as graduate student interns trained) and outcomes (such as graduate students hired by their intern host organizations) were identified but impacts and a social return on investment were generally lacking.
Nonetheless we are pleased that the evaluation reported many successes and achievements and the Committee left the KMb Unit with a number of areas on which we can focus for continued growth. Since the Evaluation Committee meeting on February 19, 2010, Stan Shapson has met three times with David Dewitt (Associate VP Research, Social Sciences & Humanities), David Phipps (Director, Research Services & Knowledge Exchange) and Michael Johnny (Manager, Knowledge Mobilization) to consider the outcomes of the evaluation report and the Committee’s input to inform decisions about future KMb developments at York and ResearchImpact. See the table below for recommendations and action items arising from York’s response to the review committee’s comments on the evaluation report. See also the following documents for more information on York’s evaluation of the KMb Unit and ResearchImpact.
• ResearchSnapshot summary of evaluation report Hynie Evaluation of KMb ResearchSnapshot
• Presentation made by Michael Hynie to Evaluation Committee Evaluation Report Feb 19, 2010
• Final Evaluation report KMb Evaluation Final Report
If you have any feedback please comment on this article using the Leave a Comment feature above.
We wish to thank Michaela Hynie and her team for undertaking the evaluation. We also wish to thank the members of the Evaluation Review Committee:
Lisa Drouillard, Director NSERC Liaison,
Science and Innovation Sector, Industry Canada
Pierre-Gerlier Forest, President
Trudeau Foundation
Janet Halliwell
Former Executive VP, Social Sciences and Humanities Research Council
Bill Hogarth
Former Director, York Region District School Board
Valerie Preston
Professor, Department of Geography, York University
Director, Centre of Excellence for Research on Immigration and Settlement (CERIS)
Stephen Gaetz,
Associate Professor, Faculty of Education
Director, Canadian Homelessness Research Network

Recommendation Action Items Next Steps
1. Capacity Building Include KMb tools in KM in AM schedule
  • Develop KMb tool kit (already developing tools for brokers and P2P
  • Alternate meet and greet with KMb tools
KM in PM
  • Themes will emerge from KM in AM
  • Currently exploring youth engagement
KM in AM
  • Diabetes, aging workforce May 2010
Roll out KMb Tool Kit
  • Clear language research summary completed in draft
  • Schedule of new tools developed for release
  • Roll out: P2P, national knowledge brokers teleconference, KM in AM
2. Faculty Engagement Link to White Paper
  • MJ and PR developing document
Articulate Value Proposition for faculty
  • Identify faculty already working in York Region (completed)
  • Highlight faculty working in York Region on blog and in newsletters
  • Focus with faculty working in York Region – what can they contribute to a governance structure
  • Launch new Joint Advisory Committee
3. Student Engagement 2010 Interns (UWYR-York)
  • One page vision including budget (completed)
  • SS speak with DZ (completed)
  • Materials for competition prepared (completed)
4. Strategic vs Portfolio Approach to knowledge brokering

  • Consensus for a blended approach
UW Priorities

  • helping our youth grow up strong
  • enabling individuals and families to achieve economic independence
  • improving the wellbeing of individuals and community

York Capacity

  • Possible York capacity
  • Mental health
  • Immigration & settlement
  • Poverty & Homelessness
  • Climate Change
  • Confirm York capacity
  • Map unique York capacity onto UW goals
  • Determine where there is “bench strength” and capacity to work in public policy
  • Meeting with York Region to explore YR priorities
  • Thematic meetings to explore collaborations; 1) reactivate poverty working group; 2) explore CERIS to lead immigration; 3) successful SSHRC Public Outreach grant for climate change; 4) mental health
  • Need to identify funding opportunities
5. Convergence Centre Move Community Collaboration Stations to CC
  • Open convergence centre Fall 2010
Hold events at CC
  • Every other KM in AM
  • KMb Expo 2011
KMb Unit presence at CC
  • MJ spend 1 day/week at CC
Community Awareness
  • Identify key community partners and develop communication strategy for dissemination to encourage use of CC for community uses
6. Evaluation & Impacts Evaluation Report: what are the most important/critical findings

  • Evaluation didn’t capture some of the impacts experienced in York Region
  • Develop clear language research summary
  • MJ and DP review report and presentation
  • Provide points to VPRI/Community for comments
Continue to track all activities
Case Study Method

  • possibly with UWYR as case study partner
  • identify faculty with case study experience
  • discuss with Janet Halliwell
Break through evidence based evaluation

  • Contact Janet Halliwell for guidance on methods
  • Review materials from SSHRC Evaluation meeting at Congress 2010
  • Big thinking: CIFAR, John Halliwell (UBC), others?
7. ResearchImpact Functions:

  • Community of Practice
  • Source of research for receptors

Role for Industry Liaison?

  • United Way-University meeting April 2010 (MUN, UQAM, Guelph, USask, UVic)
  • Others?
  • Explore opportunities for funding