20
Evaluation is the collection of, analysis and interpretation of information about any aspect of a programme of education or training as part of a recognized process of judging its effectiveness, its efficiency and any other outcomes it may have.[1]
Mary Thorpe
How are evaluation and sustainability related?
Evaluating whether your open working group has met its goals and stated outcomes is an essential part of ensuring its sustainability. (See Goals and Purpose.) Communicating to stakeholders—including senior administration and student groups—about success and impacts of activities such as events, grants, or program/policy changes may lead to increased support and further initiatives. However, evaluating the impact of OER adoption and advocacy can be challenging, so it will be important to consider the type of tools to use and evidence to gather early when establishing your open working group:
…evidence generated by complex and innovative processes such as OER release is often itself complex, context-specific and difficult to generalise. These kinds of initiatives require significant organisational change and may include external partners and stakeholders with very different cultures and practices. Evaluation, in particular is challenging and ranges from evaluating specific OER for fitness of purpose, changes in staff attitudes, impact on learning and teaching and longer term impact on institutional practices and the wider community.”[2]
Questions to Consider: About Evaluation & Sustainability
- What are we evaluating?
- How do you build in evaluation?
- What is the open working group’s goal in evaluation?
- How do you report?
- How do report your own activity (e.g. advocacy, capacity building, awareness)?
- How do we report on and show value for the working group itself?
Why do we evaluate?
There are many reasons why open working groups should evaluate their work. It allows you to:
- Share cost savings with students, governments, and institutions. This can provide a rationale for government and institutional funding and student buy-in.
- Link adoption of resources with student learning (i.e., students contributing to resources, customization by faculty).
- Determine and ensure OER quality. Quality is a central concern for faculty in considering OER adoption.[3]
- Share success and OER adoption. Sharing adoptions and the success of open education programs and projects encourages adoptions and increases the awareness of OER.
Who evaluates?
Different institutional stakeholders often are charged with evaluating OER:
- Students. For example, students at UBC led an environmental scan of OER [PDF].
- Faculty researching OER. For example, Gill Green at UBCO, Christina Hendricks at UBC, Rajiv Jhangiani at KPU, and Georg Rieger at UBC.
- Libraries. For example, this research into a community of librarians supporting OER [PDF].
- Instructional support staff researching OER.
- Centres for teaching and learning.
In Practice
The TRU Special Investment Fund grant, which supports faculty development, has the following requirements built in. The open working group must collect and report on the following:
- Total investment in grants programs,
- Number of funded courses,
- Total enrollment in funded courses over time,
- Average textbook costs (before OER development),
- Total student savings after one year, and
- The extent of faculty dissemination of their work in developing and integrating OER.
- Fred Percival, Henry Ellington, and Phil Race, A Handbook of Educational Technology, 3rd Edition (London: Kogan Page Ltd., 1993). ↵
- Lou McGill, "Open Educational Resources: Sustainability," Jisc, https://www.jisc.ac.uk/guides/open-educational-resources/sustainability (accessed January 25, 2019). ↵
- I. Elaine Allen and Jeff Seaman, “Online Report Card: Tracking Online Education in the United States,” Babson Survey Research Group, 2016. ↵
