4

Paul Huckett
Christian Utara

Abstract

The Engineering for Professionals (EP) program, a division of the Johns Hopkins University Whiting School of Engineering, provides graduate education to working engineers and applied scientists. To meet the needs of working professionals, EP offered its first online course in 2001 (Bjerkaas, 2005). Today, it offers 21 on-site, Virtual Live (VL—synchronous online), and online (asynchronous online) master’s degree programs. To support the rapid growth in synchronous and asynchronous online courses, a “gold standard” was created to ensure quality and consistency across all courses in all modalities. This chapter will highlight the importance of creating a system to enable the expansion of courses and programs across multiple modalities and the synergies that need to occur for this to be successful. A case study from the Systems Engineering (SE) program will show how it achieved, via successful implementation of the gold standard,

1. A perfect score on the Accreditation Board for Engineering and Technology (ABET) review;
2. An improved student learning experience;
3. Alignment of core courses across multiple modalities;
4. Consistent application of materials at all sites and modalities;
5. A consistent look and feel for both students and instructors across the entire program;
6. The ability to rapidly expand the program to new locations and modalities.

Introduction

With the turn of the 21st century, advancements in technology created opportunities for higher education institutions to reach a broader base of students through distance education, commonly referred to as online education (Radford, 2011). Demand for online education has increased year after year since 2000, and in 2018, online enrollments hit an all-time high (Seaman, Allen, & Seaman, 2018).

The Johns Hopkins University Whiting School of Engineering recognized this trend and began offering online master’s courses in 2001 through its part-time Engineering for Professionals (EP) program. Today, EP offers 21 master’s programs—19 of which can be completed online—14 post-master’s certificates, and 11 graduate certificates. In total, it offers over 350 online courses taught by approximately 500 adjunct faculty and has a student headcount of over 4,000—making it the second-largest part-time engineering school in the United States (Johns Hopkins Whiting School of Engineering, 2017). In addition, it partners with industry to bring master’s programs to their organizations in an accelerated degree format.

The exponential growth in online courses at EP has yielded many positive results including greater access to courses for students, increased applications and enrollments, and the creation of the Center for Learning Design and Technology (CLDT), which provides instructional design and technology. That said, online courses are not the only modality offered at EP. Courses can be completed in a variety of modalities including on-site, Virtual Live (synchronous online learning), online (asynchronous online learning), and industry partnership programs—meeting students where they are.

This evolution at EP faced several challenges. Through these challenges, the program identified opportunities. For example, the university needed to address how to (1) design a curriculum for courses that were to be offered in a variety of modalities and taught by different instructors; and (2) develop online courses that were engaging, rigorous, and interactive for geographically dispersed students—a relatively new phenomenon in higher education in 2001. These challenges and more became the basis for the creation of the gold standard of program and course development at EP.

Establishing a Need for the Gold Standard

The Systems Engineering (SE) program at EP was among the first programs to offer online courses within its curriculum. Not long after, applications and enrollments increased dramatically. Multiple sections of on-site and online courses were offered to meet students demand. This growth placed stress on resources, producing a significant variation in learning outcomes of students, content, and course structure.

Prior to the establishment of the gold standard approach, course materials being developed for the online program were based on the materials of each individual instructor offering the course, resulting in significant inconsistencies across offerings. Since instructors were being used from different sites, the online program experienced gaps and overlaps not experienced at a specific face-to-face site.

Therefore, program leadership conducted an in-depth evaluation of the SE program at EP. The objective of the evaluation was threefold:

1. To review the program content used at all locations where courses were offered;
2. To review program learning goals and ensure the curriculum provided an opportunity for the demonstration of mastery from the students;
3. To produce a set of actionable outcomes to improve the SE curriculum.

Two years after the initial program assessment, the SE Program Chair Dr. Ron Luman enlisted multiple instructors to assess the first course of the program, Introduction to Systems Engineering. This activity brought in perspectives from instructors at different sites and resulted in the first true course baseline recommendations to include topics, course content, and assessments. Since this course was a survey of the entire program, it allowed the evaluators to use material from the initial program assessment and realign core course topics to each of the program’s core courses and advanced electives (see Table 1).

Initial topic-to-course alignment matrix.

Table 1. Initial topic-to-course alignment matrix.

However, the problem remained: How would the SE program align all of its courses, across modalities and locations, being taught by an even larger set of instructors? Course materials needed to be effective in both synchronous face-to-face and asynchronous online environments. Content also need to support 9-, 12-, and 14-week course offerings. This program assessment led to the creation of the gold standard now used for all course development at EP.

Creation of the Gold Standard

An important outcome of the study was the identification of significant divergence in curricula and materials used by various instructors teaching the same courses but in different modalities. For example, the on-site and online courses did not attempt to teach the students the same content topics and concepts. As the program was beginning to expand to other modalities beyond the standard in-class, face-to-face format, the divergence issue and the alignment of program material became a primary focus, serving as the catalyst to create the gold standard.

The gold standard is a model that can help define quality and manage consistency in the design and development of courses that are offered in multiple modalities. Its core components are:

1. Instructional support;
2. Faculty roles and responsibilities;
3. Curriculum and course quality;
4. Configuration control.

Figure 1 demonstrates the relationships among these components.

Figure 1. The “gold standard” model.

Instructional Support

To support the creation of online courses, EP created the Center for Learning Design and Technology in 2001. The CLDT team comprised instructional designers who helped course instructors design and develop online courses in the learning management system. Advancements in educational technology enabled online courses to offer more engaging content and more opportunities for interaction (Berge, 2008; Garrison, Anderson, & Archer, 2001; Moore, 1989). Therefore, EP restructured its instructional support into two centers: the Center for Learning Design (CLD) and the Center for Digital and Media Initiatives (CDMI). (Note that the CLD and CDMI support the Whiting School of Engineering specifically. Other schools at Johns Hopkins University (JHU) have their own centers for supporting faculty.)

Center for Learning Design

The CLD includes instructional designers (IDs) who focus on all facets of learning design, including adult learning theory, best practices in course design, technology, and accessibility of content. IDs also manage all components of the project from start to finish. The CLD also employs JHU students as instructional design assistants to help with lower level tasks, such as PowerPoint conversions to new templates and accessibility checks. See Figure 2 for the current organizational structure.

Figure 2. Center for Learning Design team structure.

It is important to highlight that the current structure of the CLD has evolved; the number of staff has grown, and the ways it supports new course developments and redevelopments to answer the increased demand for new online programs, and to educate/train faculty, have changed. We have also learned that program/course quality and faculty development/support are interconnected and essential to our success (this is captured in section 4.1.3. on Faculty Development). In the following section, the responsibilities of the staff members of the CLD are described.

CLD Roles and Responsibilities

The tasks and responsibilities associated with the roles within the CLD are as follows.

Assistant Dean, Learning Design and Innovation

The assistant dean of learning design and innovation oversees the CLD in conjunction with other tasks, which typically fall into one of five core functions:

• Administrative: Supervision of the IDs, planning, and management of instructional design support for new and redeveloped online courses;
• Strategy: Definition of new paths for EP to meet market and student demand;
• Course quality: Definition of quality standards for all areas of online education, including best practices in pedagogy, instructional design, and technology;
• Faculty development: Generation of opportunties for faculty development;
• Program planning: Curriculum design including program goals and learning assessment plans.

Instructional Designers

The IDs support the development of all new and redeveloped EP online courses, following a systematic instructional design and development process (see section 6.1. on the iADDIE Model). More specifically, IDs “conceptualize the faculty member’s vision for the course and guide him or her in incorporating appropriate instructional strategies and media as the course is developed” (Truman-Davis, Futch, Thompson, & Yonekura, 2000, p. 47). But turning a vision into a high-quality learning experience takes a variety of additional competencies, including good communication, project and time management, and emotional intelligence (Pan & Thompson, 2009).

This high-touch support model involves many tasks for IDs, including:

• Project management for all assigned course development;
• Compliance with Quality Matters standards;
• Implementation of research-based best practices for designing content and courses;
• Alignment among all aspects of the course, including the course learning objectives, instructional strategies, and assessments;
• Selection of appropriate technologies to enhance the teaching and learning experience;
• Production and editing of video content;
• Instructional and graphical design of course materials, such as slides and lecture notes;
• Accessibility of course materials to meet web accessibility standards; and
• Course creation in the learning management system (LMS).

Instructional Design Assistant (Student Worker)

The student Instructional Design Assistants’ duties include:

• Conversion of files to our accessible templates for Microsoft Word and PowerPoint; and
• Following the course accessibility checklist for files.

Center for Digital and Media Initiatives

The CDMI includes instructional technologists (ITs) who help maintain course platforms and services, and support; and multimedia technicians (MTs) who help plan and create custom multimedia assets for courses. See Figure 3 for the current organizational structure.

Figure 3. Center for Digital and Media Initiatives team structure.

Faculty Development

As mentioned above, faculty development is a critical component of the overall quality of course design and teaching. Therefore, both centers focus on creating opportunities for faculty development in teaching and learning, online course design, and innovative uses of technology. These opportunities include the following:

• Edtech Showcases: Technology demonstrations;
• EdTalks: Interactive sessions focused on research-based best practices in teaching, learning, and technology;
• Learning Roadmap for New Online Instructors: Online just-in-time training for faculty working with an instructional designer from the CLD to design and develop an online course;
• EP Knowledge Base: An online resource for faculty containing instructional support resources, guides, and information on the learning technologies used in our courses;
• EP Help Desk: An email-based ticketing system for faculty to receive support for technical problems and/or to gain additional training;
• Live Webinars: A live webinar series offered in connection with the Edtech and EdTalk sessions;
• Faculty Forward Program: This program includes several days of intensive workshop sessions combined with online modules; it models the best practices in teaching and learning, online course design, and innovative uses of technology.

Program Leadership and Instructor Roles and Responsibilities

The success of a model/process is only as good as the collaboration among the people working on the project. Therefore, synergy among the instructional support staff, program leadership, and course instructors is critical to the success of the gold standard model. Below, the roles and responsibilities of the program leadership and course instructors within a program are outlined.

Program Chair/Vice Chair

The program chair or vice chair:

• Is responsible for managing the program of study;
• Establishes the direction for the program as a whole;
• Selects program faculty and instructor staff;
• Engages with other program chairs to maintain alignment between programs of study.

Quality and Assessment Coordinator

The quality and assessment coordinator:

• Applies the program chair’s vision to course development and refresh efforts;
• Reviews course design to ensure program alignment, applicability to multiple modalities, and consistency with the gold standard;
• Assesses course developments and updates mid-cycle to ensure alignment with course plans;
• Performs a comprehensive course quality review on completion of all course developments/updates.

Course Manager

The course manager:

• Leads technical course material development;
• Leads course updates and revisions;
• Maintains configuration control of the course material;
• Revises course templates with updated course information.

Course Instructor

The course instructor:

• Supports the course manager in course development as a subject matter expert (SME);
• Supports the course manager in course redevelopment;
• Delivers the course to the students in the selected modality.

Curriculum and Course Quality

The gold standard contains a systematic instructional design process for developing new courses and redeveloping existing ones. The process is an adaption of the ADDIE (analysis, design, development, implementation, evaluation) model of instructional design, with the addition of a project initiation phase that captures the planning steps involved in the selection of courses to be supported (Branson et al., 1975). The CLD refers to this as the iADDIE model. The six steps of the iADDIE model are outlined below.

iADDIE Model

Initiation of Project

The CLD team supports course development in three separate cohorts: spring, summer, and fall. Each development cohort usually comprises 12–20 new or redeveloped courses across EP. With each new cohort, the assistant dean and program leadership work collaboratively to define which courses in the respective programs will receive instructional design support. For the SE program, once the courses have been selected, a course manager (CM) is identified and partnered with an ID. The CM is typically the lead instructor for the online course and maintains configuration control of all course content. Once identified, the CM establishes a course collaboration site to allow interaction with other course instructors/stakeholders. Allowing course instructors from all teaching modalities to contribute directly to course development results in a higher quality course and facilitates broader acceptance of the material.

Analysis

The analysis phase for new course developments differs greatly from that for course redevelopments. Therefore, each type of development will be detailed separately.

New Course Development

For all new course developments, the ID reviews all existing materials for the course. This enables the ID to gain a better sense of how much work is needed on the development to meet the quality standards. Next, the ID and instructor(s) define a development timeline, typically around 6–9 months from start to finish.

Course Redevelopment

For course redevelopments, the ID will conduct a course quality review (CQR) that involves examining the current materials, reviewing the LMS course site, and analyzing student evaluations. Using this data, the ID will propose redevelopment recommendations for the course. The ID will then meet with the CM to discuss the recommendations and to collaborate on a redevelopment scope. The redevelopment scope captures the overall instructor effort on the course redevelopment. It calculates effort by adding the total hours for each module of the course in which the content needs to be redeveloped. Then, based on total hours, the redevelopment falls into one of four tiers which is tied directly to compensation for redevelopment, as outlined below:

Course redevelopment tiers

Table 2. Course redevelopment tiers.

This redevelopment scope must be agreed upon by all stakeholders (course manager(s), instructional designer, program chair, and assistant dean). Like the new development process, the finalized redevelopment scope leads to the ID and instructor(s) defining a development timeline.

Design

The course design phase involves the creation of a course design matrix (CDM). The CDM ensures that all components of the course are aligned, including the course description, the course learning objectives, the assessment strategy, and the module level instructional strategies. A sample extract from a CDM can be found in Table 3. The ID and instructor(s) work collaboratively to develop the CDM, ensuring that it follows the Quality Matters standards. The Quality & Assessment Coordinator (Q&AC—described later in this chapter) reviews the CDM to ensure program alignment, applicability to multiple modalities, and consistency with the gold standard. If approved, the CDM is considered a working draft and the course development will then move into the development phase.

Course design matrix extract

Table 3. Course design matrix extract.

Development

Using the working draft of the CDM, the instructor(s) and ID collaborate to develop the course materials. This includes the creation of text and video-based content, assessments, and interactions, as well as the curation of any external materials such as open educational resources (OER). Upon completion of the course development, the CDM is updated to capture the final “as built” version of the course.

Implementation

When all course modules are developed, they are uploaded to a university repository area called the institutional content (IC) repository, located in the University’s LMS. A course template is created in the LMS to provide the course shell format. The template course is then linked to the course material in the IC. All course offerings are then copied from the template course to guarantee configuration control. Since the course template and subsequent course offerings are linked to the IC material, all future course shells are automatically updated to include the most current material, allowing the program to maintain complete consistency across all course offerings. Any changes to the course materials are made in the IC area. All course content is marked with a configuration control block to capture relevant metadata (see Table 4). This approach to configuration control is critical to prevent divergence of material over time. See Figure 4 for a graphical representation of the deployed course structure.

Example configuration control block.

Table 4. Example configuration control block.

Figure 4. Deployed course structure.

Evaluation

After the course has been offered for the first time since development or redevelopment, the ID and instructor(s) meet to review and discuss the student evaluations. This may result in proposed revisions to the course to ensure student feedback is incorporated. Student feedback is also reviewed by the program leadership and assistant dean as part of EP’s commitment to high quality.

Quality Matters Course Design Rubric

The iADDIE instructional design allows the process to progress through each phase in a systematic manner. But the process itself does not define the quality of a course. Therefore, the CLDT uses the Quality Matters standards as defined in their rubric (Quality Matters, 2005) to help guide the instructional design and technology decisions within the course. All courses must include all essential recommendations to be considered ready to be offered to students. This enables the CLDT to ensure a level of quality and consistency within all courses.

Program Leadership Reviews

Another layer of quality control occurs throughout the iADDIE process during which program leadership conducts four comprehensive reviews:

1. After the working draft of the CDM is completed, the ID requests a Q&AC review. The Q&AC reviews the course topics, learning objectives, and assessments to ensure they are current with and relevant to the industry.
2. After the successful completion of the prototype module (the first content module), the ID requests a Q&AC review. The Q&AC reviews the structure and material for quality and consistency with the gold standard.
3. At the midpoint of the cycle, the ID requests a Q&AC review to ensure that all materials are being developed as intended in the course design process.
4. After the course development, the ID requests a comprehensive Q&AC review.

Configuration Control

So far in this chapter, we have discussed the instructional support, program leadership, and instructor roles and responsibilities, as well as the importance of these relationships, and how the courses are designed and developed at EP using a systematic instructional design process and quality standards defined by Quality Matters. Now, we will discuss the glue that keeps it all together: the configuration control process.

Course Alignment Matrix

As shown above (see Table 1), the course alignment matrix is updated each time a core course is redeveloped or a new core course is added to the curriculum.

Course Structure

To provide maximum flexibility in course design and development while simultaneously maintaining a consistent format and adhering to the quality standards for all courses in the program curriculum, a course structure was designed. Figure 6 provides a graphical overview of the course structure.

Figure 6. The gold standard course format.

• Course shell: The course shell is the context boundary for an entire course. It includes the course outline, syllabus, instructor information, and course content.
• Module: The module is the “container” for all content relevant to a course topic. To provide maximum flexibility, a module is not associated with a particular week. It is possible to have multiple modules per week, one module that spans multiple weeks, or one module that equals one week’s worth of content. The module is the core element of the course architecture.
• Module overview: The module overview is the “header” element for each module, providing students with a description of the module’s content. When designed properly, one module overview can be adapted to support multiple modalities.
• Presentation: A presentation is the key component of the material being instructed. For online courses, the presentation is typically a professionally developed video. For face-to-face courses, the presentation accompanies a classroom lecture as a primary teaching tool.
• Presentation video: For online courses, each presentation is paired with a video communicating the material to the asynchronous student. Many tools are available to the instructor to produce a professional product, including voice/video embedded in presentations, embedded web links, and embedded quizzes.
• Assessments: Assessments can include assignments, quizzes, discussion questions, exams, and projects. Assessments should be designed for the most stringent application (in most cases online), and a subset can then be allocated to other modalities. In most cases, exams and course projects are separated from learning modules to permit the greatest flexibility when setting up course offerings with different durations (such as 9-, 12-, and 14-week formats).

Results

The development and deployment of the gold standard across the SE program resulted in an improved student learner experience, defined according to the perceptions of students who took courses in the SE program. To measure the student learning experience, we looked at data from two sources: student course evaluations and an alumni survey.

Student Course Evaluations

Mid-term and end-of-term course evaluations are sent to all students each academic term. Mid-term evaluations allow qualitative feedback. End-of-term evaluations provide the opportunity to collect quantitative and qualitative data, and receive a much higher response rate. For these reasons, Figure 7 focuses on quantitative data from the end-of-term evaluations.

Figure 7. Systems Engineering (SE) student evaluation data (fixed average).

Since 2014, despite a slight decrease (0.001) from 2016 to 2017, student evaluation data shows an upward trend in students’ perceptions of their learning experience in the respective course.

Alumni Survey

The alumni survey was used to capture holistic feedback on the learning experience within the program, and was originally distributed every three years. Since 2018, it has been sent annually. The survey questions and results for a specific time period are displayed in Table 5.

Survey questions and answers (2010–2019).

Table 5. Survey questions and answers (2010–2019).

Table 6 presents the percentage increase for each question in the top two answers for each question. In summary, the SE program achieved increases in all areas from the survey that captured students’ responses from 2010 through 2019.

Systems engineering program improvement from 2010 to 2019.

Table 6. Systems engineering program improvement from 2010 to 2019.

Additional Results

Implementing the gold standard achieved additional positive results, including:

1. A perfect score on the Accreditation Board for Engineering and Technology (ABET) review;
2. Alignment of core courses across multiple modalities;
3. Consistent application of materials at all sites and modalities;
4. A consistent look and feel for both students and instructors across the entire program;
5. The ability to rapidly expand the program to new locations and modalities.

Lessons Learned

Lessons were also learned with the development of the gold standard, including the following:

1. Online course delivery is the most challenging format to develop; however, course materials developed for the online program, if designed properly, can be effectively used for other modalities.
2. Tools designed specifically for online learning can also be beneficial for in-class learning (e.g., video lessons, quizzes). In-class students have highlighted the benefit of including these as supplemental materials.
3. Including all instructors teaching a course as reviewers of the course materials is critical to gain buy-in across the instructor base, and promotes proper use of the materials.
4. Re-baselining an entire program of study cannot be done in a single phase. Multiple phases are needed first to build quality materials, then to establish configuration control, and finally to address linkages between courses.
5. As modalities expand, the gold standard can be rapidly applied to new course approaches, such as partnerships. Investing to create high-quality course materials in a consistent and modular format provides great flexibility in the growth of a program of study.

Index of Terms

New online course development

All courses scheduled to be offered online go through the design and development process, supported by instructional support staff.

Online course redevelopment

Existing online courses are redeveloped to ensure content and instructional design approaches are current and relevant.

Course design matrix

The CDM is the artifact that captures all components of the course, including instructional and assessment strategies, course learning objectives, and resources.

Course shell

The course shell created in the learning management system (LMS) is where students will interact with the course content, instructor(s), and fellow students.

Course template

The course template, also created in the LMS, is the centralized location for all content for each course. The course template is maintained and updated by course managers. All course shells are copied from the course template.

Course manager

The course manager leads course content revisions and applies them to the course template.

Course section

A course may be offered multiple times in the same academic term, and in different modalities. Each instance of the course is referred to as a course section.

Module

The module is a container for all content relevant to a course topic.

References

Berge, Z. L. (2008). Changing instructor’s roles in virtual worlds. Quarterly Review of Distance Education, 9(4), 408–414.

Bjerkaas, A. W. (2005). Engineering and applied science programs for professionals. Johns Hopkins APL Technical Digest, 26(3). Retrieved June 23, 2019, from https://www.jhuapl.edu/Content/techdigest/pdf/V26-N03/26-03-Bjerkaas.pdf

Branson, R. K., Rayner, G. T., Cox, J. L., Furman, J. P., King, F. J., & Hannum, W. H. (1975). Interservice procedures for instructional systems development (Vols 1–5). TRADOC Pam 350-30, NAVEDTRA 106A. Ft. Monroe, VA: U.S. Army Training and Doctrine Command.

Council for Higher Education Accreditation. (2002). Assuring Quality in Distance Learning. CHEA Monograph Series 2002, No. 1. Retrieved June 24, 2019, from https://www.chea.org/accreditation-and-assuring-quality-distance-learning

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23.

Johns Hopkins Whiting School of Engineering. (2017, March 22). Programs and courses. Retrieved February 27, 2019, from https://ep.jhu.edu/programs-and-courses/program-pathways/online

Moore, M. (1989). Editorial: Three types of interaction. American Journal of Distance Education, 3(2), 1–7.

Pan, C. C. S., & Thompson, K. (2009). Exploring dynamics between instructional designers and higher education faculty: An ethnographic case study. Journal of Educational Technology Development and Exchange (JETDE), 2(1), art. 3.

Quality Matters. (2005). Quality Matters rubric standards (1st ed.). Retrieved from https://www.qualitymatters.org/qa-resources/rubric-standards

Radford, A. W. (2011). Learning at a distance: Undergraduate enrollment in distance education courses and degree programs. Stats in brief. NCES 2012-154. National Center for Education Statistics. Retrieved June 24, 2019, from https://eric.ed.gov/?id=ED524625

Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. Babson Survey Research Group. Retrieved March 3, 2019, from https://onlinelearningsurvey.com/reports/gradeincrease.pdf

Truman-Davis, T., Futch, L., Thompson, K., & Yonekura, F. (2000). Support for online teaching and learning: The University of Central Florida keeps faculty ahead of the curve with a creative development program. EDUCAUSE Quarterly, 2, 44-51.