Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Open, Online, and Blended: Transactional
Interactions with MOOC Content by Learners in
Three Different Course Formats
Jeffrey P. Emanuel & Anne Lamb
Harvard University
Abstract
During the 2013-14 academic year, Harvard University piloted the use of Massive Open Online
Courses (MOOCs) as tools for blended learning in select undergraduate and graduate residential
and online courses. One of these courses, The Ancient Greek Hero, combined for–credit
(Harvard College and Harvard Extension School) and open online (HarvardX) groups into a
single online unit, marking the first time a MOOC was used simultaneously by both tuition–
paying, credit–seeking students and non–paying, non–credit students enrolled exclusively online.
In this article, we analyze and compare the online behavior of students and participants in three
groups that simultaneously participated in The Ancient Greek Hero via the edX platform. We
find that, in similar fashion to a traditional learning setting, students enrolled in all three versions
of the course engaged the online content in a transactional way, spending their time and effort on
activities and exercises in ways that would optimize their desired outcomes. While user behavior
was diverse, HarvardX participant engagement tended to be either very deep or virtually
nonexistent, while College and Extension School students displayed relatively homogenous
patterns of participation, viewing most of the content but interacting mostly with that which
affected their overall course grades. Ultimately, educators who intend to utilize MOOC content
in an effort to apply blended learning techniques to their classrooms should carefully consider
how best to incorporate each online element into their overall pedagogical strategy, including
how to incentivize interaction with those elements. Further, for MOOCs to have maximum
impact, they must address multiple learner motivations and provide participants with multiple
modes of interaction with the content and with their peers.
Keywords: MOOCs, Blended Learning, Distance Learning, Humanities, technology in higher
education
Emanuel, Jeffrey P. & Lamb, Anne (2017). Open, online, and blended: transactional interactions
with MOOC content by learners in three different course formats, Online Learning 21 (2)
doi: 10.24059/olj.v21i2.845
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Introduction
During the 2013–14 academic year, Harvard University strategically piloted the use of
massive open online courses (MOOCs) as a primary tool for the implementation of blended
learning techniques in select undergraduate and graduate courses offered by Harvard College,
Harvard School of Public Health, and Harvard Extension School. One of these pilots, CB22.1x
The Ancient Greek Hero (Fall 2013), combined the for–credit and open online groups into one
instance of the online platform, rather than creating separate instances for each participating
group to enroll in and utilize, as was done in the other courses. To our knowledge, this pilot was
the first time that a single instance of an existing or in–production MOOC was used
simultaneously by both tuition–paying, credit–seeking students and non–paying, non–credit
students enrolled exclusively online. In this paper, we describe the online behavior of students
and participants in the three groups that simultaneously participated in CB22.1x.
Online and blended learning are neither new nor unfamiliar in higher education (Rooney,
2003; Twigg, 2003; Dziuban, Hartman & Moskal, 2004; Ross & Gage, 2006). However,
MOOCs are still relatively recent phenomena, and educational institutions remain in the early
stages of experimenting with their integration into traditional courses (e.g. Bruff, et al., 2013;
Bebell, Fernandes & Petersen, 2014; Hollands & Tirthali, 2013, 40–41, 124–125; Fowler, et al.,
2014; Picciano, 2014). Thus far, research suggests that learning outcomes are maximized when
MOOCs are utilized as supplements to, rather than wholesale replacements for, traditional
learning experiences. For example, San Jose State saw negative results from their attempt to
replace traditional remedial and introductory courses with online learning experiences, but has
experienced more positive results from blended learning experiments in which select MOOC
content supplements traditional classroom instruction (Ghadiri, et al., 2013).
Since the initial San Jose State experiment, several more recent studies have been
produced which evaluate blended learning experiences, in which MOOCs are only one facet of
content delivery and engagement (Kolowich, 2013a; 2013b; Lewin, 2013; Firmin, et al., 2014;
Bebell, Fernandes & Petersen, 2014). A wide-ranging experiment at the University System of
Maryland, for example, resulted in positive academic outcomes while reinforcing the importance
of the in–person experience (Griffiths, et al., 2014). An experiment in integrating a Stanford
MOOC on machine learning into a residential course at Vanderbilt University, on the other hand,
surfaced as a concern the relationship between online and in-person portions of the class (Bruff,
et al., 2013). These studies and their peers have demonstrated that the strategic incorporation of
MOOCs into traditional courses can produce a range of academic results. These extend from
being on par with standard course offerings (Bowen, et al., 2012; Griffiths, et al., 2014; Yousef,
et al., 2015; Zhao & Ho, 2014) to significantly improving student outcomes in comparison to
non-blended versions of the same course (Ghadiri, et al., 2013). These data points suggest that
MOOCs may most effectively serve as a multimedia resource, such as an interactive textbook,
rather than as a self–contained course in the traditional sense (Bruff, et al., 2013; Emanuel, 2015;
Fowler, et al., 2014; Means, et al., 2013; Yousef, et al., 2015; Zhao & Breslow, 2013).
HarvardX, the organization within Harvard University dedicated to producing and
delivering MOOCs, was founded in 2012. Within the first year of the organization’s existence,
researchers had already documented common patterns among participants, which have remained
largely consistent to the present. Participants in HarvardX MOOCs (also referred to as
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
“learners”) tend to have relatively high levels of education, with over half of the participants in
each course holding a Bachelor’s degree or higher (Ho, et al., 2014). Many sign up for courses,
but far fewer show broad engagement with the content: most engage with less than half the
course content and a majority of participants cease activity within the first week of a learning
experience (ibid). Great diversity exists among the archetypical users (Ho, et al., 2014; DeBoer,
et al., 2014; Reich, et al., 2014). For instance, users may achieve similar grades, while displaying
drastically different amounts of time spent on the content. Some users may view all videos and
engage with all of the readings and assessments, while others may do none of these things but
have the same overall level of satisfaction with the learning experience (Ho, et al., 2014; Reich,
et al., 2014).
This blended learning pilot coincided with the second offering of the Ancient Greek Hero
MOOC, so the initial iteration (CB22x, Spring 2013) serves as a baseline for comparing
demographics, activity, and engagement among online learners. Similar to HarvardX users in
general, CB22x Spring 2013 participants tended to be between 20 and 40 years of age with high
levels of formal education. They demonstrated diverse patterns of online behavior, although over
90% engaged with less than half the content, and only 3.2% of registrants obtained a certificate
of completion (though the certification rate doubles to 5.5% if registrants – those who registered
but never participated – are removed from the equation) (Reich, et al., 2014, p. 10; Emanuel,
2015, p. 267).
Following the pilot study, Bergeron (2014) performed a qualitative analysis of five
courses that used MOOC content in such a way, while Zhao and Ho (2014) utilized a quasi–
experimental approach to identify the causal effect of a flipped classroom approach on student
learning in one of those courses, an offering on Chinese history. While Zhao and Ho found no
substantial difference in outcomes among students in the blended course, both they and Bergeron
suggest that the nature of this pilot as the first of multiple iterations may mean it represents a
lower bound with respect to the effect of MOOCs as tools for blended learning on students’
academic outcomes.
This paper does not attempt to evaluate the overall efficacy of blended learning or of
Harvard’s pilot effort. However, we do intend to contribute additional evidence to the existing
literature on blended learning, MOOCs, and the nexus between the two, for the dual purpose of
(1) informing practitioners and (2) providing data for further research in this space. In this study,
we examine the use of the same MOOC, CB22.1x ‘The Ancient Greek Hero’, across three
different student subpopulations engaged in three different implementation formats. The first is
Harvard College undergraduates taking the course Culture and Belief (CB) 22 in residence for
General Education credit. The second is students at Harvard Extension School (HES) taking the
writing–intensive course CLAS E–116/w for undergraduate or graduate credit. The third is
participants in the edX MOOC (known informally as HeroesX) engaging in the project wholly
online, at no cost and for no formal academic credit. Though all three subgroups were enrolled in
a single instance of the online content, and interacted together via a single integrated online
discussion board, Harvard College and HES students were presented with a blended learning
format that incorporated texts, video, and assessments from the MOOC with in–person or web–
based class meetings. HeroesX participants engaged the online content in an asynchronous, self–
paced learning experience.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Background
In the fall semester of 2013, a team from HarvardX and the Derek Bok Center for
Teaching and Learning at Harvard University’s Faculty of Arts and Sciences partnered with the
professors and teaching teams from five Harvard courses. The purpose of the collaboration was
to investigate the use of existing or in–development HarvardX MOOCs as tools for supporting
blended and activity–based learning techniques in undergraduate and graduate for–credit courses.
The approach supplemented in–class meetings with online multimedia content, including videos,
discussion forums, textbooks, and assessments, with the intent of enriching the learning
experience by increasing the modes by which students interact with the material, and by
repurposing the class time traditionally used for lecture to allow for different types of
engagement (Bergeron, 2014).
The MOOC CB 22.1x ‘The Ancient Greek Hero’ was utilized in an effort to implement a
blended learning approach in the Fall 2013 iteration of the long–running residential course of the
same name. In this approach, students spent considerable time outside of class engaging with
multimedia content (readings, videos, self–assessments, and online forum discussions), thus
allowing in–class time to be spent engaging more deeply with the overall themes of the course,
primarily via a form of Socratic dialogue in which the professor was a participant rather than the
sole source of information and opinion.
The Ancient Greek Hero provides a rich case study in the simultaneous use of a common
platform and content repository by three distinct groups of users. For undergraduate students at
Harvard College (henceforth College), who enrolled in the traditional general education course
for college credit, and for Harvard Extension School (HES) students, including local and online
students taking the course for undergraduate or graduate credit, the MOOC platform provided the
required course textbooks in digital, annotatable format, video dialogues building on the readings,
online forum discussions, and self–assessment exercises. For College students, the online
material supplemented twice–weekly class meetings. HES students were provided videos of the
College class meetings and attended a weekly section meeting either in–person or online,
depending on each student’s physical location (section meetings were omitted from the College
course in 2013 only, and were reinstated in fall 2014 (Bergeron, 2014)). Both the College and
HES versions required written assignments in the form of weekly two–to–three sentence micro–
essays and 500–word position papers for which select self–assessment questions and discussion
topics from the HeroesX platform served as prompts.
The third version of the course consisted of edX registrants (from here, “HeroesX”
learners) taking the MOOC at no cost and for no academic credit. These participants had access
to the same online content as the College and HES students, and interacted with members of the
other groups via the same online discussion board, but they did not have class meetings, sections,
or other exercises to supplement their experience. Online participants could achieve “Honor
Code Certificates” in recognition of their level of participation by achieving a cumulative score
of 50 percent or greater on the online assessments.
Structurally, the MOOC’s content was divided into 24 chapters, or top–level units of
content (referred to in CB22.1x as “Hours” for their reflection of the classroom time in an
academic semester and, aptly, of the units in Homer’s epics). In addition, two opening segments
were provided, titled “Welcome” and “Introduction” (the latter of which included practice
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
exercises, though they did not count toward final grade or certification), as well as a conclusion,
for 27 chapters. The entirety of this content was made available to all three groups of learners
(College, HES, and HeroesX) on the first day of the course, allowing for asynchronous access
and engagement. College and HES students followed a structured syllabus, with specific
readings, assignments, and due dates each week. Although this syllabus was presented as an
“idealized schedule” to MOOC participants who wished to follow along with their for–credit
counterparts, HeroesX users were both allowed and explicitly encouraged to engage with the
content at whatever pace, and in whatever depth, was most comfortable to them.
Demographic Characteristics of Participants
Harvard College has offered The Ancient Greek Hero for four decades. The MOOC version of
this course was offered for the first time in spring 2013, with 43,563 students enrolling (Reich, et
al., 2014). Enrollment was lower in the fall 2013 second iteration, with 20,598 HeroesX
participants, and a for–credit population of 237 students from the College and 37 from HES.
Immediately below, we briefly describe the demographics of the population participating in this
project.
Age
In both iterations of HeroesX, the median age was 29, close to the HarvardX average
median age of around 28 years. HeroesX users who attained certificates of completion were, on
average, slightly older than the average age of all HeroesX participants. While HES students
were similar in age, on average, to HeroesX participants, students enrolled in the College version
unsurprisingly tended to be younger than their HES and HeroesX counterparts (see Figure 1).
Gender
Gender balance varies widely between edX courses, although most courses offered by
HarvardX and MITx to date have tended to have higher proportions of male students. In both the
spring 2013 and fall 2013 iterations of HeroesX, gender distribution was nearly equal. Slightly
more male HeroesX students earned certificates in the fall iteration, although the opposite was
the case in the prior offering (Reich, et al., 2014). In the College version of the course, male
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
students outnumbered females by a nearly 2 to 1 margin, while females outnumbered males by
the same proportion in the HES version (see Figure 2).
Education
Researchers administered a pre-course survey to users in all three groups in an effort to
shed light on participant information and motivation. In response to this survey, HeroesX
participants self–reported as having higher overall levels of formal education than both College
and HES students, a trend also seen in previous MOOCs (Ho, et al., 2014; see Figure 3). Almost
two–thirds of HeroesX enrollees had previously earned a Bachelor’s degree or higher. HES
students had, on average, higher levels of formal education than the College students (over 90%
of whom, unsurprisingly, had previously earned a high school diploma at most), but less than
participants enrolled in HeroesX.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Categorization of Participation
Great heterogeneity has been observed in the behavioral patterns of MOOC participants
(Kizilcec, Piech & Schneider, 2013; Anderson, et al., 2014; DeBoer, et al., 2014; Ho, et al.,
2014; cf. Yuan & Powell, 2013; Liyanagunawardena, Parslow & Williams, 2014). Because
students have a variety of personal motivations for enrolling and participating in MOOCs, their
activity on the edX platform (watching videos, reading texts, completing assessments,
participating in forum discussions, etc.) can vary widely. Despite the vast number of individual
patterns of activity, Ho, et al. (2014) employed a simple categorization of participation patterns
among HarvardX users based on the extent to which users engaged in course content:
•
•
•
•
Registrants, who enroll in the course, but do not engage with any of the content;
Viewers, who enroll and engage with less than half of the content;
Explorers, who enroll and engage with more than half the content; and
Certificate Earners, who enroll and who meet the requirements to obtain certification.
As Figure 4 shows, each of these categories is made up of a subset of its antecedent. For
HeroesX participants to qualify for a certificate of completion (officially referred to as an “Honor
Code Certificate”), they were required to achieve an average of 50% or above on the assessments
included in the courseware. Scoring in the MOOC was solely dependent on these assessments,
which consisted of four multiple–choice content questions (worth one point each) and four
multiple–choice annotation questions based on close readings of primary texts that accompanied
each of the 24 “Hours.”
In Figure 5, we apply this categorization to the Fall 2013 version of The Ancient Greek
Hero. The grades of HeroesX, College, and HES participants are plotted against the number of
total chapters they viewed out of 27, using the above paradigm to describe users based on the
extent of their course participation. Participants who signed up for HeroesX but viewed no
chapters are the registrants. The lower left quadrant consists of viewers, who examined less than
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
half the content. The lower right quadrant shows the explorers, who viewed more than half the
chapters but did not obtain certification. The certificate earners are those in the right upper
quadrant.
Where do College and HES students fall within this categorization paradigm? The plus
signs (+) and open circles (o) in Figure 5 denote College and HES students, respectively.
Overwhelmingly, students in these two versions of the course viewed over half the content,
making them primarily explorers and certificate earners. Very few in either group viewed less
than half the chapters, and no students from these versions merely registered (compared to over
one–quarter of online users who remained only registrants, never engaging the MOOC content at
all). Interestingly, while explorers comprise the largest percentage of College and HES students,
the smallest proportion of HeroesX participants – just 1% – fall into this classification, with
registrants (34%), viewers (62%), and certificate earners (3%) far outnumbering them. This
suggests that College and HES students kept with the requirements of their respective credit–
bearing courses, as might be expected in a traditional educational environment, compared to the
nontraditional ways in which MOOC participants engage online content.
To compare the activity patterns of the for–credit students with the HeroesX participants,
we conducted a principal components analysis. To measure participation patterns, we include
participants’ final course grades, the number of chapters and videos viewed, their number of
forum posts, and the number of days active. The first principal component, which accounts for
over half of the variance in the participation construct, is primarily characterized by higher
grades, higher numbers of days active, and more chapters viewed (see loading plot displayed in
Figure 6). The second component, which accounts for about one-fifth of the variance in the
participation construct, seems to measure the number of videos watched.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Using these two components, in Table 1 we show mean principal component scores by
participant group. The mean scores suggest that HeroesX participants who attain certification
tend to have the highest number of chapters viewed, the highest grades, and highest numbers of
days active, while HeroesX Viewers have the lowest.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Comparing the difference in mean scores, HES and College students appear most like
each other in their mean scores for both components 1 and 2 (see Table 1 mean component
scores). When comparing the mean scores of HES and College students to HeroesX participants,
HES students appear most similar in their participation patterns to HeroesX explorers (diffCertified
= 1.132 v. diffExplorers = .944), while College students appear somewhere in between HeroesX
explorers and certificate earners (diffCertified = 0.714 v. diffExplorers = 1.362). Throughout the
remainder of this article, we explore patterns of participation along several dimensions, and
further compare activity among HES, College, and HeroesX participants.
Patterns of Participation
Although students in all three versions of The Ancient Greek Hero used the same edX
instance, their course requirements, motivations, and performance incentives differed in critical
ways. As a result, differences in their use of the MOOC platform were expected. College and
HES students took The Ancient Greek Hero for undergraduate or graduate credit; for the former,
the course fulfilled a general education requirement, while for HES students, it fulfilled a
writing–intensive requirement. Only 7 out of 237 College students were concentrating in a
Humanities-related field, and only one student in the entire class self–identified as a Classics
concentrator. Because earning college credit is a key motivator, we might expect these students
to use the MOOC platform in a transactional way, focusing on the content and interactions that
had the greatest impact on their final grades.
Conversely, HeroesX participants were not eligible for college credit in exchange for
successful completion of the MOOC. These enrollees registered and participated for a variety of
reasons: Around one–fifth of registrants expressed the desire to “engage in lifelong learning,”
according to the pre–course survey and approximately the same number wanted “to learn from
the best professors and universities.” Around 13% of HeroesX respondents said at the outset of
the MOOC that they planned on “completing enough course activities to earn a certificate,”
while nearly 16% of HeroesX users intended to complete most of all of the readings and 15%
intended to complete most or all of the assignments.
Although their credit statuses differed, students in all three versions of CB22.1x likely
overlapped to some degree in their internal motivations for taking the course. HES and HeroesX
users were more likely to take the course in addition to full–time jobs or family responsibilities,
while College students chose the course from among a variety of possible course electives. Thus,
all participants were to some degree interested in, and motivated by, the content of the course.
Whether this internal motivation dominated external motivations in driving individuals’ actions
is a matter of interpretation of the empirical evidence.
Structure and Participation
All of the course’s content was housed on the MOOC platform, including videos, self–
assessments, the shared discussion board, and the two textbooks used in the course, The Ancient
Greek Hero in 24 Hours (Nagy 2013) and the Sourcebook of Ancient Greek Texts Translated
into English. As a result, participants from all three groups had to create (free) user accounts to
access the required material.
College and HES students’ grades were dependent on their interactions with the
platform content – specifically, the online assessment questions – but not in the same way that
HeroesX participants’ scores were. Rather than completing the eight multiple choice questions
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
provided with each Hour of online content, College and HES students were required to write
brief essays in response to selected questions from those exercises. Each week, these students
submitted one 500–word “position paper,” which was to be written in response to a pre–selected
question (or set of questions) from the eight annotation exercise questions included in the two
Hours that made up that week’s online content. The position papers comprised 39% of College
students’ final grades, and 50% of HES students’ grades. Additionally, students in both for–
credit groups were responsible for 22 two–to–three–sentence “micro–essays” over the course of
the semester (two per week, beginning in the second week of class), which were to be posted on
the online discussion board on the edX platform. The micro–essays made up 22% of College
students’ final grades, and the other 50% of HES students’ grades. The students responded by
writing micro–essays that attended to a stand–alone discussion question provided at the end of
each hour. For logistical reasons, students were also required to email their papers and micro–
essays to the appropriate members of each course’s teaching staff. As a result of the latter policy,
students were not held strictly accountable for first posting their micro–essays to the MOOC
discussion board, and because of this, many decided to forego the extra step of posting to the
forum altogether.
Conversely, HeroesX students did not have a writing requirement; to earn certificates of
completion, they only had to demonstrate their mastery of the concepts and content presented in
the MOOC by achieving a score of 50% or greater in the online self–assessments. Online
students’ grades depended exclusively on their performance on the two quizzes that accompanied
each hour. Users could attempt each assessment question only once, and could receive half or
full credit (1 or 2 points) for the annotation exercises.
Given their differences in motivations, incentives, and course requirements, we might
expect HeroesX users to show different patterns of usage than students in the for–credit classes.
For example, HeroesX students might engage the multimedia materials to a greater degree,
particularly with the videos, which were meant to serve not as a centerpiece of the course’s
content, but as a supplement to, and extension of, the focal point of the learning experience for
all three groups: the texts. Additionally, in the absence of a traditional class meeting, HeroesX
participants might engage with the online forums more than College and HES students.
In Table 2, descriptive statistics for participation and performance are displayed for
students in each version of The Ancient Greek Hero. As expected, College and HES students’
patterns of participation with the edX platform often differed from those of online participants.
For instance, although the number of course chapters viewed by students in the College and HES
versions was similar to those in the online version (median of 25 chapters viewed for
College/HES students compared to a median of 23 chapters viewed for explorers and 27 chapters
viewed for certified HeroesX participants), students in the former versions often had higher
numbers of active days compared to students in the online version (median of 51/50 days active
for College/HES students versus a median of just 17 days active/28 days active for online
participants who explored/were certified).
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Table 2 Summary statistics of participation and performance among exclusive categories of user groups
College and HES students had similar patterns of performance in comparison to online
explorers and certificate earners according to other measures as well. For instance, College
students and online explorers had very similar rates of correct assessment questions, with median
proportions of 16% and 14% total questions correct. Moreover, when considering the proportion
of problems they correctly answered among those they attempted, College, HES, online
explorers and online certificate earners had remarkably similar median statistics (ranging from a
median low of 50% correct among online explorers to a median high of 65% correct among
online certificate earners, with College and HES students’ medians falling in between.
Because College and HES students’ institutional grades did not depend on the online
assessments, and because these students had no requirement to achieve certification in the edX
learning experience, the distributions of grades and problems attempted differed between these
groups and HeroesX learners in key ways. In Figure 7, the grade distributions for all HeroesX
participants for HeroesX explorers and certificate earners and for HES and College students
appear. In both HeroesX distributions, a noticeable discontinuity exists at the certification cutoff
of 50% (indicated by the vertical line). This discontinuity could result from “optimizers,” or
those users who are most incentivized by the certificate as their end goal, who viewed just over
half the content and completed just enough assessment questions to earn certification.
The HeroesX grade distribution also includes a spike for users who had perfect or near
perfect grades. These “completers” may be online users who were most interested in lifelong
learning (as evidenced by viewing all chapters and completing all assessment questions) and who
wanted the certificate to demonstrate such learning (as evidenced by the high grade). Neither the
discontinuity nor the spike of perfect grades appears in the distributions for the College and HES
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
users, who had no incentive to answer the self–assessment questions correctly or to secure an
online score of above 50%.
In general, as shown in Figure 8, a positive relationship exists between the number of
active days on the MOOC site and participants’ final online score (the grade resulting from the
online self–assessments, not the institutional grade administered to students in the for–credit
courses). This remains true until around the 75–day mark, at which point increasing the number
of active days begins to show a negative return (cf. Nesterko & Seaton, 2014). At the same time,
among HeroesX participants who earned certification, those with very few days active were
nearly as likely to earn certification as those with many days active, as shown by the relatively
flat trend line in Figure 8, and by the scatterplot in Figure 9.
Online Discussion
Online discussion was a key component of The Ancient Greek Hero, particularly for
HeroesX users. The HeroesX discussion board provided the primary mechanism for fostering
discussion among participants from all versions of the course. In fact, promoting conversation
within and between all three groups was a key factor in the decision to utilize a single instance of
the MOOC for the two for–credit courses and for HeroesX. To help promote this dialogue,
College and HES students were explicitly incentivized to initiate online discussions by posting
each of their 22 “micro–essays,” which counted toward their final grades, to the discussion board.
However, because of technical difficulties in searching the forums and identifying posting
histories by user, these “micro–essays” were also emailed to the appropriate teaching fellow(s),
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
thus allowing students to bypass the forums despite the official requirement that they post.
HeroesX participants, as noted above, had no external incentive for posting to the discussion
board, as it was not taken into consideration for scoring or certification purposes. However, we
might expect to see those who were primarily motivated by the opportunity to engage in a
community learning experience post to the discussion board with some frequency.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
In Figure 10, the distributions of the number of discussion board posts appear for
HeroesX certificate earners, College, and HES students. The modal number of discussion posts
was zero for HeroesX certificate earners and HES students, and three posts for College users.
This, in combination with the fact that the HES and College syllabi required 22 posts each,
demonstrates the extent to which the option to email “micro–essay” responses directly to the
teaching staff trumped the requirement to also post these brief responses to the HeroesX
discussion board. Overall, the College distribution is closer to a normal distribution than either
the HES or online distributions.
For the most part, except for a small percentage of outliers, students from all groups
seemed to use the discussion board in a transactional way. Because it was required as part of
their final grades, College and HES students posted the most on average, though the email
submission option de–incentivized posting to a sufficient degree that, as noted above, both
groups’ median and mean posting output was below the amount specifically required in their
respective syllabi. Interestingly, HeroesX participants, who in the absence of a traditional
academic community may have stood to benefit most from the discussions, posted the least on
average. At the same time, although College students post the most on average, HeroesX viewers
– participants who enrolled and engaged with less than half of the content – actually accounted
for the majority of users who ever contributed posts to the discussion board, with just over 50%
of those who posted falling into this category. Around 20% of those who posted were College
students, and 20% certificate earners.
Patterns of Assessment Behavior
Differences in incentives may also account for differences in patterns of quiz taking. In
the first panel of Figure 11, the distributions of problems attempted appear for three subsets of
HeroesX participants – those who expressed the intent to earn a certificate in response to the pre–
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
course survey, explorers, and certificate earners. Those participants who indicated that earning a
certificate was important on the survey did not demonstrate behavior in line with their stated
intentions, as nearly two–thirds did not attempt any quiz problems. On the other hand, almost
60% of explorers attempted more than one–tenth of the problems, while over 70% of HeroesX
participants who eventually earned certification attempted nearly all of the problems.
Figure 11. Distribution of the proportion of problems attempted out of total problems for
HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX
motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n =
659; College n = 237; HES n = 37).
HES and College students also had an incentive to attempt specific self–assessment
problems. Each week, a selected annotation exercise question (or, in some cases, multiple
questions) was identified as the prompt for students’ “position paper” essays, and students were
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
informed ahead of time which question(s) that would be. Students could take questions
selectively, and since attempting a question, (whether the response was right or wrong) would
display both the correct answer and a paragraph–length explanation from the professor why the
answer was correct and the other options were incorrect. Because of this, it was beneficial for
students to at least attempt the self–assessment questions on which each week’s essay was based.
In Table 3, we provide evidence to support this behavior among College students. While
HeroesX participants attempted around 30% of questions (essay prompts did not apply to this
group), College students attempted nearly twice as many questions that served as essay prompts
as they did non–essay prompt questions.
Table 3 Comparison of proportion of questions attempted, grouped by questions that served as
essay prompts or did not
Group
Total Questions
Essay Questions
Non-Essay Questions
HeroesX Viewers
31%
25%
32%
Harvard Extension
52%
57%
52%
Harvard College
39%
67%
35%
Note. Sample includes 2,622 HeroesX participants, 32 HES students, and 229 College students.
To explore this behavior, we plotted the percentage of essay prompt versus non–essay–
prompt questions attempted by two groups, College students and HeroesX participants, and
included a fitted linear function (see Figure 12). The left panel reflects the actual behavior of
HeroesX participants; we expect the curved functional form given the uneven distribution of
essay versus non-essay prompts throughout the course. At most, one of eight questions was an
essay prompt for each hour, although some of the early hours included no essay questions.
Therefore, conditional on attempting a certain total proportion of questions, more of those
questions were likely to be non-essay questions than essay questions. Many of the HeroesX
participants also attempted no questions or attempted all questions. Thus, the overall pattern for
HeroesX participants, who had no incentive to answer questions selectively, is that they ended up
attempting more non-essay questions. Conversely, College students attempted substantially more
essay–prompt questions than they did the remaining 70% of assessment questions, suggesting
that they did extensively engage in selectively attempting the questions that were most
incentivized toward their end-of-course grades.
As noted above, the number of assessment questions engaged by the three broader groups,
and by the users within the HeroesX population, varied widely, from a median of 28% of total
questions attempted by College students, to 0% attempted by HeroesX participants overall, to
97% attempted by HeroesX certificate earners (see Table 3). Interestingly, among those
questions that were attempted, HeroesX participants’ patterns of success were similar to those of
College and HES students (see Figure 13). Thus, in comparison to their College and HES
counterparts, HeroesX certificate earners seem to have achieved certification as a result of
attempting most or all of the questions, while HES and College students may have failed to
obtain certification as a result of not attempting as many problems, rather than because they
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
answered the questions they took incorrectly. These patterns reinforce two important points: that
edX certification signifies the completion of the requirements to a stated standard rather than
mastery of the content and that failure to obtain certification may not indicate failure to master
the content and skills of the course.
Figure 12. Proportion of essay prompt questions attempted against non–essay prompt questions
attempted, for both College and HeroesX students who attempted at least one quiz problem
(HeroesX n = 2,622; College n = 229). HES plot is not included, but shows relatively
homogenous scatter along the diagonal.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Figure 13. Distribution of proportion of problems correct out of total problems attempted for
HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX
motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n =
659; College n = 237; HES n = 37).
Conclusion and Implications
Three key themes emerged from our study of patterns of learner behavior. First, despite
theoretical aspirations of MOOCs offering both breadth and depth for engaging in the online
learning space, students in all three versions of the course utilized the edX platform in a
transactional way, spending their time and effort on activities and exercises in ways that would
optimize their grades. Second, as cautioned by prior researchers (e.g. Ho, et. al., 2014),
certification in the HarvardX space does not imply a specific set of behaviors. It is a binary
outcome representing a multidimensional concept. While some certified students spend
significant time engaged in course content, carefully completing quizzes, and engaging with the
community discussion boards, others seem to engage only with quizzes, over the course of very
few active hours or days. Given this, we should note that certification in a MOOC does not
necessarily equate to learning, nor does it necessarily indicate a similar level of content and skill
mastery as participants in a traditional course (and vice versa for MOOC participants who elect
not to pursue certification). Likewise, we should be cautious when attempting to compare online
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
certification and scores to for–credit course completion and grades. Third, as in many HarvardX
learning experiences, while user behavior was diverse, HeroesX participants generally trended
toward one end of the participation spectrum or the other, either registering and viewing less than
half the content (“registrants” and “viewers”), or immersing themselves in the majority of the
content and earning a certificate of completion (certificate earners). College and HES students
displayed relatively homogenous patterns of participation, viewing most of the content but on
average, not obtaining the edX certificate of completion. These students produced patterns of
behavior most similar to a minority of HeroesX participants, which we call explorers (see below
for further explanation of these terms).
Similar to a traditional learning setting, students from all three versions of The Ancient
Greek Hero spent the time and effort necessary to maximize their desired outcomes. In other
words, both for–credit students and HeroesX participants engaged the online content in ways that
were best aligned with the incentives they were provided (either internally or externally). Online
users who achieved certification spent a higher percentage of time attempting problems relative
to other users, as the online assessments were their sole mechanism for achieving that goal.
College and HES students, on the other hand, chose to spend time and effort on activities which
best complemented the offline components of their respective courses, allowing them to
maximize their own overall performance (cf. Bergeron 2014: 4, 11, 16–18). Other than engaging
the textbooks and essay prompts made available on the HeroesX platform, this meant spending
comparatively little time and effort online.
We draw a few practical conclusions from this analysis. First, educators intending to
utilize MOOC content in an effort to apply blended learning techniques to their classrooms
should carefully consider how best to incorporate each online element into their overall
pedagogical strategy, including how interaction with those elements is to be incentivized (cf.
Nazarenko, 2015; Cheng, et al., 2017; Sharaj, et al., 2017). In the case of Culture and Belief 22
(Harvard College) and CLAS E–116/w (Harvard Extension School), the professor and teaching
fellows ensured that students interacted with some of the self–assessment questions by basing
essays on them. This spurred more discussion than might otherwise have taken place by
requiring use of the forums by for–credit students (though, as noted above, technical limitations
curtailed this). Some other MOOCs use different tactics to stimulate interaction between
participants, such as counting forum posts as part of enrollees’ grades. EdX examples of this
include AmPoX.1 Poetry in America: The Poetry of Early New England, SW12x China,
HDS1544.1x Early Christianity: The Letters of Paul, and USW30x Tangible Things:
Discovering History through Artworks, Artifacts, Scientific Specimens, and the Stuff around You.
Additionally, one of the courses participating in the fall 2013 pilot, Science of the Physical
Universe (SPU) 27 Science and Cooking, required students to post their final project to the
online discussion board. In many other cases, courses forgo this altogether, instead using online
discussion boards more as a tool of instructor–student communication or for crowdsourcing
questions and problems; examples of this include, among many others, MCB80x Fundamentals
of Neuroscience (edX), Understanding Research: An Overview for Health Professionals
(Coursera), and Linear Circuits (Coursera). In the case of HeroesX, community engagement and
discussion was a core component of the overall project’s aims, and one of the chief reasons
College and HES students were included in the same instance of the MOOC as HeroesX learners
(Reich, et al., 2014: 19; Emanuel, 2015). In light of this, one aspect of the pilot, which was not
considered in the present study, is what effect the three learner populations may have had on
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
each other because of their sharing of a single instance of HeroesX. This question may be a
potential avenue for future research.
Second, for MOOCs to have maximum impact, they must address multiple learner
motivations and provide participants with multiple modes of interaction with the content and
with their peers. As is becoming increasingly emphasized in the literature on MOOCs,
certification and completion are not the only outcomes that designers and educators should
consider when designing online learning experiences or when measuring outcomes (Koller, et al.,
2013; Lizilcec, Piech & Schneider, 2013; Stewart, 2013; Yuan & Powell, 2013; Anderson, et al.,
2014; DeBoer, et al., 2014; Ho, et al., 2014; Liyanagunawardena, Parslow & Williams, 2014;
Emanuel, 2015). Researchers and educators must continue to experiment with more flexible
ways to measure engagement and learning both in MOOCs and in blended, hybrid, and flipped
credit–bearing courses – particularly those that utilize MOOCs and their content as primary
resources. In order to do this, practitioners must first gain a better understanding why participants
are enrolling and participating in MOOCs in the first place, and, insofar as it is possible, think
about alignment of learner outcomes with their own motivations. This is an area that is rich with
future research potential, as certain key design concepts may work better in making the content
of MOOC, online for-credit, and residential courses more useful and engaging to students with
different sources and levels of motivation. For instance, some MOOCs, particularly in STEM
fields, allow students to make multiple attempts in order to get the correct answer on quiz
questions, emphasizing learning the material eventually rather than only the first time (cf. Han,
Veeramachaneni & O’Reilly, 2013; Kortemeyer, 2014). HarvardX is also relaunching past
courses with different formats – including The Ancient Greek Hero, which was offered in fall
2014 as a series of five discrete modules, rather than as a single contiguous learning experience –
in an effort both to allow learners to better self–direct their experiences, and to gain a better
understanding of user motivation and engagement. Further studies are possible as more data
points in this area become available, both within this specific learning experience and across the
MOOC and blended learning landscape.
Massive Open Online Courses have the potential to serve both students in blended
learning courses and learners worldwide who are internally motivated to engage with and learn
about the content on their own. Practitioners involved in the online learning space are only
beginning to imagine the possibilities for MOOC design and experimentation to help meet
learners’ needs. Over time, we can further conduct evidence-based redesigns and reimaginings of
learning experiences in the pursuit of dynamic learning experiences and improving learning for
students in all contexts.
References
Anderson, A., Huttenlocher, D., Kleinberg, J. & Leskovec, J. (2014). Engaging with massive
online courses. Paper presented at the 23rd International World Wide Web Conference,
Seoul, Korea.
Bali, M. (2014). MOOC Pedagogy: Gleaning good practice from existing MOOCs. Journal of
Online Learning and Teaching, 10(1), 44–56.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Bebell, D., Fernandes, K. & Petersen, R. (2014). Applying MOOCs in on–campus settings:
Opportunities, obstacles, and results. Paper presented at the ELI Annual Meeting, New
Orleans, LA.
Bergeron, J. M. (2014). Blended learning in the college. Cambridge, MA: Harvard University,
Derek Bok Center for Teaching and Learning.
Bowen, W.G., Chingos, M. M., Lack, K. A. & Nygren, T. I. (2012). Interactive learning online
at public universities: Evidence from randomized trials. Ithaca S + R Report.
Bruff, D. O., Fisher, D. H., McEwen, K. E. & Smith, B. E. (2013). Wrapping a MOOC: Student
perceptions of an experiment in blended learning. MERLOT Journal of Online Learning
and Teaching, 9(2), 187–199.
Cheng, H. N. H., Liu, Z., Sun, J., Liu, S. & Yang, Z. (2017). Unfolding online learning
behavioral patterns and their temporal changes of college students in SPOCs. Interactive
Learning Environments, 25(2), 176–188.
DeBoer, J., Ho, A. D., Stump, G. S. & Breslow, L. (2014). Changing “course”:
Reconceptualizing educational variables for Massive Open Online Courses. Educational
Researcher, 43(2), 74–84.
Dzubian, C. D., Hartman, J. L. & Moskal, P. D. (2004). Blended learning. ECAR Research
Bulletin, 2004(7).
Emanuel, J. P. (2015). Massive open online opportunity: MOOCs and Internet-based
communities of archaeological practice. In F. Djindjian, L. Costa, P. Moscati & S. Robert
(Eds.), 21st Century Archaeology: Concepts, Methods, and Tools (pp. 265–270). Oxford,
UK: Archaeopress.
Firmin, R., Schiorring, E., Whitmer, J., Willett, T., Collins, E. D. & Sujitparapitayae, S. (2014).
Case study: Using MOOCs for conventional college coursework. Distance Education,
35(2), 178–201.
Fowler, R., Meinking, K. A., Morrell, K., Sandridge, N. & Walker, B. (in press.) Adapting
content from a Massive Open Online Course to a liberal arts setting. Transformations.
Ghadiri, K., Qayoumi, M. H., Junn, E., Hsu, P. & Sujitparapitaya, S. (2013). The transformative
potential of blended learning using MIT edX’s 6.002x online MOOC content combined
with student team-based learning in class. Japanese Universities Association for
Computer Education Journal, 3, 1–15.
Griffiths, R., Chingos, M., Mulhern, C. & Spies, R (2014). Interactive online learning on
campus: Testing MOOCs and other platforms in hybrid formats in the University System
of Maryland. Ithaka S+R Report. New York, NY: Ithaca.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Han, F., Veeramachaneni, K. & O’Reilly, U–M. (2013). Analyzing millions of submissions to
help MOOC instructors understand problem solving. Paper presented at the NIPS
Workshop on Data Driven Education, Lake Tahoe, NV.
Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014).
HarvardX and MITx: The first year of open online courses. HarvardX/MITx Working
Papers 1.
Ho, A. D. & Reich, J. (2014). The tricky task of figuring out what makes a MOOC successful.
The Atlantic, 23 Jan. http://www.theatlantic.com/education/archive/2014/01/the–tricky–
task–of–figuring–out–what–makes–a–mooc–successful/283274/.
Hollands, F. M. & Tirthali, D. (2013). MOOCs: Expectations and reality. New York, NY:
Palgrave Macmillan.
Hollands, F. M. & Tirthali, D. (2015). MOOCs in higher education: Institutional goals and paths
forward. New York, NY: Palgrave Macmillan.
Koller, D., Ng, A., Do, C. & Chen, Z. (2013). Retention and intention in Massive Open Online
Courses: In depth. Educause Review Online, 3 June.
http://www.educause.edu/ero/article/retention–and–intention–massive–open–online–
courses–depth–0.
Kolowich, S. (2013a). San Jose State U. puts MOOC project with Udacity on hold. The
Chronicle of Higher Education, 19 July.
Kolowich, S. (2013b). The MOOC ‘revolution’ may not be as disruptive as some had imagined.
The Chronicle of Higher Education, 8 Aug. http://chronicle.com/article/MOOCs–May–
Not–Be–So–Disruptive/140965/.
Kortmeyer, G. (2014). How to reduce unproductive and undesirable behavior in online courses.
Paper presented at the AAPT summer meeting, Minneapolis, MN.
Lack, K. A. (2013). Current status of research on online learning in postsecondary education.
Ithaka S + R Report. New York, NY: Ithaca.
Lewin, T. (2013). After setbacks, online courses are rethought. The New York Times, 10 Dec.
Lizilcec, R. F., Piech, C. & Schneider, E. (2013, April). Deconstructing disengagement:
Analyzing learner subpopulations in Massive Open Online Courses. Paper presented at
the Third International Conference on Learning Analytics and Knowledge, Leuven,
Belgium.
Liyanagunawardena, T. R., Parslow, P. & Williams, S. (2014). Dropout: MOOC participants’
perspective. In U. Cress & C. D. Kloos (Eds.), Proceedings of the European MOOC
stakeholder summit 2014 (pp. 95–100).
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Major, C. H. (2010). Do virtual professors dream of electric students? University faculty
experiences with online distance education. Teachers College Record, 112(8), 2154–2208.
Means, B., Toyama, Y., Murphy, R. F. & Bakia, M. (2013). The effectiveness of online and
blended learning: A meta–analysis of the empirical literature. Teachers College Record,
115(3), 1–47.
Nazarenko, A. L. (2015). Blended learning vs traditional learning: What works? (A case study
research). Procedia – Social and Behavioral Sciences, 200, 77–82.
Nesterko, S., & Seaton, D. T. (2014). Understanding user engagement through time on site in
HarvardX MOOCs. Unpublished manuscript.
Picciano, A. (2014). Blended learning meets MOOCs: Education’s digital future. Paper
presented at the ELI Annual Meeting, New Orleans, LA.
Reich, J., Emanuel, J. P., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I., & Ho,
A. D. (2014). HeroesX: The Ancient Greek Hero: Spring 2013 course report.
HarvardX/MITx Working Papers 3.
Rooney, J. E. (2003). Knowledge infusion: Blended learning opportunities to enhance
educational programming and meetings. Association Management, 55(6), 26–32.
Ross, B. & Gage, K. (2006). Global perspectives on blended learning: Insight from WebCT and
our customers in higher education. In C. J. Bonk & C. R. Graham (Eds.), Handbook of
blended learning: Global perspectives, local designs (pp. 155–168). San Francisco, CA:
Pfeiffer.
Sharaj, A. P., Pattankar, R., Lochan, P. R. & Murthy, M. (2017). Hybrid learning: A paradigm
shift. Journal of Engineering Education Transformations, 2017, 1–6.
Stewart, B. (2013). Massiveness + openness = new literacies of participation? Journal of Online
Learning and Teaching, 9(2), 228–238.
Twigg, C. A. (2003). Improving learning and reducing costs: New models for online learning.
Educause Review, 38(5), 28–38.
Yousef, A. M. F. (2015). Effective design of blended MOOC environments in higher education
(Unpublished doctoral dissertation). RWTH Aachen University, Germany.
Yousef, A. M. F., Chatti, M. A., Schroeder, U. & Wosnitza, M. (2015). A usability evaluation of
a blended MOOC environment: An experimental case study. International Review of
Research in Open and Distributed Learning, 16(2), 69–93.
Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats
Yuan, L. & Powell, S. (2013). MOOCs and open education: Implications for higher education –
a white paper. Bolton, England: Centre for Educational Technology & Interoperability
Standards.
Zhao, Y. & Breslow, L. (2013). Literature review on hybrid/blended learning. HarvardX and
MIT Teaching & Learning Laboratory.
Zhao, Y. & Ho, A. D. (2014). Evaluating the flipped classroom in an undergraduate history
course (HarvardX Research Memo).