Open, Online, and Blended: Transactional Interactions with MOOC Content by Learners in Three Different Course Formats Jeffrey P. Emanuel Anne Lamb jemanuel@fas.harvard.edu HarvardX Harvard University anl443@mail.harvard.edu Graduate School of Education Harvard University In 2013-14, Harvard University piloted the use of MOOCs as tools for blended learning in select undergraduate and graduate residential and online courses. One of these courses, The Ancient Greek Hero, combined for–credit (Harvard College and Harvard Extension School) and open online (HarvardX) groups into a single online unit, marking the first time the same instance of a MOOC was used simultaneously by both tuition–paying, credit–seeking students and non–paying, non–credit students enrolled exclusively online. In this article, we analyze and compare the online behavior of students and participants in the three groups that simultaneously participated in The Ancient Greek Hero via the edX platform. We find that, in similar fashion to a traditional learning setting, students enrolled in all three versions of the course engaged the online content in a transactional way, spending their time and effort on activities and exercises in ways that would optimize their desired outcomes. While user behavior was diverse, HarvardX participant engagement tended to be either very deep or virtually nonexistent, while College and Extension School students displayed relatively homogenous patterns of participation, viewing most of the content but interacting mostly with that which affected their overall course grades. Ultimately, we conclude that educators who intend to utilize MOOC content in an effort to apply blended learning techniques to their classrooms should carefully consider how best to incorporate each online element into their overall pedagogical strategy, including how to incentivize interaction with those elements. Further, for MOOCs to have maximum impact, they must address multiple learner motivations and provide participants with multiple modes of interaction with the content and with their peers. Keywords: MOOC, Blended Learning, Humanities, Academic Technology Open, Online, and Blended Overview During the 2013–14 academic year, Harvard University strategically piloted the use of massive open online courses (MOOCs) as a primary tool for the implementation of blended learning techniques in select undergraduate and graduate courses offered by Harvard College, Harvard School of Public Health, and Harvard Extension School. One of these pilots, CB22.1x The Ancient Greek Hero (Fall 2013), combined the for–credit and open online groups into one instance of the online platform, rather than creating separate instances for each participating group to enroll in and utilize, as was done in the other courses. To our knowledge, this pilot was the first time that a single instance of an existing or in–production MOOC was used simultaneously by both tuition–paying, credit–seeking students and non–paying, non–credit students enrolled exclusively online. In this paper, we describe the online behavior of students and participants in the three groups that simultaneously participated in CB22.1x. Although HarvardX is only in its third year, researchers have already documented common patterns among participants. They tend to have relatively high levels of education, with over half of the participants in each course holding a Bachelor’s degree or higher (Ho, et al., 2014). Many sign up for courses, but far fewer show broad engagement with the content: most engage with less than half the course content, and a majority of participants cease activity within the first week of a learning experience (ibid). Great diversity exists among the archetypical users (Ho, et al., 2014; DeBoer, et al., 2014; Reich, et al., 2014). For instance, users may achieve similar grades, while displaying drastically different amounts of time spent on the content. Some users may view all videos and engage with all of the readings and assessments, while others may do none of these things but have the same overall level of satisfaction with the learning experience (Ho, et al., 2014; Reich, et. al., 2013). This blended learning pilot coincided with the second offering of the Ancient Greek Hero MOOC, so the initial iteration (CB22x, Spring 2013) serves as a baseline for comparing demographics, activity, and engagement among online learners. Similar to HarvardX users in general, CB22x Spring 2013 participants tended to be between 20 and 40 years of age with high levels of formal education. They had diverse patterns of online behavior, although over 90% engaged with less than half the content, and only 3.2% of registrants obtained a certificate of completion (though the certification rate doubles to 5.5% if registrants – those who registered but never participated – are removed from the equation) (Reich, et al., 2014, esp. p. 10). Online and blended learning are neither new nor unfamiliar in higher education (Rooney, 2003; Dziuban, Hartman & Moskal, 2004; Ross & Gage, 2006). However, MOOCs are still relatively recent phenomena, and educational institutions remain in the early stages of experimenting with their integration into traditional courses (e.g. Bruff, et al., 2013; Bebell, Fernandes & Petersen, 2014; Fowler, et al., 2014; Picciano, 2014). Thus far, research suggests that learning outcomes are maximized when MOOCs are utilized as supplements to, rather than wholesale replacements for, traditional learning experiences. For example, San Jose State University, one of the earliest institutions of higher education to begin experimenting with MOOCs in residential learning, saw negative results from their attempt to replace traditional remedial and introductory courses with online learning experiences, but has experienced more positive results from blended learning experiments in which se- 2 Jeffrey P. Emanuel and Anne Lamb | 2015 lect MOOC content supplements traditional classroom instruction (Kolowich 2013a; 2013b; Lewin 2013; Firmin, et al., 2014; Bebell, Fernandes & Petersen, 2014). Bowen, et al. (2012) found that students enrolled in courses that blended interactive learning online (ILO) with traditional presentation and material performed at a level equal to that of students in traditional courses. In other words, when designing residential learning experiences, MOOCs may most effectively serve as a form of multimedia, such as an interactive textbook, rather than as a self–contained “course” in the traditional sense (Bruff, et al., 2013; Fowler, et al., 2014; Emanuel, forthcoming; cf. Means, et al., 2013; also Lack, 2013 and Zhao & Breslow, 2013, both with numerous further references). In a 2013–14 pilot of blended learning at Harvard, Bergeron (2014) performed a qualitative analysis of five courses that used MOOC content in such a way, while Zhao and Ho (2014) utilized a quasi– experimental approach to identify the causal effect of a flipped classroom approach on student learning in one of those courses, an offering on Chinese history. While Zhao and Ho (2014) found no substantial difference in outcomes among students in the blended course, both they and Bergeron (2014) suggest that the nature of this pilot as the first of multiple iterations may mean it represents a lower bound with respect to the effect of MOOCs as tools for blended learning on students’ aca40demic outcomes. This paper does not attempt to evaluate the overall efficacy of blended learning or of Harvard’s pilot effort. However, we do hope to contribute additional evidence to the existing literature on blended learning, MOOCs, and the nexus between the two, for the dual purpose of (1) informing practitioners and (2) providing data for further research in this space. We examine the use of the same MOOC, CB22.1x The Ancient Greek Hero, across three different student subpopulations engaged in three different implementation formats: Harvard College undergraduates taking the course Culture and Belief (CB) 22 in residence for General Education credit; students at Harvard Extension School (HES) taking the writing–intensive course CLAS E–116/w for undergraduate or graduate credit; and participants in the edX MOOC (known informally as HeroesX) engaging in the project wholly online, at no cost and for no formal academic credit. Though all three subgroups were enrolled in a single instance of the online content, and interacted together via a single integrated online discussion board, Harvard College and HES students were presented with a blended learning format that incorporated texts, video, and assessments from the MOOC with in–person or web–based class meetings, while HeroesX participants engaged the online content in an asynchronous, self–paced learning experience. Throughout the remainder of this paper, we describe patterns of behavior consistent with a few key themes: 1. Despite theoretical aspirations of MOOCs offering both breadth and depth for engaging in the online learning space, students in all three versions of the course utilized the edX platform in a transactional way, spending their time and effort on activities and exercises in ways that would optimize their grades. 2. As cautioned by prior researchers (e.g. Ho, et. al., 2014), certification in the HarvardX space does not imply a specific set of behaviors. It is a binary outcome that represents a multidimensional concept. While some certified students spend signifi- 3 Open, Online, and Blended cant time engaged in course content, carefully completing quizzes, and engaging with the community discussion boards, others seem to engage only with quizzes, over the course of very few active hours or days. Given this, we should note that certification in a MOOC does not necessarily equate to learning, nor does it necessarily indicate a similar level of content and skill mastery as participants in a traditional course (and vice versa for MOOC participants who elect not to pursue certification). Likewise, we should be cautious when attempting to compare online certification and scores to for–credit course completion and grades. 3. As in many HarvardX learning experiences, while user behavior was diverse, HeroesX participants generally trended toward one end of the participation spectrum or the other, either registering and viewing less than half the content (“registrants” and “viewers”), or immersing themselves in the majority of the content and earning a certificate of completion (certificate earners). College and HES students displayed relatively homogenous patterns of participation, viewing most of the content but, on average, not obtaining the edX certificate of completion. These students produced patterns of behavior most similar to a minority of HeroesX participants which we refer to as explorers (see below for further explanation of these terms). Background In the fall semester of 2013, a subset of personnel from HarvardX and the Derek Bok Center for Teaching and Learning at Harvard University’s Faculty of Arts and Sciences partnered with the professors and teaching teams from five Harvard courses to experiment with the use of existing or in– development HarvardX MOOCs as tools for experimentation with blended and activity–based learning techniques in undergraduate and graduate for–credit courses. The approach supplemented in–class meetings with online multimedia content, including videos, discussion forums, textbooks, and assessments, with the intent of enriching the learning experience by increasing the modes by which students interact with the material, and by repurposing the class time traditionally used for lecture to allow for different types of engagement (Bergeron, 2014). The MOOC CB 22.1x The Ancient Greek Hero was utilized in an effort to implement a blended learning approach in the Fall 2013 iteration of the long–running residential course of the same name. In this approach, students would spend considerable time outside of class engaging with multimedia content (readings, videos, self–assessments, and online forum discussions), thus allowing in–class time to be spent engaging more deeply with the overall themes of the course, primarily via a form of Socratic dialogue in which the professor was a participant rather than the sole source of information and opinion. The Ancient Greek Hero provides a rich case study in the simultaneous use of a common platform and content repository by three distinct groups of users. For undergraduate students at Harvard College (from here, “College” students), who enrolled in the traditional general education course for college credit, and for Harvard Extension School (HES) students, including local and online stu- 4 Jeffrey P. Emanuel and Anne Lamb | 2015 dents taking the course for undergraduate or graduate credit, the MOOC platform provided the required course textbooks in digital, annotatable format, video dialogues building on the readings, online forum discussions, and self–assessment exercises. For College students, the online material supplemented twice–weekly class meetings. HES students were provided videos of the College class meetings and attended a weekly section meeting either in–person or online, depending on each student’s physical location (section meetings were omitted from the College course in 2013 only, and were reinstated in Fall 2014 (Bergeron, 2014)). Both the College and HES versions required written assignments in the form of weekly two–to–three sentence “micro–essays” and 500–word “position papers” for which select self–assessment questions and discussion topics on the HeroesX platform served as prompts. The third version of the course consisted of edX registrants (from here, “HeroesX” users) taking the MOOC at no cost and for no academic credit. These participants had access to the same online content as the College and HES students, and interacted with members of the other groups via the same online discussion board, but they did not have class meetings, sections, or other exercises to supplement their experience. Online participants could achieve “Honor Code Certificates” in recognition of their level of participation by achieving a cumulative score of 50 percent or greater on the chapter assessments. Structurally, the MOOC’s content was divided into 24 chapters, or top–level units of content (referred to in CB22.1x as “Hours” for their reflection of the classroom time in an academic semester and, aptly, of the units in Homer’s epics). In addition, two opening segments were provided, titled “Welcome” and “Introduction” (the latter of which included practice exercises, though they did not count toward final grade or certification), as well as a conclusion, for a total of 27 chapters. The entirety of this content was made available to all three groups of learners (College, HES, and HeroesX) on the first day of the course, allowing for asynchronous access and engagement. College and HES students followed a structured syllabus, with specific readings, assignments, and due dates each week. Although this syllabus was presented as an “idealized schedule” to MOOC participants who wished to follow along with their for–credit counterparts, HeroesX users were both allowed and explicitly encouraged to engage with the content at whatever pace, and in whatever depth, was most comfortable to them. Demographic Characteristics of Participants Although The Ancient Greek Hero has been offered as a residential course at Harvard College for 39 years and counting, the MOOC was offered for the first time in Spring 2013, with 43,563 students enrolling (Reich, et al., 2014). Enrollment was lower in the Fall 2013 second iteration, with 20,598 HeroesX participants, and a for–credit population of 237 students from the College and 37 from HES. 5 Open, Online, and Blended I. Age In both iterations of HeroesX, the median age was 29, consistent with the HarvardX average median age of around 28 years. HeroesX users who attained certificates of completion were, on average, slightly older than the average age of all HeroesX participants. While HES students were similar in age, on average, to HeroesX participants, students enrolled in the College version unsurprisingly tended to be younger than their HES and HeroesX counterparts (see Figure 1). 40 Age in Years 35 30 25 20 15 10 5 0 College HES All Certi.ied HeroesX Mean 20 33 33 37 Median 20 29 29 33 Figure 1. Average and median age among College, HES, and HeroesX students in different versions of The Ancient Greek Hero. 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Male Female College HES All Certi.ied HeroesX Figure 2. Gender distribution of students in College, HES, and HeroesX versions of The Ancient Greek Hero, Fall 2013 (College n = 237; HES n = 37; edX n = 20,598). 6 Jeffrey P. Emanuel and Anne Lamb | 2015 II. Gender Gender balance varies widely between edX courses, although most courses offered by HarvardX and MITx to date have tended to have higher proportions of male students. In both the Spring 2013 and Fall 2013 iterations of HeroesX, gender distribution was nearly equal. Slightly more male HeroesX students earned certificates in the Fall iteration, although the opposite was the case in the prior offering (Reich, et al., 2014). In the College version of the course, male students outnumbered females by a nearly 2 to 1 margin, while females outnumbered males by the same proportion in the HES version (see Figure 2). 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Master's or Higher Bachelor's High School Less than High School College HES All Certi.ied HeroesX Figure 3. Highest levels of educational attainment among students in College, HES, and HeroesX versions of The Ancient Greek Hero (College n = 237; HES n = 37; edX n = 20,598). III. Education A pre–course survey was administered to users in all three groups in an effort to shed light on participant information and motivation. In response to this survey, HeroesX participants self–reported as having higher overall levels of formal education than both College and HES students, a trend also seen in previous MOOCs (Ho, et al., 2014; see Figure 3). Almost two–thirds of HeroesX enrollees had previously earned a Bachelor’s degree or higher. HES students had, on average, higher levels of formal education than the College students (over 90% of whom, unsurprisingly, had previously earned a high school diploma at most), but less than participants enrolled in HeroesX. Categorization of Participation Great heterogeneity has been observed in the behavioral patterns of MOOC participants (Kizilcec, Piech & Schneider, 2013; Anderson, et al., 2014; DeBoer, et al., 2014; Ho, et al., 2014; cf. Yuan & 7 Open, Online, and Blended Powell, 2013; Liyanagunawardena, Parslow & Williams, 2014). Because students have a variety of personal motivations for enrolling and participating in MOOCs, their activity on the edX platform (watching videos, reading texts, completing assessments, participating in forum discussions, etc.) can vary widely. Despite the vast number of individual patterns of activity, Ho, et al., (2014) employed a simple categorization of participation patterns among HarvardX users based on the extent to which users engaged in course content: • • • • Registrants, who enroll in the course, but do not engage with any of the content; Viewers, who enroll and engage with less than half of the content; Explorers, who enroll and engage with more than half the content; and Certificate Earners, who enroll and who meet the requirements to obtain certification. For HeroesX participants to qualify for a certificate of completion (officially referred to as an “Honor Code Certificate”), they were required to achieve an average of 50% or above on the assessments included in the courseware. Scoring in the MOOC was solely dependent on these assessments, which consisted of four multiple–choice content questions (worth one point each) and four multiple–choice annotation questions based on close readings of primary texts that accompanied each of the 24 “Hours.” In Figure 4, we apply this categorization to the Fall 2013 version of The Ancient Greek Hero. The grades of HeroesX, College, and HES participants are plotted against the number of total chapters they viewed out of 27, using the above paradigm to describe users based on the extent of their course participation. Participants who signed up for HeroesX but viewed no chapters are the registrants. The lower left quadrant is made up of viewers, who examined less than half the content. The lower right quadrant shows the explorers, who viewed more than half the chapters but did not obtain certification. The certificate earners are those in the right upper quadrant. Where do College and HES students fall within this categorization paradigm? The plus signs (+) and open circles (o) denote College and HES students respectively. Overwhelmingly, students in these two versions of the course viewed over half the content, making them primarily explorers and certificate earners. Very few from either version viewed less than half the chapters, and no students from these versions merely registered (compared to over one–quarter of online users who remained only 8 Jeffrey P. Emanuel and Anne Lamb | 2015 registrants, never engaging the MOOC at all). Interestingly, while explorers comprise the largest percentage of College and HES students, the smallest proportion of HeroesX participants – just 1% – fall into this classification, with registrants (34%), viewers (62%), and certificate earners (3%) far outnumbering them. This suggests that College and HES students kept with the requirements of their respective credit–bearing courses, as might be expected in a traditional educational environment, compared to the nontraditional ways in which MOOC participants engage online content. Comple3onist% CERTIFICATE( EARNERS( Op3mizer%% VIEWERS( ONLY( Only% Registered%% EXPLORERS(( ONLY( Listener%% Figure 4. Scatterplot of grades versus chapters viewed for College, HES, and online students in HeroesX (College n = 237; HES n = 37; HeroesX n = 20,598). To more precisely compare the activity patterns of the for–credit students with the HeroesX participants, we conducted a principal components analysis. To measure participation patterns, we include participants’ final course grades, the number of chapters and videos viewed, their number of forum posts, and the number of days active. The first principal component, which accounts for over half of the variance in the participation construct, is primarily characterized by higher grades, higher numbers of days active, and more chapters viewed (see loading plot displayed in Figure 5). The second component, which accounts for about one-fifth of the variance in the participation construct, seems to primarily measure the number of videos watched. Using these two components, in Table 1 we show mean principal component scores by participant group. The mean scores suggest that HeroesX participants who attain certification tend to have the highest number of chapters viewed, the highest grades, and highest numbers of days active, while HeroesX Viewers have the lowest. 9 Open, Online, and Blended Figure 5. Loading plot of principal components 1 and 2, showing that PCA 1 is primarily characterized by days active, chapters viewed, and grades, while PCA 2 is primarily characterized by the number of videos viewed. Comparing the difference in mean scores, HES and College students appear most similar to each other in their mean scores for both compoTable 1. Mean composites measuring "participation" nents 1 and 2 (see Table 1 mean component based on grades, chapters viewed, videos viewed, numscores). When comparing the mean scores of ber of forum posts, and the number of days active. HES and College students to HeroesX participants, HES students appear most similar 1 2 in their participation patterns to HeroesX (Viewers) –0.444 0.037 explorers (diffCertified = 1.132 v. diffExplorers HeroesX (Explorers) 2.976 0.927 = .944), while College students appear (Certified) 5.052 –0.739 somewhere in between HeroesX explorers HES 3.920 –0.381 College 4.338 –0.393 and certificate earners (diffCertified = 0.714 v. diffExplorers = 1.362). Throughout the remainNote: The first principal component explains over half of the varider of this article, we explore patterns of parance in the construct. The second principal component explains nearly one–fifth of the variance in the construct. The remaining three ticipation along a number of dimensions, principal components are not shown, as combined they only account and further compare activity among HES, for approximately 28% of the variance. College, and HeroesX participants. Patterns of Participation Although students in all three versions of The Ancient Greek Hero used the same edX instance, their course requirements, motivations, and performance incentives differed in critical ways. As a result, differences in their use of the MOOC platform were expected. College and HES students took The Ancient Greek Hero for undergraduate or graduate credit; for the former, the course fulfilled a general 10 Jeffrey P. Emanuel and Anne Lamb | 2015 education requirement, while for HES students, it fulfilled a writing–intensive requirement. Only 7 out of 237 College students were Humanities concentrators, and only one student in the entire class self–identified as a Classics major. Because earning college credit is a key motivator, we might expect these students to use the MOOC platform in a transactional way, focusing on the content and interactions that had the greatest impact on their final grades. Conversely, HeroesX participants were not eligible for college credit in exchange for successful completion of the MOOC. These enrollees registered and participated for a variety of reasons: Around one–fifth of registrants expressed the desire to “engage in lifelong learning,” according to the pre– course survey, and approximately the same number wanted “to learn from the best professors and universities.” Around 13% of HeroesX respondents said at the outset of the MOOC that they planned on “completing enough course activities to earn a certificate,” while nearly 16% of HeroesX users intended to complete most of all of the readings and 15% intended to complete most or all of the assignments. Although their credit statuses differed, students in all three versions of CB22.1x likely overlapped to some degree in their internal motivations for taking the course. HES and HeroesX users were more likely to take the course in addition to full–time jobs or family responsibilities, while College students chose the course from among a variety of possible course electives. Thus, all participants were interested in, and motivated by, the content of the course. Whether this internal motivation dominated external motivations in driving individuals’ actions is a matter of interpretation of the empirical evidence. I. Structure and Participation All of the course’s content was housed on the MOOC platform, including videos, self–assessments, the shared discussion board, and the two textbooks used in the course, The Ancient Greek Hero in 24 Hours (Nagy 2013) and the Sourcebook of Ancient Greek Texts Translated into English. As a result, participants from all three groups had to create (free) user accounts in order to access the required material. College and HES students’ grades were dependent on their interactions with the platform content – specifically, the online assessment questions – but not in the same way that HeroesX participants’ scores were. Rather than completing the eight multiple choice questions provided with each Hour of online content, College and HES students were required to write brief essays in response to selected questions from those exercises. Each week, these students submitted one 500–word “position paper,” which was to be written in response to a pre–selected question (or set of questions) from the eight annotation exercise questions included in the two Hours that made up that week’s online content. The position papers comprised 39% of College students’ final grades, and 50% of HES students’ grades. Additionally, students in both for–credit groups were responsible for 22 two–to–three– sentence “micro–essays” over the course of the semester (two per week, beginning in the second week of class), which were to be posted on the online discussion board on the edX platform. The micro–essays made up 22% of College students’ final grades, and the other 50% of HES students’ grades. The micro–essays were written in response to a stand–alone discussion question provided at the end of each hour. For logistical reasons, students were also required to email their papers and 11 Open, Online, and Blended micro–essays to the appropriate members of each course’s teaching staff. As a result of the latter policy, students were not held strictly accountable for first posting their micro–essays to the MOOC discussion board, and because of this, many decided to forego the extra step of posting to the forum altogether. Conversely, HeroesX students did not have a writing requirement; to earn certificates of completion, they only had to demonstrate their mastery of the concepts and content presented in the MOOC by achieving a score of 50% or greater in the online self–assessments. Online students’ grades depended exclusively on their performance on the two quizzes that accompanied each hour. Users could attempt each assessment question only once, and could receive half or full credit (1 or 2 points) for the annotation exercises. Given their differences in motivations, incentives, and course requirements, we might expect HeroesX users to show different patterns of usage than students in the for–credit classes. For example, HeroesX students might engage the multimedia materials to a greater degree, particularly with the videos, which were meant to serve not as a centerpiece of the course’s content, but as a supplement to, and extension of, the focal point of the learning experience for all three groups: the texts. Additionally, in the absence of a traditional class meeting, HeroesX participants might engage with the online forums to a greater extent than College and HES students. In Table 2, descriptive statistics for participation and performance are displayed for students in each version of The Ancient Greek Hero. As expected, College and HES students’ patterns of participation with the edX platform often differed from those of online participants. For instance, although the number of course chapters viewed by students in the College and HES versions was similar to those in the online version (median of 25 chapters viewed for College/HES students compared to a median of 23 chapters viewed for explorers and 27 chapters viewed for certified HeroesX participants), students in the former versions often had higher numbers of active days compared to students in the online version (median of 51/50 days active for College/HES students versus a median of just 17 days active/28 days active for online participants who explored/were certified). College and HES students had similar patterns of performance in comparison to online explorers and certificate earners according to other measures as well. For instance, College students and online explorers had very similar rates of correct assessment questions, with median proportions of 16% and 14% total questions correct. Moreover, when considering the proportion of problems they correctly answered among those they attempted, College, HES, online explorers and online certificate earners had remarkably similar median statistics (ranging from a median low of 50% correct among online explorers to a median high of 65% correct among online certificate earners, with College and HES students’ medians falling in between. 12 Jeffrey P. Emanuel and Anne Lamb | 2015 Table 2. Summary statistics of participation and performance among exclusive categories of user groups. HeroesX College Extension All Registered Viewed Explored Certified Number of Days Active (Median) Number of Chapters Viewed (Median, out of 27 total chapters) Videos viewed (Median) Videos viewed (Range) Forum Posts (Median) Forum Posts (Mean) 51 50 2 1 2 17 28 25 0 0 3 4.9 25 0 0–4 2 6.8 2 0 0 – 236 0 0.4 N/A 0 0 0 0.01 2 0 0 – 120 0 0.3 23 0 0 – 236 0 7.2 27 0 0 – 62 0 6.5 % Certified Grade (Mean) Grade (Median) 24% 31% 25% 41% 38% 41% 3% 37% 0% 0% 0% 0% 0% 1% 0% 0% 18% 10% 100% 75% 75% % of Total Problems Attempted (Median) % of Total Problems Correct (Median) Proportion of Problems Correct of those Attempted (Median) 28% 16% 46% 29% 0% 0% N/A N/A 0% 0% 20% 14% 97% 65% 55% 60% 0% N/A 0% 50% 65% Total N Percent of Total Participants 237 1.1% 37 0.2% 20,598 6,971 33.4% 12,765 61.2% 203 1% 659 3.2% 13 Open, Online, and Blended Online Explorers and Certified Users 30 0 0 10 20 Frequency 200 100 Frequency 40 50 300 All EdX Online Users 0 20 40 60 80 Grade (Out of 100 Points) 100 0 40 60 80 Grade (Out of 100 Points) 100 College 10 5 Frequency 3 2 0 0 1 Frequency 4 5 15 HES 20 0 20 40 60 Grade (Out of 100 Points) 80 0 20 40 60 Grade (Out of 100 Points) 80 Note: Distributions include only those with grades greater than 0%. Figure 6. Distribution of grades among all HeroesX, HES, and College students in The Ancient Greek Hero (All Online Users n = 5,244; HeroesX Explorers and Certified Users n = 786; HES n = 32; College n = 228). Because College and HES students’ institutional grades did not depend on the online assessments and they had no requirement to qualify for certification, the distributions of grades and problems attempted differed between these groups and HeroesX learners in key ways. In Figure 6, the grade distributions for all HeroesX participants, for HeroesX explorers and certificate earners, and for HES and College students are shown. In both HeroesX distributions, a noticeable discontinuity exists at the certification cutoff of 50% (indicated by the vertical line). This discontinuity could result from “optimizers,” or those users who are most incentivized by the certificate as their end goal, who viewed just over half the content and completed just enough assessment questions to earn certification. The HeroesX grade distribution also includes a spike for users who had perfect or near perfect grades. These “completers” may be online users who were most interested in lifelong learning (as evidenced by viewing all chapters and completing all assessment questions) and who wanted the certificate to demonstrate such learning (as evidenced by the high grade). Neither the discontinuity nor the spike of perfect grades appears in the distributions for the College and HES users, who had no incentive to correctly answer the self–assessment questions or to secure an online score of above 50%. 14 Jeffrey P. Emanuel and Anne Lamb | 2015 Figure 7. Grades from edX conditional on the number of days active for HeroesX explorers & certificate earners, HeroesX registrants and viewers, College, and HES students (HeroesX registrants and viewers n = 4,458; HeroesX Explorers and Certified Users n = 786; HES n = 32; College n = 228). In general, as displayed in Figure 7, a positive relationship exists between the number of active days on the MOOC site and participants’ final online score (the grade resulting from the online self– assessments, not the institutional grade administered to students in the for–credit courses). This remains true until around the 75–day mark, at which point increasing the number of active days begins to show a negative return (cf. Nesterko & Seaton, 2014). At the same time, among HeroesX participants who earned certification, those with very few days active were nearly as likely to earn certification as those with many days active, as shown by the relatively flat trend line in Figure 7, and by the scatterplot in Figure 8. 15 0 20 40 60 80 100 Open, Online, and Blended 0 15 30 45 60 75 90 105 Number of Days Active 120 HeroesX Explorers and Certificate Earners College 135 150 165 HES Figure 8. Grades from edX by number of days active for College, HES, and HeroesX participants (HeroesX registrants and viewers n = 4,458; HeroesX Explorers and Certified Users n = 786; HES n = 32; College n = 228). II. Online Discussion Online discussion was a key component of The Ancient Greek Hero, particularly for HeroesX users. The HeroesX discussion board provided the primary mechanism for fostering discussion among participants from all versions of the course. In fact, promoting conversation within and between all three groups was a key factor in the decision to utilize a single instance of the MOOC for the two for–credit courses and for HeroesX. To help promote this dialogue, College and HES students were explicitly incentivized to initiate online discussions by posting each of their 22 “micro–essays,” which counted toward their final grades, to the discussion board. However, because of technical difficulties in searching the forums and identifying posting histories by user, these “micro–essays” were also emailed to the appropriate teaching fellow(s), thus allowing students to bypass the forums despite the official requirement that they post. HeroesX participants, as noted above, had no external incentive for posting to the discussion board, as it was not taken into consideration for scoring or certification purposes. However, we might expect to see those who were primarily motivated by the opportunity to engage in a community learning experience post to the discussion board with some frequency. 16 Jeffrey P. Emanuel and Anne Lamb | 2015 Figure 9. Boxplot of the number of forum posts by certified HeroesX, HES, and College students for all values less than 50 posts (College n = 237; HES n = 36; HeroesX Certified n = 645). Beyond what is shown, 14 HeroesX users posted greater than 50 posts, but these users were at the extreme end of the distribution. In Figure 9, the distributions of the number of discussion board posts are shown for HeroesX certificate earners, College, and HES students. The modal number of discussion posts was zero for HeroesX certificate earners and HES students, and three posts for College users. This, in combination with the fact that the HES and College syllabi required 22 posts each, demonstrates the extent to which the option to email “micro–essay” responses directly to the teaching staff trumped the requirement to also post these brief responses to the HeroesX discussion board. Overall, the College distribution is closer to a normal distribution than either the HES or online distributions. For the most part, except for a small percentage of outliers, students from all groups seemed to use the discussion board in a transactional way. Because it was required as part of their final grades, College and HES students posted the most on average, though the email submission option de– incentivized posting to a sufficient degree that, as noted above, both groups’ median and mean posting output was below the amount specifically required in their respective syllabi. Interestingly, HeroesX participants, who in the absence of a traditional academic community may have stood to benefit most from the discussions, posted the least on average. At the same time, although College students post the most on average, HeroesX viewers – participants who enrolled and engaged with less than half of the content – actually accounted for the majority of users who ever contributed posts to the discussion board, with just over 50% of those who posted falling into this category. Around 20% of those who posted were College students, and 20% certificate earners. 17 Open, Online, and Blended HES 15 5 10 Percent 40 0 20 Percent 60 20 Students who are Motivated by Certification .2 .4 .6 .8 Proportion of Problems Attempted out of Total Problems 1 0 0 0 .2 .4 .6 .8 Proportion of Problems Attempted out of Total Problems 1 40 Explorers 15 30 20 .2 .4 .6 .8 Proportion of Problems Attempted out of Total Problems 1 5 0 Percent 0 10 10 Percent College 0 80 Certificate Earners 1 40 .2 .4 .6 .8 Proportion of Problems Attempted out of Total Problems 0 20 Percent 60 0 .4 .6 .8 Proportion of Problems Attempted out of Total Problems 1 Figure 10. Distribution of the proportion of problems attempted out of total problems for HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n = 659; College n = 237; HES n = 37). III. Patterns of Assessment Behavior Differences in incentives may also account for differences in patterns of quiz–taking. In the first panel of Figure 10, the distributions of problems attempted are displayed for three subsets of HeroesX participants – those who expressed the intent to earn a certificate in response to the pre–course survey, explorers, and certificate earners. Those participants who indicated that earning a certificate was “important” on the survey did not, for the most part, demonstrate behavior in line with their stated 18 Jeffrey P. Emanuel and Anne Lamb | 2015 intentions, as nearly two–thirds did not attempt any quiz problems. On the other hand, almost 60% of explorers attempted more than one–tenth of the problems, while over 70% of HeroesX participants who eventually earned certification attempted nearly all of the problems. Table 3. Comparison of proportion of questions attempted, HES and College students also had an grouped by questions that served as essay prompts or did incentive to attempt specific self– not. assessment problems. Each week, a se"Essay Non–"Essay lected annotation exercise question (or, Total QuesPrompt" Prompt" in some cases, multiple questions) was tions Questions Questions identified as the prompt for students’ HeroesX 31% 25% 32% “position paper” essays, and students HES 52% 57% 52% were informed ahead of time which quesCollege 39% 67% 35% tion(s) that would be. Since students could take questions selectively, and since Sample includes 2,622 HeroesX participants, 32 HES students, attempting a question (whether the reand 229 College students. sponse was right or wrong) would display both the correct answer and a paragraph–length explanation from the professor why the answer was correct and the other options were incorrect, it was beneficial for students to at least attempt the self–assessment questions on which each week’s essay was based. In Table 3, we provide evidence to support this behavior among College students. While HeroesX participants attempted around 30% of questions (essay prompts did not apply to this group), College students attempted nearly twice as many questions that served as essay prompts as they did non–essay prompt questions. To further explore this behavior, we plotted the percentage of essay prompt versus non–essay– prompt questions attempted by two groups, College students and HeroesX participants, and included a fitted linear function (see Figure 11). The left panel reflects the actual behavior of HeroesX participants; we expect the curved functional form given the uneven distribution of essay versus nonessay prompts throughout the course. At most, one of eight questions was an essay prompt for each hour, although some of the early hours included no essay questions. Therefore, conditional on attempting a certain total proportion of questions, more of those questions were likely to be non-essay questions than essay questions. Many of the HeroesX participants also attempted no questions or attempted all questions. Thus, the overall pattern for HeroesX participants, who had no incentive to selectively answer questions, is that they ended up attempting more non-essay questions. Conversely, College students attempted substantially more essay–prompt questions than they did the remaining 70% of assessment questions, suggesting that they did extensively engage in selectively attempting the questions that were most incentivized toward their end-of-course grades. 19 Open, Online, and Blended Figure 11. Proportion of essay prompt questions attempted against non–essay prompt questions attempted, for both College and HeroesX students who attempted at least one quiz problem (HeroesX n = 2,622; College n = 229). HES plot is not included, but shows relatively homogenous scatter along the diagonal. As noted above, the number of assessment questions engaged by the three broader groups, and by the users within the HeroesX population, varied widely, from a median of 28% of total questions attempted by College students, to 0% attempted by HeroesX participants overall, to 97% attempted by HeroesX certificate earners (see Table 3). Interestingly, among those questions that were attempted, HeroesX participants’ patterns of success were similar to those of College and HES students (see Figure 12). Thus, in comparison to their College and HES counterparts, HeroesX certificate earners seem to have achieved certification as a result of attempting most or all of the questions, while HES and College students may have failed to obtain certification as a result of not attempting as many problems, rather than because they answered the questions they took incorrectly. These patterns reinforce two important points: that edX certification signifies the completion of the requirements to a stated standard rather than mastery of the content, and that failure to obtain certification may not indicate failure to master the content and skills of the course. 20 Jeffrey P. Emanuel and Anne Lamb 2015 HES .2 .4 .6 .8 Proportions of Problems Correct out of Total Attempted 1 20 0 0 10 0 20 Percent 40 30 60 40 Students who are Motivated by Certification Percent | Explorers 30 0 .2 .4 .6 .8 Proportions of Problems Correct out of Total Attempted 1 20 20 1 10 .2 .4 .6 .8 Proportions of Problems Correct out of Total Attempted 5 0 Percent 15 10 0 Percent College .2 .4 .6 .8 Proportions of Problems Correct out of Total Attempted 1 4 6 0 0 2 Percent 8 0 10 Certificate Earners .2 .4 .6 .8 Proportions of Problems Correct out of Total Attempted 1 Figure 12. Distribution of proportion of problems correct out of total problems attempted for HeroesX participants, HES, and College students in The Ancient Greek Hero (HeroesX motivated by certification n = 3,395; HeroesX explorers n = 203; HeroesX certificate earners n = 659; College n = 237; HES n = 37). Conclusion and Implications Similar to a traditional learning setting, students from all three versions of The Ancient Greek Hero spent the time and effort necessary to maximize their desired outcomes. In other words, both for– credit students and HeroesX participants engaged the online content in ways that were best aligned with the incentives they were provided (either internally or externally). Online users who achieved certification spent a higher percentage of time attempting problems relative to other users, as the online assessments were their sole mechanism for achieving that goal. College and HES students, on the other hand, chose to spend time and effort on activities which best complemented the offline components of their respective courses, allowing them to maximize their own overall performance (cf. 21 Open, Online, and Blended Bergeron 2014: 4, 11, 16–18). Other than engaging the textbooks and essay prompts made available on the HeroesX platform, this meant spending comparatively little time and effort online. We draw a few practical conclusions from this analysis. First, educators intending to utilize MOOC content in an effort to apply blended learning techniques to their classrooms should carefully consider how best to incorporate each online element into their overall pedagogical strategy, including how interaction with those elements is to be incentivized. In the case of Culture and Belief 22 (Harvard College) and CLAS E–116/w (Harvard Extension School), the professor and teaching fellows ensured that students interacted with some of the self–assessment questions by basing essays on them, and spurred more discussion than might otherwise have taken place by requiring use of the forums by for–credit students (though, as noted above, technical limitations curtailed this). Some other MOOCs use different tactics to stimulate interaction between participants, such as counting forum posts as part of enrollees’ grades. EdX examples of this include AmPoX.1 Poetry in America: The Poetry of Early New England, SW12x China, HDS1544.1x Early Christianity: The Letters of Paul, and USW30x Tangible Things: Discovering History Through Artworks, Artifacts, Scientific Specimens, and the Stuff Around You. Additionally, one of the courses participating in the Fall 2013 pilot, Science of the Physical Universe (SPU) 27 Science and Cooking required students to post their final project to the online discussion board. In many other cases, courses forgo this altogether, instead using online discussion boards more as a tool of instructor–student communication or for crowdsourcing questions and problems; examples of this include, among many others, MCB80x Fundamentals of Neuroscience (edX), Understanding Research: An Overview for Health Professionals (Coursera), and Linear Circuits (Coursera). In the case of HeroesX, community engagement and discussion was a core component of the overall project’s aims, and one of the chief reasons College and HES students were included in the same instance of the MOOC as HeroesX learners (Reich, et al., 2014: 19; Emanuel, forthcoming). Second, for MOOCs to have maximum impact, they must address multiple learner motivations and provide participants with multiple modes of interaction with the content and with their peers. As is beginning to be emphasized in the still–nascent literature on MOOCs, certification and completion are not the only outcomes that designers and educators should consider when designing online learning experiences or when measuring outcomes (inter alia, Koller, et al., 2013; Lizilcec, Piech & Schneider, 2013; Stewart, 2013; Yuan & Powell, 2013; Anderson, et al., 2014; DeBoer, et al., 2014; Ho, et al., 2014; Liyanagunawardena, Parslow & Williams, 2014; Emanuel, forthcoming). Researchers and educators must continue to experiment with more flexible ways to measure engagement and learning both in MOOCs and in blended, hybrid, and flipped credit–bearing courses – particularly those that utilize MOOCs and their content as primary resources. In order to do this, practitioners must first gain a better understanding why participants are enrolling and participating in MOOCs in the first place, and, insofar as it is possible, think about alignment of learner outcomes with their own motivations. Certain key design concepts may work better in making the platform useful to students with different motivations. For instance, some MOOCs, particularly in STEM fields, allow students to make multiple attempts in order to get the correct answer on quiz questions, emphasizing learning the material eventually rather than only the first time (cf. Han, Veeramachaneni & O’Reilly, 2013; Kortemeyer, 2014). HarvardX is also relaunching past courses with different formats – including The Ancient Greek Hero, which is being offered in Fall 2014 as a 22 Jeffrey P. Emanuel and Anne Lamb | 2015 series of five discrete modules – in an effort both to allow learners to better self–direct their experiences, and to gain a better understanding of user motivation and engagement. MOOCs have the potential to serve both students in blended learning courses and learners worldwide who are internally motivated to engage with and learn about the content on their own. Practitioners involved in the online learning space are only beginning to imagine the possibilities for MOOC design and experimentation to help meet learners’ needs. Over time, we can continue to redesign and reimagine in the pursuit of making more dynamic learning experiences and improving learning for students in all contexts. 23 Open, Online, and Blended APPENDIX How accurately did residential and distance learning students self–identify themselves in HarvardX student surveys? Table A. Cross–tabulation of the number of students in each version according to self–identified survey data and to institutionally verified data. Institutionally–Verified Data Online HES College Total Online 8,798 3 8 8,809 HES 49 29 0 78 College 64 0 167 231 Self– Identified Missing* 11,687 5 62 11,756 Total 20,598 37 237 20,874 * Missing counts represent users enrolled in the course who did not complete the online pre–course survey. A voluntary pre–course survey was made available at the front of the courseware, and students were asked to complete it before proceeding to the rest of the online content. Only 175 students in the College course took the pre–course HarvardX survey, with 167 of them correctly identifying themselves as College students and eight identifying themselves as edX learners (231 total users actually self–identified as College students, with 64 of those self–identifying incorrectly). Of the 37 total HES students, 32 took the pre–course survey; 29 correctly identified themselves as HES students, while the other 3 identified themselves as online students. An additional 49 users identified as being in the HES course; some of these may have originally registered for the course but left during the drop–add period. 24 Jeffrey P. Emanuel and Anne Lamb | 2015 References Anderson, A., Huttenlocher, D., Kleinberg, J. & Leskovec, J. (2014). Engaging with Massive Online Courses. Paper presented at the 23rd International World Wide Web Conference, Seoul, Korea. http://cs.stanford.edu/people/ashton/pubs/mooc–engagement–www2014.pdf. Bebell, D., Fernandes, K. & Petersen, R. (2014). Applying MOOCs in On–Campus Settings: Opportunities, Obstacles, and Results. Paper presented at the ELI Annual Meeting, New Orleans, LA. http://www.educause.edu/events/eli–virtual–annual–meeting–2014/2014/applying– moocs–campus–settings–opportunities–obstacles–and–results. Bergeron, J. M. (2014). Blended Learning in the College. Cambridge, MA: Harvard University, Derek Bok Center for Teaching and Learning. Bowen, W.G., Chingos, M. M., Lack, K. A. & Nygren, T. I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials. Ithaka S + R Report. http://www.sr.ithaka.org/sites/default/files/reports/sr–ithaka–interactive–learning–online–at– public–universities.pdf. Bruff, D. O., Fisher, D. H., McEwen, K. E. & Smith, B. E. (2013). Wrapping a MOOC: Student Perceptions of an Experiment in Blended Learning. MERLOT Journal of Online Learning and Teaching, 9(2), 187–199. DeBoer, J., Ho, A. D., Stump, G. S. & Breslow, L. (2014). Changing "Course": Reconceptualizing Educational Variables for Massive Open Online Courses. Educational Researcher, 43(2), 74– 84. Dzubian, C. D., Hartman, J. L. & Moskal, P. D. (2004). Blended Learning. ECAR Research Bulletin, 2004(7). Emanuel, J. P. (forthcoming). MOOC as Multimedia Textbook: Rethinking the Open Online ‘Course’ in Light of Learner Intent. Firmin, R., Schiorring, E., Whitmer, J., Willett, T., Collins, E. D. & Sujitparapitayae, S. (2014). Case Study: Using MOOCs for Conventional College Coursework. Distance Education, 35(2), 178–201. Fowler, R., Meinking, K. A., Morrell, K., Sandridge, N. & Walker, B. (in press.) Adapting Content from a Massive Open Online Course to a Liberal Arts Setting. Transformations. Han, F., Veeramachaneni, K. & O’Reilly, U–M. (2013). Analyzing Millions of Submissions to Help MOOC Instructors Understand Problem Solving. Paper presented at the NIPS Workshop on Data Driven Education, Lake Tahoe, NV. Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses (HarvardX Working Paper No. 1). 25 Open, Online, and Blended Ho, A. D. & Reich, J. (2014). The Tricky Task of Figuring Out What Makes a MOOC Successful. The Atlantic, 23 Jan. http://www.theatlantic.com/education/archive/2014/01/the–tricky– task–of–figuring–out–what–makes–a–mooc–successful/283274/. Koller, D., Ng, A., Do, C. & Chen, Z. (2013). Retention and Intention in Massive Open Online Courses: In Depth. Educause Review Online, 3 June. http://www.educause.edu/ero/article/retention–and–intention–massive–open–online– courses–depth–0. Kolowich, S. (2013a). San Jose State U. Puts MOOC Project with Udacity on Hold. The Chronicle of Higher Education, 19 July. http://chronicle.com/article/MOOCs–May–Not–Be–So– Disruptive/140965/. Kolowich, S. (2013b). The MOOC ‘Revolution’ May Not be as Disruptive as Some Had Imagined. The Chronicle of Higher Education, 8 Aug. http://chronicle.com/article/MOOCs–May–Not– Be–So–Disruptive/140965/. Kortmeyer, G. (2014). How to Reduce Unproductive and Undesirable Behavior in Online Courses. Paper presented at the AAPT Summer Meeting, Minneapolis, MN. Lack, K. A. (2013). Current Status of Research on Online Learning in Postsecondary Education. Ithaka S + R Report. http://www.sr.ithaka.org/research–publications/current–status–research– online–learning–postsecondary–education. Lewin, T. (2013). After setbacks, online courses are rethought. The New York Times, 10 Dec. http://www.nytimes.com/2013/12/11/us/after–setbacks–online–courses–are–rethought.html. Lizilcec, R. F., Piech, C. & Schneider, E. (2013, April). Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. Paper presented at the Third International Conference on Learning Analytics and Knowledge, Leuven, Belgium. http://dl.acm.org/citation.cfm?doid=2460296.2460330. Liyanagunawardena, T. R., Parslow, P. & Williams, S. (2014). Dropout: MOOC Participants’ Perspective. In U. Cress & C. D. Kloos (Eds.), Proceedings of the European MOOC Stakeholder Summit 2014 (pp. 95–100). http://emoocs2014.eu/sites/default/files/Proceedings–Moocs– Summit-2014.pdf Major, C. H. (2010). Do Virtual Professors Dream of Electric Students? University Faculty Experiences with Online Distance Education. Teachers College Record, 112(8), 2154–2208. Means, B., Toyama, Y., Murphy, R. F. & Bakia, M. (2013). The Effectiveness of Online and Blended Learning: A Meta–Analysis of the Empirical Literature. Teachers College Record, 115(3), 1–47. Nesterko, S. & Seaton, D. T. (2014). Understanding User Engagement Through Time on Site in HarvardX MOOCs. Unpublished manuscript. 26 Jeffrey P. Emanuel and Anne Lamb | 2015 Picciano, A. (2014). Blended Learning Meets MOOCs: Education’s Digital Future. Paper presented at the ELI Annual Meeting, New Orleans, LA. Reich, J., Emanuel, J. P., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I. & Ho, A. D. (2014). HeroesX: The Ancient Greek Hero: Spring 2013 Course Report (HarvardX Working Paper No. 3). Rooney, J. E. (2003). Knowledge Infusion: Blended Learning Opportunities to Enhance Educational Programming and Meetings. Association Management, 55(6), 26–32. Ross, B. & Gage, K. (2006). Global Perspectives on Blended Learning: Insight from WebCT and Our Customers in Higher Education. In C. J. Bonk & C. R. Graham (Eds.), Handbook of Blended learning: Global Perspectives, Local Designs (pp. 155–168). San Francisco, CA: Pfeffer. Stewart, B. (2013). Massiveness + Openness = New Literacies of Participation? Journal of Online Learning and Teaching, 9(2), 228–238. Yuan, L. & Powell, S. (2013). MOOCs and Open Education: Implications for Higher Education – A White Paper, JISC CETIS (Centre for Educational Technology & Interoperability Standards). http://publications.cetis.ac.uk/2013/667. Zhao, Y. & Breslow, L. (2013). Literature Review on Hybrid/Blended Learning. HarvardX and MIT Teaching & Learning Laboratory. https://tll.mit.edu/sites/default/files/library/Blended_Learning_Lit_Reveiw.pdf Zhao, Y. & Ho, A. D. (2014). Evaluating the Flipped Classroom in An Undergraduate History Course (HarvardX Research Memo). http://harvardx.harvard.edu/files/harvardx/files/evaluating_the_flipped_classroom_– _zhao_and_ho.pdf. 27