Final Lecture: MOOC Planning – Part 9

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I gave my last lecture of the course yesterday (discounting the tutorial session that will go out next week), and we are now starting a two week exam period.

“Giving” a lecture means the video becomes available for streaming. For logistic reasons (high among them, my survival and continued sanity — assuming anyone who organizes and gives a MOOC, for no payment, is sane), I recorded all the lectures weeks ago, well before the course started.  The weekly tutorial sessions come the closest to being live. I record them one or two days before posting, so I can use them to respond to issues raised in the online course discussion forum.

The initial course enrollment of 63,649 has dropped to 11,848 individuals that the platform says are still active on the site. At around 20%, that’s pretty high by current MOOC standards, though I don’t know whether that is something to be pleased about, since  it’s not at all clear what the right definition of “success” is for a MOOC.

Some might argue that 20% completion indicates that the standards are too low. I don’t think that’s true for my course. Completion does, after all, simply mean that a student is still engaged. The degree to which they have mastered the material is unclear. So having 80% drop out could mean the standard is too high.

In my case, I did not set out to achieve any particular completion rate; rather I adopted a WYSIWOSG approach — “What You See Is What Our Students Get.” I offered a MOOC that is essentially the first half of a ten week course I’ve given at many universities over the years, including Stanford. That meant my students would experience a Stanford-level course. But they would not be subject to passing a Stanford-level exam.

In fact, I could not offer anything close to a Stanford-exam experience. There is a Final Exam, and it has some challenging questions, but it is not taken under controlled, supervised conditions. Moreover,  since it involves constructing proofs, it cannot be machine graded, and thus has to be graded by other students, using a crowd sourcing method (Calibrated Peer Review). That put a significant limitation on the kinds of exam questions I could ask. On top of that, the grading is done by as many different people as there are students, and I assume most of them are not expert mathematicians. As a result, it’s at most a “better-than-nothing” solution. Would any of us want to be treated by a doctor whose final exam had been peer graded (only) by fellow students, even if the exam and the grading had been carried out under strictly controlled conditions?

On the other hand, looking at and attempting to evaluate the work of fellow students is a powerful learning experience, so if you view MOOCs as vehicles for learning, rather than a route to a qualification, then peer evaluation has a lot to be said for it. Traditional universities offer both learning and qualifications. MOOCs currently provide the former. Whether they eventually offer the latter as well remains to be seen. There are certainly ways it can be done, and that may be one way that MOOCs will make money. (Udacity already does offer a credentialing option, for a fee.)

In designing my course, I tried to optimize for learning in small groups, perhaps five to fifteen at a time. The goal was to build learning communities, within which students could help one another. Since there is no possibility of regular, direct interaction with the instructor (me) and my one TA (Paul), students have to seek help from fellow students. There is no other way. But, on its own, group work is not enough. Learning how to think mathematically (the focus of my course) requires feedback from others, but it needs to include feedback from people already expert in mathematical thinking. This means that, in order to truly succeed, not only do students need to work in groups (at least part of the time), and subject their attempts to the scrutiny of others, some of those interactions have to be with experts.

One original idea I had turned out not to work, though whether through the idea itself being flawed or the naive way we implemented it is not clear to me. That was to ask students at the start of the course to register if they had sufficient knowledge and experience with the course material to act as “Community TAs”, and be so designated in the discussion forums. Though over 600 signed up to play that role, many soon found they did not have sufficient knowledge to perform the task. Fortunately,a relatively small number of sign-ups did have the necessary background, as well as the interpersonal skills to give advice in a supporting, non-threatening way, and they more or less  ensured that the forum discussions met the needs of many students (or so it seems).

Another idea was to assign students to study groups, and use an initial survey to try to identify those with some background knowledge and seed them into the groups. Unfortunately, Coursera does not (yet) have functionality to support the creation and running of groups, apart from the creation of forum threads. So instead, in my first lecture, I suggested to the students that they form their own study groups in whatever way they could.

The first place to do that was, of course, the discussion forums on the course website, which very soon listed several pages of groups. Some used the discussion forum itself to work together, while others migrated offsite to some other location, physical or virtual, with Skype seeming a common medium. Shortly after the course launched, several students discovered GetStudyRoom, a virtual meeting place dedicated to MOOCs, built by a small startup company.

In any event, students quickly found their own solutions. But with students forming groups in so many different ways on different media, there was no way to track how many remained active or how successful they have been.

The study groups listed on the course website show a wide variety of criteria used to bring the groups together. Nationality and location were popular, with groups such as Brazil Study Group, Grupo de Estudo Português, All Students From Asia, and Study Group for Students Located in Karachi, Pakistan. Then there were groups with a more specific focus, such as Musicians, Parents of Homeschooled Children, Older/Retired English Speakers Discussion for Assignment 1, and, two of my favorites, After 8pm (UK time) English speakers with a day job and the delightfully named Just Hanging on Study Group.

The forum has seen a lot of activity: 15,088 posts and 13,622 comments, spread across 2712 different threads, viewed 430,769 times. Though I have been monitoring the forums on an almost daily basis, to maintain an overall sense of how the course is going, it’s clearly not possible to view everything. For the most part I restricted my attention to the posts that garnered a number of up-votes. Students vote posts up and down, and once a post shows 5 or more up-votes, I take that as an indication that the issue may be worth looking at.

The thread with the highest number of up-votes (165) was titled Deadlines way too short. Clearly, the question of deadlines was a hot topic. How, if at all, to respond to such feedback is no easy matter. In a course with tens of thousands of students, even a post with hundreds of up-votes represents just a tiny fraction of the class. Moreover, threads typically include opinions on both sides of an issue.

For instance, in threads about the pace of the course, some students complained that they did not have enough time to complete assignments, and pleaded for more relaxed deadlines, whereas others said they thrived on the pace, which stimulated them to keep on top of the material. For many, an ivy-league MOOC offers the first opportunity to experience an elite university course, and I think some are surprised at the level and pace. (I fact, I did keep the pace down for the first three weeks, but I also do that when I give a transition course in a regular setting, since I know how difficult it is to make that transition from high school math to university level mathematics.)

A common suggestion/request was to simply post the course materials online and let students access them according to their own schedules, much like Khan Academy. This raises a lot of issues about the nature of learning and the role MOOCs can (might? should?) play. But this blog post has already gone on long enough, so I’ll take up that issue next time.

To be continued …


8 Responses to “Final Lecture: MOOC Planning – Part 9”

  1. 1 Dr. Carol Collins October 20, 2012 at 2:50 pm

    Dr. Devlin:

    Thank you for taking the time to provide this class to so many of us. There is something very special in knowing that you provided these lessons free of charge…. to so many … effectively.

    Even though I was not able to submit my assignments on time —most of the time, I value and appreciate your instruction. I believe that I am better for having taken this course —and more so because of the care in which you designed this course; and, the manner in which you executed the instruction.

    I am in my mid 60’s and I signed up for your class in spite of my busy schedule because I believe we learn new skills each moment of every day. And technology advances each month exponentially, it seems.

    Your class provides intangible benefits that I am not certain your team can evaluate inclusively. Intangibles are priceless and may very well be the most important measures of a full life lived. Thank you again.
    Dr. Carol Collins

  2. 2 Paul Reiners October 20, 2012 at 6:06 pm

    I thought about taking this class, but didn’t because (1) I was a math major in college and I thought this might be a little bit too much review for me, and (2) I was already taking 3 or so other online classes.

    Anyway, what I want to say is that I hope you teach an online version of your book “The Computer as Crucible” sometime. I bought the book awhile ago, but have not gotten very far in it yet (not the fault of the book—it’s my fault; I flit around too much). Having class deadlines will really push me to get through it.

    I really like deadlines and I’ve really enjoyed the fast pace of some Stanford and Princeton classes I’ve taken online.

  3. 3 Denise October 21, 2012 at 11:29 pm

    “…assuming anyone who organizes and gives a MOOC, for no payment, is sane…”
    I’ve been wondering the same about a person with a full “regular life” schedule who signs up to take a MOOC. I’ve only been able to keep up because I was already familiar with many of the ideas. Still, I’ve thoroughly enjoyed the class. Thank you for offering it!

  4. 4 alfonso October 22, 2012 at 8:58 pm

    I would like to thank you for your work and enthusiasm.
    I have enjoyed the course, It has challenged me since I studied part of the material more than 20 years ago.
    I found only two problems, time setting (PST-PDT-GMT) and few tolerance when posting out of a thread.

    Thanks and good luck.

    I am a chemical engineer in my very last 30’s….

  5. 5 Keith Devlin October 27, 2012 at 5:29 am

    Thanks to all for the positive feedback!

    • 6 mgozaydin October 28, 2012 at 10:52 am

      Dear Prof Devlin
      If I were you ” I would thank more the negative feedbacks in order to correct my discussable sides . ”
      But I appreciate your being very sincere

  6. 7 Joao October 27, 2012 at 6:56 pm

    I should warn you though, that there isn’t much studying getting done in getstudyroom, it’s all kind of a scam from their CEO to get their company up and running.

    • 8 mgozaydin October 28, 2012 at 10:29 am

      Prof. Devlin
      Thanks for your sharing of your experience of your online course .
      This is what I needed in many years . Your sincere approach is most appreciated.
      1.- I claim for years that online cost is nill if it is offerred to scale
      Then fee can be can be also very small.
      Think now every body of your 12,000 serious followers can pay only $ 10 that makes $ 120,000 and it is good,it makes the whole GOOD ONLINE sustainable . You do not need 64,000 not followers . It is just sickening and it makes online education unattractive , worthless.
      Even if you have only $ 50-80,000 students at the end
      $ 50 – 80,000 income is not bad .
      Important point is can it be sustainable at that rate every semester .
      By the way I am from Caltech and Stanford. I learned a lot math ar Caltech therefore my MS degrees in Engineering from Stanford was very easy.

Comments are currently closed.

I'm Dr. Keith Devlin, a mathematician at Stanford University. I gave my first free, open, online math course in fall 2012, and have been offering it twice a year since then. This blog chronicles my experiences as they happen.

Twitter Updates

New Book 2012

New book 2011

New e-book 2011

New book 2011

October 2012

%d bloggers like this: