Coursera | MOOCtalk

Posts Tagged 'Coursera'

MathThink MOOC v4 – Part 6

Published December 23, 2013 Free online courses 2 Comments
Tags: Byron Reeves, calibrated peer review, Coursera, distance learning, education design, higher education, mathematical thinking, MOOC, online education, Stanford MOOC, student engagment, student expectations, Test Flight, transition course, transition math, video game learning

In Part 6, I talk about the new Test Flight process.

In the past, when students enrolled for my MOOC, they essentially had three options. One was not to take it as a course at all, but just regard it as a resource to peruse over time or to pick and choose from. A second was to take the entire course, but do so on their own time-scale. Or they could take it as a course, and go through it at the designated pace.

As do many MOOC designers, I tried to make sure my course could be used in all three ways. Though the vast majority of MOOC students fall into the first category, the other two are the ones that require by far the greatest effort by the course designer. They are the learners who have significant ambitions and will put in a lot of effort over several weeks.

The students in the last category will surely gain the most. In particular, they move through the course in lockstep with a cohort of several thousand other students who can all learn from and support one another, as they face each course deadline at the same time. Those students form the core community that is the heart of the course.

When the new class enrolls at the start of February, the ones intending to take an entire course as scheduled will have a new choice. They can take what I am calling the Basic Course, which lasts eight weeks, or the Extended Course, which lasts ten. As I described in my last post, those extra two weeks are devoted to a process I am calling Test Flight.

In the previous two versions of the course, the final weeks nine and ten had been devoted to a Final Exam, one week for completion of the (open book) exam itself, the following week to peer evaluation. In peer evaluation, which started as soon as the class had completed and submitted their exam solutions, each student went through a number of activities:

1. Using a rubric I supplied, each student evaluated three completed examination scripts assembled by me, and then compared their results to mine. (Those three samples were selected by me to highlight particular features of evaluation that typically arise for those problems.)

2. Having thus had some initial practice at evaluation, each student then evaluated three examination scripts submitted by fellow students. (The Coursera platform randomly and anonymously distributed the completed papers.)

3. Each student then evaluated their own completed examination.

This was the system Coursera recommended, and for which they developed their peer evaluation module. (Actually, they suggested that each student evaluated five peer submissions, but at least for my course, that would have put a huge time requirement on the students, so I settled for three.)

Their original goal, and mine, was to provide a means for assigning course grades in a discipline where machine evaluation is not possible. The theory was that, if each student is evaluated by sufficiently many fellow students, each of whom had undergone an initial training period, then the final grade – computed from all the peer grades plus the self-grade – would be fairly reliable, and indeed there is research that supports this assumption. (Certainly, students who evaluate their own work immediately after evaluating that of other students tend to be very objective.)

As far as I could tell, the system worked as intended. If the goal of a MOOC is to take a regular university course and make it widely available on the Internet, then my first three sessions of the course were acceptably successful. But MOOCifying my regular Mathematical Thinking (transition) class was always just my starting point.

Since I was aware from the outset that the MOOC version of my regular classroom course was just a two-dimensional shadow of the real thing, where I interact with my class on a regular basis and give them specific feedback on their work, my intention always was to iteratively develop the MOOC into something that takes maximum advantage of the medium to provide something new of value – whatever that turns out to be.

I expected that, as MOOCs evolve, they would over time come to be structured differently and be used in ways that could be very different from our original design goals. That, after all, is what almost always happens with any new product or technology.

One thing I observed was that, while students often began feeling very nervous about the requirement that they evaluate the work of fellow students, and (justifiably) had significant doubts about being able to do a good job, the majority found the process of evaluating mathematical arguments both enjoyable and a hugely beneficial learning process.

Actually, I need to say a bit more about that “majority” claim. My only regular means of judging the reactions of the class to the various elements of the course was to read the postings on the course discussion forums. I spent at least an hour every day going through those forums, occasionally posting a response of my own, but mostly just reading.

Since the number of regular forum posters is in the hundreds, but the effective (full-term) class was in excess of 5,000 in each of the sessions, forum posters are, by virtue of being forum posters, not representative. Nevertheless, I had to proceed on the assumption that any issue or opinion that was shared (or voted up) by more than one or two forum posters was likely to reflect the views of a significant percentage of the entire (full-term) class.

Since I made gradual changes to the course based on that feedback, this means that over time, my course has been developing in a way that suits the more active forum posters. Arguably that is reasonable, since their level of activity suggests they are the ones most committed, and hence the ones whose needs and preferences the course should try to meet. Still, there are many uncertainties here.

To return to my point about the learning and comprehension benefits evaluators gained from analyzing work of their peers, that did not come as a surprise. I had found that myself when, as a graduate student TA, I first had to evaluate students’ work. I had observed it in my students when I had used it in some of my regular classes. And I had read and heard a number of reports from other instructors who noted the same thing.

It was when I factored the learning benefits of evaluating mathematical arguments in with my ongoing frustration with the degree to which “grade hunting” kept getting in the way of learning, that I finally decided to turn the whole exam part on its head.

While some universities and some instructors may set out to provide credentialing MOOCs, my goal was always to focus on the learning, drawing more on my knowledge of video games and video-game learning (see my blog profkeithdevlin.org) than on my familiarity with university education (see my Stanford homepage).

Most of what I know about giving a university-level course involves significant student-faculty interaction and interpersonal engagement, whereas a well-designed video game maintains the player’s attention and involvement using very different mechanisms. With a MOOC of necessity being absent any significant instructor-student interaction, I felt from the outset that the worlds of television and gaming would provide the key weapons I needed to create and maintain student attention in a MOOC.

[A lot of my understanding of how TV captures the viewer’s attention I learned from my close Stanford colleague, Prof Byron Reeves, who did a lot of the groundbreaking research in that area. He subsequently took his findings on television into the video game business, co-authoring the book Total Engagement: Using Games and Virtual Worlds to Change the Way People Work and Businesses Compete.]

So from the outset of my foray into the world of online education, I was looking to move away from traditional higher-education pedagogic models and structure, and towards what we know about (television and) video games, hopefully ending up with something of value in between.

The idea of awarding a Statement of Accomplishment based on accumulated grade points had to go sooner or later, and along with it the Final Exam. Hence, with Session Four, both will be gone. From now on, it is all about the experience – about trying (and failing!).

The intention for the upcoming session is that a student who completes the Basic Course will have learned enough to be able to make useful, and confident use of mathematical thinking in their work and in their daily lives. Completion of the Test Flight process in the Extended Course will (start to) prepare them for further study in mathematics or a mathematically-dependent discipline – or at least provide enough of a taste of university-level mathematics to help them decide if they want to pursue it further.

At heart, Test Flight is the original Final Exam process, but with a very different purpose, and accordingly structured differently.

As a course culmination activity, building on but separate from the earlier part of the course – and definitely not designed to evaluate what has been learned in the course – Test Flight has its own goal: to provide those taking part with a brief hands-on experience of “life as a mathematician.”

The students are asked to construct mathematical arguments to prove results, and then to evaluate other proofs of the same results. The format is just like the weekly Problem Sets that have met throughout the course, and performance level has no more or less significance.

The evaluation rubric, originally employed to try to guarantee accurate peer grading of the exam, has been modified to guide the evaluator in understanding what factors go into making a good mathematical argument. (I made that change in the previous session.)

After the students have used the rubric to evaluate the three Problem Set solutions supplied by me, they view a video in which I evaluate the same submissions. Not because mine provides the “correct” evaluations. There is usually no single solution to a question and no such thing as the “right” one. Rather, I am providing examples, so they can compare their evaluations with mine.

After that, they then proceed to evaluate three randomly-assigned, anonymously-presented submissions from other students, and finally they evaluate their own submission.

Procedurally, it is essentially the same as the previous Final Exam. But the emphasis has been totally switched from a focus on the person being evaluated (who wants to be evaluated fairly, of course) to the individual doing the evaluation (where striving for a reliable evaluation is a tool to aid learning on the part of the evaluator).

Though I ran a complete trial of the process last time, the course structure was largely unchanged. In particular, there was still a Final Exam for which performance affected the grade, and hence the awarding of a certificate. As a consequence, although I observed enough to give me confidence the Test Flight process could be made to work, there was a square-peg-in-a-round-hole aspect in what I did then that caused some issues.

I am hoping (and expecting) things will go smoother next time. For sure, further adjustments will be required. But overall, I am happy with the way things are developing. I feel the course is moving in the general direction I wanted to go when I set out. I believe I (and the successive generations of students) are slowly getting there. I just don’t know where “there” is exactly, what “there” looks like, and how far in the future we’ll arrive.

As the man said, “To boldly go …”

The MOOC Express – Less Hype, More Hope

Published December 8, 2013 Free online courses Closed
Tags: Audrey Watters, Bonnie Stewart, Coursera, David Cornier, distance learning, edX, George Siemens, Harvard University, higher education, Huffington Post, Jim Groom, MIT, MOOC, MOOC Research Initiative, online education, Stanford, Stephen Downes, Udacity, Web 2.0

A real-time chronicle of a seasoned professor just about to launch the fourth edition of his massively open online course.

Last week, I headed off to Arlington, Texas, to participate in a large, international conference on MOOC education, part of the Gates Foundation funded MOOC Research Initiative (MRI). While the founders of the big, massively-funded American MOOC (“MFAM”) platforms Coursera, edX, Udacity, and Novo Ed capture most of the media’s attention, this conference was led by the small band of far less well known Canadian online-education pioneers who actually developed the MOOC concept some years earlier, in particular George Siemens and Stephen Downes who organized and ran the first MOOC in 2008, and David Cornier who forever has to live with having coined the name “MOOC”.

(There were so many Canadians in Arlington, they brought their own weather with them, as you can see from the photograph. The conference ended with participants having to change flights and book additional nights in the conference hotel, as a severe ice storm hit the area. With a return flight that happened to lie within a brief lull in the storm, I managed to get out on time, though only after a slip-sliding taxi ride at a snail’s pace to get to DFW Airport. Others had it a lot worse.)

While teams of engineers in Silicon Valley and Cambridge (MA) are building out MOOC platforms that provide huge opportunities for massive scale-up, the two hundred or so researchers and educators who came together in Texas represent the vanguard of the educational revolution that is underway. If you wanted to know what MOOC learning might look at in a few years time, you would have better spent your time at the University of Texas in Arlington last week rather than in Silicon Valley. In his October announcement of the conference, organizer Siemens described it as “the greatest MOOC conference in the history of MOOCs,” a bit of satirical hype that is almost certainly true.

http://www.flickr.com/photos/gforsythe/8028605773/

You need both, of course, the technology to reach millions of people and the appropriate, quality pedagogy. For example, the movie industry required a series of major advances in motion picture technology, but fancy film cameras alone are not what gave us Hollywood. As the technology advanced, so too did the art and craft of motion picture writing, acting and directing. Similarly with MOOCs. The focus in Arlington was on the educational equivalent of the latter, human expertise factors.

To pursue the movie analogy a bit further (and with that great spoof movie poster as a visual aid, how could I resist?), the MOOC action that gets reported in the Chronicle of Higher Education and the breathless (but hopelessly off-base and over-hyped) prose of Thomas Friedman in the New York Times is the equivalent of taking an early movie camera into a theater and recording what happens on the stage.

If you want to see the future of MOOCs, you need to hang around with the instructors in the lesser known, small universities and community colleges who, for many years, have been experimenting with online learning. Most of the leaders of that loosely knit band could be found in a large, cavernous room in Arlington for two days last week – along with key figures from such new-MOOC, Ivy League players as Stanford, MIT, and Georgia Tech.

What was the take-home message? In my case, and I suspect everyone else’s, confirmation that we really don’t know where the MOOC train is taking us. The problem is not an absence of good ideas or useful leads; rather the opposite. Don’t expect a “Conference Proceedings” volume any day soon. The best summary you will find is probably the conference Twitter stream (hashtag #mri13).

The fact is, there isn’t even a clear definition of what a MOOC is. The common classifications of c-MOOC (for the original, Canadian, connectionist animal) and x-MOOC (for the later, scalability-focused, Stanford version) don’t help much, since many (most?) of the MOOCs being developed now have elements of both. Commenting on the lack of a generally accepted definition in his welcoming remarks, conference organizer George Siemens opined that it may be that no agreed definition will emerge, and that the term “MOOC” will be similar to “Web 2.0”, remaining an undefined term that “reflect[s] a mess of concepts that represent foundational changes that we don’t yet understand, can barely articulate, but that will substantially impact the system.”

Actually, I should modify my remark above that Stanford is a MOOC player. That’s not really accurate. Some Stanford scholars (including me) are developing and offering MOOCs, but the main focus at my university is research into learning in a digital age, including online learning. There is a lot of such research, but MOOCs are just one part of it. And those MOOC platform providers you keep reading about, Udacity, Coursera, and Novo Ed? They are all private companies that span out of Stanford. They are now (and were from their creation) Silicon Valley entities, outside the university. Stanford’s original, in-house MOOC platform, eventually called Class2Go, was recently folded into edX, to be run by MIT and Harvard as an open source platform.

As the MOOC research train goes forward, don’t expect the big, Ivy League universities to exclusively dominate the leading edge of the research. The reason so many participants in the Arlington conference came from far less wealthy, and often smaller institutions is that they have been developing and using online education for years to cater to their geographically-diverse, economically-diverse, and education-preparedness-diverse student bodies. The connections made or strengthened at the Arlington conference are likely to result in many collaborations between institutions both big and small, from the wealthy to the impoverished. The Digital World is like that. When it comes to expertise, today’s academic world is just as flat as the economic one – as Siemens clearly realized in drawing up his conference invitation list.

Expect to see MOOC pioneer Stephen Downes leading a lot of the action, as the head of a new, $19 million, 5-year R&D initiative for the Canadian National Research Council. Keep your eyes too on Siemens himself, who at the conference announced his move from Athabacsa University in Canada to be a professor in, and executive director of, the University of Texas at Arlington’s new Learning Innovations & Networked Knowledge (LINK) Research Lab. (So new, it doesn’t have a website yet!) Track conference keynoter Jim Groom, who whiled away the time holed up at snowed-in DWF Airport to draft this summary of the gathering. And don’t overlook the University of Prince Edward Island, where Dave Cormier and Bonnie Stewart brave the weather of the far North East. There are a whole host more. Check the list of recipients of MRI grants for other names to follow, and remember that this is just the tip of, dare I say it under the circumstances, the iceberg.

And never, ever forget to read the excellent (and highly knowledgable) ed tech commentator Audrey Watters, who was invited to attend the Arlington conference but had a scheduling conflict. New readers interested in MOOCs, start here.

Recent headlines may have given the impression that MOOCs were a short-lived bubble, and the experiment has largely failed. Nothing could be further from the truth. For one thing, even the much reported business pivot of Udacity was a familiar event in Silicon Valley, as I argued in a commentary in the Huffington Post.

As Siemens put it in his conference welcome, “the ‘failure’ of MOOCs is a failure of hype and an antiquated notion of learning, not a failure of open online learning.”

How are MOOCs organized?

Published February 24, 2013 Free online courses 12 Comments
Tags: Coursera, mathematical thinking, Stanford MOOC

A real-time chronicle of a seasoned professor who is about to give his second massively open online course.

With exactly one week to go before the second edition of my MOOC Introduction to Mathematical Thinking goes live, my TA and I have been working feverishly to get everything ready — a task far more complex and time consuming than preparing for a traditional (physical) course. (If you have been following this blog since I launched it last summer, when I started to plan my first edition of the course, you likely have some idea of the complexities involved.)

MOOCs continue to be in the news. Just last week, NBC-tv used my course as an illustration in a news story (4 min 21 secs) they ran about the American Council on Education’s recommendation that some Coursera MOOCs be considered eligible to receive college credit.

But what exactly is a MOOC and how are they organized? The easiest way to find out is to simply sign up for one or more and take a look. They are all free (at least, all the ones everyone is talking about are free), and there is no requirement to do any more than hang around online and see what is going on. If you do that, you’ll find that they all exhibit some differences from one another, as well as many similarities. Moreover, almost everyone giving a MOOC approaches it as an experiment, so they often change from one edition to the next.

Taking my own MOOC as an illustration, when the course website opens to registered students next weekend (Saturday March 2), they will initially find themselves in a website populated with several pages of information about the course structure, together with a bit of background information relevant to the course content, but none of the lectures, assignments, quizzes, problem sets, or tutorials will be available. Those are released at specified times throughout the ten weeks the course will run, starting with Lecture 1 on March 4.

For a sample of a lecture, see this short clip (7min 16 sec) from Lecture 1 on YouTube. (But note that Coursera videos are much higher resolution than YouTube, so the YouTube video is hard to follow — it’s purely an illustration of the overall format of the lectures.)

One of the main informational pages the students will see describes the various components of the course. Here, verbatim, are the contents of that page.

Basic elements of the course

Consult the Daily timetable (see link on left) on a regular basis to see what is due at any one time.

1. Lectures – videos presented by the instructor.
2. In-lecture quizzes – simple multiple-choice questions that stop the lecture, designed to assist you in pacing and monitoring your progress.
3. Assignment sheets (one for each lecture) – downloadable PDF files to work through in your own time at your own pace, ideally in collaboration with other students. Not graded.
4. Problem sets (one a week for weeks 1 through 8) – in-depth problems like those on the assignment, but with a deadline for submitting your answers (in a multiple choice format). Machine graded.
5. Tutorial sessions – the instructor provides (video) comments and answers to some of the previous week’s assignment problems.
6. Reading assignments – downloadable PDFs files providing important background information.
7. Final exam – a downloadable PDF file that you will have one week to complete before participating in a peer review process. Required to be eligible for a grade of completion with distinction.

Lectures

Lecture videos are released at 10:00AM US-PDT on Wednesdays. (Weeks 1 and 2 are slightly different, with lectures released on Monday and Wednesday.) Each lecture comprises one or two videos, with each video of length 25 to 35 minutes if played straight through. Completing the embedded progress quizzes will extend the total duration of a video-play by a few minutes, and you will likely want to stop the playback several times for reflection, and sometimes you will want to repeat a section, perhaps more than once. So you can expect to spend between one and two hours going through each lecture, occasionally perhaps more.

The lecture videos are not carefully crafted, heavily edited productions. If you want a polished presentation of the course material, you can read the course textbook. My goal with the lectures is to provide as best I can the experience of sitting alongside me as we work through material together. And, guess what, I often make mistakes, and sometimes mis-speak. I want to dispel any misconception that mathematicians are people who generate perfect logical arguments all the time. We’re not. We just keep going until we get it right.

In-lecture quizzes (Ungraded)

Each lecture is broken up by short multiple-choice “progress quizzes”. The vast majority of these in-lecture quizzes are essentially punctuation, providing a means for you to check that you are sufficiently engaged with the material.

Slightly modified versions of the quizzes will also be released as standalones at the same time as the lecture goes live, so if you do not have a good broadband connection and have to download the lecture videos to watch offline, you can still take the quizzes. In which case, you should do so as close in time to viewing the lecture as possible, to ensure gaining maximim benefit from the quizzes in monitoring your progress. The standalone quizzes are grouped according to lecture.

Completion of all the quizzes is a requirement (along with watching all the lectures) for official completion of the course, but we do not record your quiz scores, so quiz performance does not directly affect your final grade. If you complete the quizzes while watching the lecture (the strongly preferred method, as it helps you monitor your progress in mastering the material), you do not need to complete the standalone versions.

BTW, you may notice that it is possible to speed up video replay up to a factor of double speed. This can be a useful device when watching a video a second or third time. Going beyond 1.50 speed, however, can sometimes lead to problems with the display of the quizzes (besides making me sound like a chipmunk (though some may find that an enhancement).

Course assignments (Self graded)

An assignment will be released at the end of each lecture, as a downloadable PDF file. The assignment is intended to guide understanding of what has been learned. Worked solutions to problems from the assignments will be demonstrated (video) or distributed (PDFs) in a tutorial session released the Monday following the lecture (so in Weeks 2 through 9). The tutorial sessions will be released at 10:00AM US-PDT.

Working on these assignment problems forms the heart of the learning process in this course. You are strongly urged to form or join a study group, discuss the assignment problems with others in the group, and share your work with them. You should also arrange to assess one another’s answers. A structured form of peer review will be used for the final exam, when you will be graded by, and grade the work of, other students, randomly (and blindly) assigned, so it will help to familiarize yourself beforehand with the process of examining the work of others and providing (constructive) feedback.

Problem Sets (Machine graded)

Each Wednesday (in weeks 1 thtough 8), following the lecture, a for-credit Problem Set will be posted, with submission due by 9:00AM US-PDT the following Monday. The scores on these problem sets will count toward the course grade. Though the Problem Set has a multiple-choice quiz format, these questions are not the kind you can answer on the spot (unlike most of the in-lecture quizzes). You will need to spend some time working on them before entering your answers.

Though you are strongly encouraged to work with others on understanding the lecture material and attempting the regular assignments, the intention is that you work alone on the Problem Sets, which are designed to give you and us feedback on how you are progressing.

Tutorial sessions

The tutorial sessions are more than mere presentations of solutions to the previous week’s assignments and problem session. They are really lectures based on problems that the student has already attempted. You can expect to expand your knowledge of the course material beyond the lectures. Not all questions on the assignments sheets and problem set will be considered in the tutorial session.

Final exam (Peer graded)

Though the lectures end after week 8 (apart from a tutorial on the final assignment), the final two weeks are intended to be highly active ones for any students seeking a grade of distinction, with considerable activity online in the various forums and discussion groups. This is when you are supposed to help one another make sense of everything.

At the start of week 9, an open-book exam will be released, to be completed by the end of the week. Completed exams will have to be uploaded as either images (or scanned PDFs) though students sufficiently familiar with TeX have an option of keyboard entry on the site. The exam will be graded during week 10 by a calibrated peer review system. The exam will be based on material covered in the entire course.

As with the weekly Problem Sets, the intention is that you work alone in completing the final exam.

NOTE: The process of peer reviewing the work of others (throughout the course, not just in the final exam) is intended to be a significant part of the learning experience and participating in the formal peer review procedure for the final exam is a requirement for getting a grade of distinction. In principle, it is during week 10 that stronger students will make cognitive breakthroughs. (Many of today’s professors really started to understand mathematics when, as graduate student TAs, they first helped others learn it!)

Course completion and final grade

There are two final course grades: “completion” and “completion with distinction”. Completion requires viewing all the lectures and completing all the (in-lecture) quizzes and the weekly problem sets. Distinction depends on the scores in the problem sets and the result of the final exam.

Pacing

The pacing of the lecture releases is designed to help you maintain a steady pace. At high school, you probably learned that success in mathematics comes from working quickly (and alone) and getting to the right answer as efficiently as possible. This course is about learning to think a certain way – the focus is on the process not the product. You will need time to understand and assimilate new ideas. Particularly if you were a whiz at high-school math, you will need to slow down, and to learn to think and reflect (and ideally discuss with others) before jumping in and doing. A steady pace involving some period of time each day is far better than an all-nighter just before a Problem Set is due.

Keeping track

Consult the Daily timetable on the website on a regular basis to see what is due.

SO NOW YOU KNOW!

Here we go again

Published February 20, 2013 Free online courses 17 Comments
Tags: Coursera, mathematical thinking, MOOC, online education, Stanford MOOC

A real-time chronicle of a seasoned professor who is about to give his second massively open online course.

The second offering of my MOOC Introduction to Mathematical Thinking begins on March 4 on Coursera. (The site actually opens on March 2, so students can familiarize themselves with its structure and start to make contact with other students before the first lecture.) So far, 13,000 students have registered. Last time I got 65,000, but back then there was the novelty factor. I’m expecting about 35,000 this time round.

For a quick overview of my current thoughts on MOOCs, see this 13 minute TV interview I did at Tallinn University of Technology in Estonia last November. (As the home of Skype, global-tech-hub Tallinn is particularly interested in MOOCs, of course.)

It’s been almost four months since my first foray into the chaotic new world of MOOCs came to an end, and ten weeks since I posted my last entry on this blog. I have decided that giving a MOOC falls into the same category as running a marathon (I’ve done maybe two dozen), completing the Death Ride (three), and – I am told – having a baby (I played a decidedly minor role in two). At the time you wonder why you are putting yourself through such stress, and that feeling continues for a while after the event is over. But then the strain of it all fades and you are left with feelings of pleasure, accomplishment, and satisfaction. And with that comes the desire to do it all again – better in the case of running, cycling, and MOOCing.

Coursera, we have a problem

It’s important to remember that genuinely massive MOOCs are a mere eighteen months old, and each one is very much a startup operation — as are the various platform providers such as Udacity, edX, Coursera, Venture Labs. and Class2Go (all except edX coming out of Global Startup Central, i.e., Stanford). One of the features of any startup operation is that there will be plenty of missteps along the way. Given the complexity of designing and delivering a university course in real time to tens of thousands of students around the world, it’s amazing that to date there have been just two missteps. The first, when the instructor had to pull the plug on a MOOC on designing online courses (yes, a particularly poignant topic as it turned out) and then more recently when the instructor pulled out, leaving the course to be run by the support staff.

Notice that I did not refer to either as a “failure.” Anyone who views such outcomes as failures has clearly never tried to do anything new and challenging, where you have to make up some of the rules as you go on. We are less than two years into this whole MOOC thing, so it’s worth reminding ourselves what it took (VIDEO) the USA to put a man on the Moon and bring him back alive, and to go on and build the Space Shuttle. The pedagogic fundamental that we gain confidence from our successes but learn from our mistakes, is as true for MOOC platform builders and MOOC instructors as it is for MOOC students.

Fortunately, I survived my first test flight relatively unscathed. I may not be so lucky second time round. I’ve made some changes that are intended to make the course better, but won’t know if they do until the course is underway.

Perhaps the most obvious change is to stretch the course from seven weeks (five weeks of lectures followed by two weeks of final exam work) to ten (8 + 2). Many students in my first course told me that the “standard university pace” with which I covered the curriculum was simply too much for online students who were fitting the course around busy professional and family schedules. I doubt that change will have any negative consequences.

More uncertain in their outcome are the changes I have made to the peer review process, that forms a major component of the course for students who are taking it for a Certificate of Completion (particularly Completion with Distinction).

Give credit where credit is due? Maybe

Talking of which, the issue of credentialing continues to generate a lot of discussion. My course does not offer College Credit (and it is not clear any Stanford MOOC ever will), but just recently, the American Council on Education’s College Credit Recommendation Service (ACE CREDIT) has evaluated and recommended college credit be given for five MOOCs currently offered (by other universities) on Coursera. (Starting this March, it will be possible to take an enhanced version of my MOOC given by Stanford Online High School, for which a credential is awarded, but that course, aimed at high flying high school juniors and seniors, has a restricted enrollment and carries a fee, so it is not a MOOC, rather a course with tutors and assessment, built around my MOOC.)

But I digress. As I observed on a number of occasions in this blog and my MAA blog Devlin’s Angle, I see group work and peer evaluation as the key to making quality mathematics education available in a MOOC. So students who took the first version of my course and are planning on enrolling again (and I know many are) will see some changes there. Not huge ones. Like NASA’s first fumbling steps into space, I think it is prudent to make small changes that have a good chance of being for the better. But I learned a lot from my first trip into MOOC-space, and I expect to learn more, and make further changes, on my second flight.

Finally, if you want to learn more about my reflections on my first MOOC and MOOCs in general, and have a two hour car drive during which you would find listening to a podcast about MOOCs marginally better than searching through an endless cycle of crackly Country and Western radio stations, download the two podcast files from Wild About Math, where host Sol Lederman grills me about MOOCs.

MOOC reflections – and Coursera’s Business Model

Published December 5, 2012 Free online courses 6 Comments
Tags: Coursera, educational tsunami, higher education, Huffington Post, MAA, MOOC, online education

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

Two of my most recent reflections on MOOCs were in many ways reflections about mathematics education in general, so instead of burying them here, where only the MOOC-curious would see them, I submitted them to the Mathematical Association of America as articles in my monthly series “Devlin’s Angle”, and that’s where you will find them.

The first, titled MOOC lessons, focused on the kind of learning that can take place in a MOOC.

In the second, The Darwinization of Higher Education, I looked at the likely effect of MOOCs on the higher education landscape. I originally submitted a shorter, “less-personal-blog” version as a post to my blog in The Huffington Post (Education Section), but much to my surprise, after sitting on it for over a week, they rejected it. Either I am way ahead of the HuffPost’s education editors or else they think I’m off my rocker (maybe both). It probably reflects on the massive uncertainty about where MOOCs are going, that I think there is a 5% chance I am off my rocker on this issue. A week ago I would have put that figure at 10%, but just this morning (a scant week after I had submitted my thoughts to HuffPost, and a mere 24 hours after I had sent the piece instead to the MAA) I received an email Coursera sent out to all past and present students.

It began thus:

Career Services: Finding great job opportunities

Coursera has begun Career Services with the goal of helping Coursera students find great jobs! Meeting great companies just got easier. Just go to <Coursera web page URL> and fill in your profile to opt-in to the service. After you opt-in, we will share your resume and other information you provide with selected partner companies who will introduce themselves if there’s a match.
We’re excited to connect you with great companies and new opportunities! Complete your profile here!

Remember, you read it first here. (As they say.)

Is this really Coursera’s business model (in the sense of the business model)? I am in no position to know. I suspect they don’t yet know either, how (and maybe if) they, or any other MOOC platform, will eventually make sufficient revenue to sustain their activities. Calling it “Coursera’s business model” in my title indicates only that it is a business model that the company has now announced. From my second article listed above, you will gather I think it is a smart move on their part. On the other hand, I can think of at least half a dozen others ways to monetize MOOCs, and at least as many ways for others to build businesses around the MOOC phenomenon.

I know many of my academic colleagues feel uneasy when education is discussed as a for-profit enterprise, but it has never been anything else. Someone has to pay. Usually, it is the student or the student’s family, either directly or indirectly. The novel aspect of the Ivy League MOOCs that I hope those colleagues see as positive is that the one person who does not pay is the student — at least on entry, which means that MOOC education is entirely free.

While on the topic of MOOC upsides, I had lunch recently with three of my fellow pioneer MOOC-instructors, and one substantial student demographic we all noticed was moms with young children (in many cases single moms, without the means to afford child care while they study). Hard to fault that.

We are entering a very different world in terms of access to higher education.

To be continued …

Coming up for air (and spouting off)

Published November 26, 2012 Free online courses 11 Comments
Tags: Coursera, education design, edX, Facebook, Harvard University, higher education, Khan Academy, Larry Cuban, mathematical thinking, MIT, MOOC, online education, Sebastian Thrun, social media, Stanford Graduate School of Education, Stanford MOOC, Udacity

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

Almost a month has passed since I last posted to this blog. Keeping my MOOC running took up so much time that, once it was over, I was faced with a huge backlog of other tasks to complete. Taking a good look at the mass of data from the course is just one of several post-MOOC activities that will have to wait until the New Year. So readers looking for statistics, analyses, and conclusions about my MOOC will, I am afraid, have to wait a little bit longer. Like most others giving these early MOOCs, we are doing so on the top of our existing duties; the time involved has yet to be figured into university workloads.

One issue that came up recently was when I put on my “NPR Math Guy” hat and talked with Weekend Edition host Scott Simon about my MOOC experience.

In the interview, I remarked that MOOCs owed more to Facebook than to YouTube. This observation has been questioned by some people, who believe Kahn Academy’s use of YouTube was the major inspiration. In making this comment, they are echoing the statement made by former Stanford Computer Science professor Sebastian Thrun when he announced the formation of Udacity.

In fact, I made my comment to Scott with my own MOOC (and many like it) in mind. Though I have noted in earlier posts to this blog how I studied Sal Khan’s approach in designing my own, having now completed my first MOOC, I am now even more convinced than previously that the eventual (we hope) success of MOOCs will be a consequence of Facebook (or social media in general) rather than of Internet video streaming.

The reason why I felt sure this would be the case is that (in most disciplines) the key to real learning has always been bi-directional human-human interaction (even better in some cases, multi-directional, multi-person interaction), not unidirectional instruction.

What got the entire discussion about MOOCs off in the wrong direction – and with it the public perception of what they are – is the circumstance of their birth, or more accurately, of their hugely accelerated growth when a couple of American Ivy League universities (one of them mine) got in on the act.

But it’s important to note that the first major-league MOOCs all came out of Stanford’s Computer Science Department, as did the two spinoff MOOC platforms, Udacity and Coursera. When MIT teamed up with Harvard to launch their edX platform a few months later, it too came from their Computer Science Department.

And there’s the rub. Computer Science is an atypical case when it comes to online learning. Although many aspects of computer science involve qualitative judgments and conceptual reasoning, the core parts of the subject are highly procedural, and lend themselves to instruction-based learning and to machine evaluation and grading. (“Is that piece of code correct?” Let the computer run it and see if it performs as intended.)

Instructional courses that teach students how to carry out various procedures, which can be assessed to a large degree by automatic grading (often multiple choice questions) are the low hanging fruit for online education. But what about the Humanities, the Arts, and much of Science, where instruction is only a small part of the learning process, and a decidedly unimportant part at that, and where machine assessment of student work is at best a goal in the far distant future, if indeed it is achievable at all?

In the case of my MOOC, “Introduction to Mathematical Thinking,” the focus was the creative/analytic mathematical thinking process and the notion of proof. But you can’t learn how to think a certain way or how prove something by being told or shown how to do it any more than you can learn how to ride a bike by being told or shown. You have to try for yourself, and keep trying, and falling, until it finally clicks. Moreover, apart from some very special, and atypical, simple cases, neither thinking nor proofs can be machine graded. Proofs are more like essays than calculations. Indeed, one of the things I told my students in my MOOC was that a good proof is a story, that explains why something is the case.

For the vast majority of students, discussion with (and getting feedback from) professors, TAs, and other students struggling to acquire problem solving ability and master abstract concepts and proofs, is an essential part of learning. For those purposes, the online version does not find its inspiration in Khan Academy as it did for Thrun, but in Facebook, which showed how social interaction could live on the Internet.

When the online version of Thrun’s Stanford AI class attracted 160,000 students, he did not start a potential revolution in global higher education, but two revolutions, only the first of which he was directly involved in. The first one is relatively easy to recognize and understand, especially for Americans, who for the most part have never experienced anything other than instruction-based education.

For courses where the goal is for the student to achieve mastery of a set of procedures (which is true of many courses in computer science and in mathematics), MOOCs almost certainly will change the face of higher education. Existing institutions that provide little more than basic, how-to instruction have a great deal to fear from MOOCs. They will have to adapt (and there is a clear way to do so) or go out of business.

If I want to learn about AI, I would prefer to do so from an expert such as Sebastian Thrun. (In fact, when I have time, I plan on taking his Udacity course on the subject!) So too will most students. Why pay money to attend a local college and be taught by a (hopefully competent) instructor of less stature when you can learn from Thrun for free?

True, Computer Science courses are not just about mastery of procedures. There is a lot to be learned from the emphases and nuances provided by a true expert, and that’s why, finances aside, I would choose Thrun’s course. But at the end of the day, it’s the procedural mastery that is the main goal. And that’s why that first collection of Computer Science MOOCs has created the popular public image of the MOOC student as someone watching canned instructional videos (generally of short duration and broken up by quizzes), typing in answers to questions to be evaluated by the system.

But this kind of course occupies the space in the overall educational landscape that McDonalds does in the restaurant business. (As someone who makes regular use of fast food restaurants, this is most emphatically not intended as a denigratory observation. But seeing utility and value in fast food does not mean I confuse a Big Mac with quality nutrition.)

Things are very, very different in the Humanities, Arts, and most of Science (and some parts of Computer Science), including all of mathematics beyond basic skills mastery – something that many people erroneously think is an essential prerequisite for learning how to do math, all evidence from people who really do learn how to do math to the contrary.

[Ask the expert. We don’t master the basic skills; we don’t need them because, early on in our mathematic learning, we acquired one – yes, just one – fundamental ability: mathematical thinking. That’s why the one or two kids in the class who seem to find math easy seem so different. In general, they don’t find math easy, but they are doing something very different from everyone else. Not because they are born with a “math gene”. Rather, instead of wasting their time mastering basic skills, they spent that time learning how to think a certain way. It’s just a matter of how you devote your learning time. It doesn’t help matters that some people managed to become qualified math teachers and professors seemingly without figuring out that far more efficient path, and hence add their own voice to those who keep calling for “more emphasis on basic skills” as being an essential prerequisite to mathematical power.]

But I digress. To get back to my point, while the popular image of a MOOC centers on lecture-videos and multiple-choice quizzes, what Humanities, Arts, and Science MOOCs (including mine) are about is community building and social interaction. For the instructor (and the very word “instructor” is hopelessly off target in this context), the goal in such a course is to create a learning community. To create an online experience in which thousands of self-motivated individuals from around the world can come together for a predetermined period of intense, human–human interaction, focused on a clearly stated common goal.

We know that this can be done at scale, without the requirement that the participants are physically co-located or even that they know one another. NASA used this approach to put a man on the moon. MMOs (massively multiplayer online games – from which acronym MOOCs got their name) showed that the system works when the shared goal is success in a fantasy game world.

Whether the same approach works for higher education remains an open question. And, for those of us in higher education, what a question! A question that, in my case at least, has proved irresistible.

This, then, is the second MOOC revolution. The social MOOC. It’s outcome is far less evident than the first.

The evidence I have gathered from my first attempt at one of these second kinds of MOOC is encouraging, or at least, I find it so. But there is a long way to go to make my course work in a fashion that even begins to approach what can be achieved in a traditional classroom.

I’ll pursue these thoughts in future posts to this blog — and in future versions of my Mathematical Thinking MOOC, of which I hope to offer two variants in 2013.

Meanwhile, let me direct you to a recent article that speaks to some of the issues I raised above. It is from my legendary colleague in Stanford’s Graduate School of Education, Larry Cuban, where he expresses his skepticism that MOOCs will prove to be an acceptable replacement for much of higher education.

To be continued …

Peer grading: inventing the light bulb

Published October 31, 2012 Free online courses 29 Comments
Tags: calibrated peer review, Coursera, Edison, Facebook, mathematical thinking, MIT, MOOC, online education, social media, Stanford MOOC

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

With the deadline for submitting the final exam in my MOOC having now passed, the students are engaging in the Peer Evaluation process. I know of just two cases where this has been tried in a genuine MOOC (where the M means what it says), one in Computer Science, the other in Humanities, and both encountered enormous difficulties, and as a result a lot of student frustration. My case was no different.

Anticipating problems, I had given the class a much simplified version of the process – with no grade points at stake – at the end of Week 4, so they could familiarize themselves with the process and the platform mechanics before they had to do it for real. That might have helped, but the real difficulties only emerged when 1,520 exam scripts started to make their way through the system.

By then the instructional part of the course was over. The class had seen and worked through all the material in the curriculum, and had completed five machine-graded problem sets. Consequently, there were enough data in the system to award certificates fairly if we had to abandon the peer evaluation process as a grading device, as happened for that humanities MOOC I mentioned, where the professor decided on the fly to make that part of the exam optional. So I was able to sleep at night. But only just.

With over 1,000 of the students now engaged in the peer review process, and three days left to the deadline for completing grading, I am inclined to see the whole thing through to the (bitter) end. We need the data that this first trial will produce so we can figure out how to make it work better next time.

Long before the course launched, I felt sure that there were two things we would need to accomplish, and accomplish well, in order to make a (conceptual, proof-oriented) advanced math MOOC work: the establishment (and data gathering from) small study groups in which students could help one another, and the provision of a crowd-sourced evaluation and grading system.

When I put my course together, the Coursera platform supported neither. They were working on a calibrated peer review module, but implementing the group interaction side was still in the future. (The user-base growth of Coursera has been so phenomenal, it’s a wonder they can keep the system running at all!)

Thus, when my course launched, there was no grouping system, nor indeed any social media functionality other than the common discussion forums. So the students had to form their own groups using whatever media they could: Facebook, Skype, Google Groups, Google Docs, or even the local pub, bar, or coffee shop for co-located groups. Those probably worked out fine, but since they were outside our platform, we had no way to monitor the activity – an essential functionality if we are to turn this initial, experimental phase of MOOCs into something robust and useful in the long term.

Coursera had built a beta-release, peer evaluation system for a course on Human Computer Interaction, given by a Stanford colleague of mine. But his needs were different from mine, so the platform module needed more work – more work than there was really time for! In my last post, I described some of the things I had to cope with to get my exam up and running. (To be honest, I like the atmosphere of working in startup mode, but even in Silicon Valley there are still only 24 hours in a day.)

It’s important to remember that the first wave of MOOCs in the current, explosive, growth period all came out of computer science departments, first at Stanford, then at MIT. But CS is an atypical case when it comes to online learning. Although many aspects of computer science involve qualitative judgments and conceptual reasoning, the core parts of the subject are highly procedural, and lend themselves to instruction-based learning and to machine evaluation and grading. (“Is that piece of code correct?” Just see if it runs as intended.)

The core notion in university level mathematics, however, is the proof. But you can’t learn how to prove something by being told or shown how to do it any more than you can learn how to ride a bike by being told or shown. You have to try for yourself, and keep trying, and falling, until it finally clicks. Moreover, apart from some very special, and atypical, simple cases, proofs cannot be machine graded. In that regard, they are more like essays than calculations. Indeed, one of the things I told my students was that a good proof is a story, that explains why something is the case.

Feedback from others struggling to master abstract concepts and proofs can help enormously. Study groups can provide that, along with the psychological stimulus of knowing that others are having just as much difficulty as you are. Since companies like Facebook have shown us how to build platforms that support the creation of groups, that part can be provided online. And when Coursera is able to devote resources to doing it, I know it will work just fine. (If they want to, they can simply hire some engineers from Facebook, which is little more than a mile away. I gather that, like Google before it, the fun period there has long since passed and fully vested employees are looking to move.)

The other issue, that of evaluation and grading, is more tricky. The traditional solution is for the professor to evaluate and grade the class, perhaps assisted by one or more TAs (Teaching Assistants). But for classes that number in the tens of thousands, that is clearly out of the question. Though it’s tempting to dream about building a Wikipedia-like community of dedicated, math-PhD-bearing volunteers, who will participate in a mathematical MOOC whenever it is offered – indeed I do dream about it – it would take time to build up such a community, and what’s more, it’s hard to see there being enough qualified volunteers to handle the many different math MOOCs that will soon be offered by different instructors. (In contrast, there is just one Wikipedia, of course.)

That leaves just one solution: peer grading, where all the students in the class, or at least a significant portion thereof, are given the task of grading the work of their peers. In other words, we have to make this work. And to do that, we have to take the first step. I just did.

Knowing just how many unknowns we were dealing with, my expectations were not high, and I tried to prepare the students for what could well turn out to be chaos. (It did.) The website description of the exam grading system was littered with my cautions and references to being “live beta”. On October 15, when the test run without the grading part was about to launch, I posted yet one more cautionary note on the main course announcements page:

… using the Calibrated Peer Review System for a course like this is, I believe, new. (It’s certainly new to me and my assistants!) So this is all very much experimental. Please approach it in that spirit!

Even so, many of the students were taken aback by just how clunky and buggy the thing was, and the forums sprung alive with exasperated flames. I took solace in the recent release of Apple Maps on the iPhone, which showed that even with the resources and expert personnel available to one of the world’s wealthiest companies, product launches can go badly wrong – and we were just one guy and two part-time, volunteer student assistants, working on a platform being built under us by a small startup company sustained on free Coke and stock options. (I’m guessing the part about the Coke and the options, but that is the prevalent Silicon Valley model.)

At which point, one of those oh-so-timely events occurred that are often described as “Acts of God.” Just when I worried that I was about to witness, and be responsible for starting, the first global, massive open online riot (MOOR) in a math class, Hurricane Sandy struck the Eastern Seaboard, reminding everyone that a clunky system for grading math exams is not the worst thing in the world. Calm, reasoned, steadying, constructive posts started to appear on the forum. I was getting my feedback after all. The world was a good place once again.

Failure (meaning things don’t go smoothly, or maybe don’t work at all) doesn’t bother me. If it did, I’d never have become a mathematician, a profession in which the failure rate in first attempts to solve a problem is somewhere north of 95%. The important thing is to get enough data to increase the chances of getting it right – or far more likely, just getting it better – the second time round. Give me enough feedback, and I count that “failure” as a success.

As Edison is said to have replied to a young reporter about his many failed attempts to construct a light bulb, “Why would I ever give up? I now know definitively over 9,000 ways that an electric light bulb will not work. Success is almost in my grasp.” (Edison supposedly failed a further 1,000 times before he got it right. Please don’t tell my students that. We are just at failure 1.)

If there were one piece of advice I’d give to anyone about to give their first MOOC, it’s this: remember Edison.

To be continued …

It’s About Time (in Part): MOOC Planning – Part 10

Published October 22, 2012 Free online courses 6 Comments
Tags: Big Data, calibrated peer review, Coursera, education design, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, Unix Time, WYSIWOSG

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

Well, lectures have ended and the course has now switched gears. For those still left in the course (17% of the final enrollment total of 64,045), the next two weeks are focused on trying to make sense of everything they have learned, and working on the final exam — which in the case of my course involves peer evaluation.

Calibrated Peer Review is not new. A study of its use in the high school system by Sadler and Good, published in 2006, has become compulsory reading for those of us planning and giving MOOCs that cover material that cannot be machine graded. [If you want to see how I am using it, just enroll in the class and read the description of the “Peer Review system”. There is no obligation to do anything more than browse around the site! No one will know you are not simply a dog that can use a computer.]

As I was working on my course, Coursera was still frantically building out their platform to support peer evaluation. There was a lot of just-in-time construction. It’s been a long time since I’ve had to go behind a user-friendly interface and dig into the underlying code to do something on a computer, and the programming languages have all changed since I last did that.

One thing I had to learn was one of the ways networked computers keep time. I now know that at the time of writing these words, 7:00AM Pacific Daylight Time on October 22, 2012, exactly 1,350,914,400 seconds have elapsed since the first second of January 1st, 1970, Eastern Standard Time. That was the start of Unix Time.

I needed to learn to work in Unix Time in order to set the various opening times and completion deadlines for the exam process. I expect that by the time the next instructor puts together a MOOC, she or he will be greeted by a nice, friendly Coursera interface with pulldown menus and boxes to tick — which probably will come as a great relief to any humanities professors reading this, who don’t have any programming in their background.

[By coincidence, Unix was the last programming language I had any proficiency in, but I did not need to know Unix to use Unix Time. I just used an online converter. Unix was developed in 1969 at AT&T Bell Laboratories in New Jersey. Hence the 1970 EST baseline.]

In fact, time conversion issues in general turned out to be a continuing, major headache in a course with students all over the world. One thing we will not do again is have 12:00PM Stanford Time, aka Coursera Time (i.e., PDT), as any of the course deadlines. It might seem a nice clean stopping point, and there are all those memories of Gary Cooper’s deadline in the classic Western movie High Noon, but many students missed the deadline for the first submitted assignment because they thought 12:00PM meant midnight, which in some parts of the world made them a whole day late.

The arbitrary illogicality of the AM/PM distinction is not apparent to those of us who grew up with it. But my course TA and I are now very aware of the problems it can lead to! In future, we’ll stick to unambiguous times that stay away from noon and midnight. But even then, with local computer systems usually working on local time, to say nothing of the different Summer and Winter Times, which change on different dates around the world, timing events in MOOCs is going to remain a problematic issue, just as it is for international travelers and professionals who collaborate globally over Skype and other conferencing services. (When I used the Unix Time conversion app, I had to remember that Unix thinks New Jersey is currently just two hours ahead of California, not the three hours United Airlines uses when it flies me there. Confusing, isn’t it?)

The reason why times are an issue in my course is that it is a course. At first glance, it may look little different from Khan Academy, where there are no time issues at all. But Khan Academy is really just an educational resource. (At least, that’s the part most people are familiar with and use, namely the video library that started it all. People use it as a video version of a textbook — or more precisely a video equivalent to that good old standby Cliffs Notes, which got many of us through an exam in an obligatory subject we were not particularly interested in.)

In contrast, in my case, as I’ve discussed earlier in this blog series (in particular, Part 6), my goal was to take a standard university course (one I’ve given many times over the years, at different universities, including Stanford) and make it available to anyone in the world, for free. To the degree I could make it happen, they would get the same learning experience.

That meant that the main goal would be to build a (short-lived) learning community. The video-recorded lectures and tutorials were simply tools to make that happen and to orchestrate events. Real learning takes place when students work on assignments on their own, when they repeatedly fail to solve a problem, and when they interact (with the professor and with one another) — not when they watch a lecture or read a book.

To achieve that goal, the MOOC would, as I stated in Part 6, involve admissions, lectures, peer interaction, professor interaction, problem-solving, assignments, exams, deadlines, and certification. To use the mnemonic I coined early on in this series, the basic design principle is WYSIWOSG: What You See Is What Our Students Get.

As we go forward, I intend to iterate on the course design, based on the data we collect from the students (and 64,000 students very definitely puts us into the Big Data realm). But my basic principle will remain that of offering a course, not the provision of a video library. And the reason for that should be obvious to anyone who has been following this blog series, as well as some of the posts on my other blogs Devlin’s Angle and profkeithdevlin.org. The focus is not on acquiring facts or mastering basic skills, but on learning to think a certain way (in my case, like a professional mathematician). And that requires both a lot of effort and (for most of us) a lot of interaction with others trying to achieve the same goal.

Our ancestors in the 11th Century started to develop what to this day remains the best way we know to achieve this at scale: the university, where people become members of a learning community in which learning takes place in a hothouse atmosphere that involves periods of intense interaction as deadlines loom, sustained by the rapidly formed social bonds that emerge as a result of that same pressure.

While I will likely experiment with variants of this model that allow for participation by students who have demanding, full-time jobs, I doubt I will abandon that basic model. It has lasted for a thousand years for a good reason. It works.

To be continued …

Final Lecture: MOOC Planning – Part 9

Published October 20, 2012 Free online courses 8 Comments
Tags: calibrated peer review, Coursera, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, transition course, transition math, WYSIWOSG

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I gave my last lecture of the course yesterday (discounting the tutorial session that will go out next week), and we are now starting a two week exam period.

“Giving” a lecture means the video becomes available for streaming. For logistic reasons (high among them, my survival and continued sanity — assuming anyone who organizes and gives a MOOC, for no payment, is sane), I recorded all the lectures weeks ago, well before the course started. The weekly tutorial sessions come the closest to being live. I record them one or two days before posting, so I can use them to respond to issues raised in the online course discussion forum.

The initial course enrollment of 63,649 has dropped to 11,848 individuals that the platform says are still active on the site. At around 20%, that’s pretty high by current MOOC standards, though I don’t know whether that is something to be pleased about, since it’s not at all clear what the right definition of “success” is for a MOOC.

Some might argue that 20% completion indicates that the standards are too low. I don’t think that’s true for my course. Completion does, after all, simply mean that a student is still engaged. The degree to which they have mastered the material is unclear. So having 80% drop out could mean the standard is too high.

In my case, I did not set out to achieve any particular completion rate; rather I adopted a WYSIWOSG approach — “What You See Is What Our Students Get.” I offered a MOOC that is essentially the first half of a ten week course I’ve given at many universities over the years, including Stanford. That meant my students would experience a Stanford-level course. But they would not be subject to passing a Stanford-level exam.

In fact, I could not offer anything close to a Stanford-exam experience. There is a Final Exam, and it has some challenging questions, but it is not taken under controlled, supervised conditions. Moreover, since it involves constructing proofs, it cannot be machine graded, and thus has to be graded by other students, using a crowd sourcing method (Calibrated Peer Review). That put a significant limitation on the kinds of exam questions I could ask. On top of that, the grading is done by as many different people as there are students, and I assume most of them are not expert mathematicians. As a result, it’s at most a “better-than-nothing” solution. Would any of us want to be treated by a doctor whose final exam had been peer graded (only) by fellow students, even if the exam and the grading had been carried out under strictly controlled conditions?

On the other hand, looking at and attempting to evaluate the work of fellow students is a powerful learning experience, so if you view MOOCs as vehicles for learning, rather than a route to a qualification, then peer evaluation has a lot to be said for it. Traditional universities offer both learning and qualifications. MOOCs currently provide the former. Whether they eventually offer the latter as well remains to be seen. There are certainly ways it can be done, and that may be one way that MOOCs will make money. (Udacity already does offer a credentialing option, for a fee.)

In designing my course, I tried to optimize for learning in small groups, perhaps five to fifteen at a time. The goal was to build learning communities, within which students could help one another. Since there is no possibility of regular, direct interaction with the instructor (me) and my one TA (Paul), students have to seek help from fellow students. There is no other way. But, on its own, group work is not enough. Learning how to think mathematically (the focus of my course) requires feedback from others, but it needs to include feedback from people already expert in mathematical thinking. This means that, in order to truly succeed, not only do students need to work in groups (at least part of the time), and subject their attempts to the scrutiny of others, some of those interactions have to be with experts.

One original idea I had turned out not to work, though whether through the idea itself being flawed or the naive way we implemented it is not clear to me. That was to ask students at the start of the course to register if they had sufficient knowledge and experience with the course material to act as “Community TAs”, and be so designated in the discussion forums. Though over 600 signed up to play that role, many soon found they did not have sufficient knowledge to perform the task. Fortunately,a relatively small number of sign-ups did have the necessary background, as well as the interpersonal skills to give advice in a supporting, non-threatening way, and they more or less ensured that the forum discussions met the needs of many students (or so it seems).

Another idea was to assign students to study groups, and use an initial survey to try to identify those with some background knowledge and seed them into the groups. Unfortunately, Coursera does not (yet) have functionality to support the creation and running of groups, apart from the creation of forum threads. So instead, in my first lecture, I suggested to the students that they form their own study groups in whatever way they could.

The first place to do that was, of course, the discussion forums on the course website, which very soon listed several pages of groups. Some used the discussion forum itself to work together, while others migrated offsite to some other location, physical or virtual, with Skype seeming a common medium. Shortly after the course launched, several students discovered GetStudyRoom, a virtual meeting place dedicated to MOOCs, built by a small startup company.

In any event, students quickly found their own solutions. But with students forming groups in so many different ways on different media, there was no way to track how many remained active or how successful they have been.

The study groups listed on the course website show a wide variety of criteria used to bring the groups together. Nationality and location were popular, with groups such as Brazil Study Group, Grupo de Estudo Português, All Students From Asia, and Study Group for Students Located in Karachi, Pakistan. Then there were groups with a more specific focus, such as Musicians, Parents of Homeschooled Children, Older/Retired English Speakers Discussion for Assignment 1, and, two of my favorites, After 8pm (UK time) English speakers with a day job and the delightfully named Just Hanging on Study Group.

The forum has seen a lot of activity: 15,088 posts and 13,622 comments, spread across 2712 different threads, viewed 430,769 times. Though I have been monitoring the forums on an almost daily basis, to maintain an overall sense of how the course is going, it’s clearly not possible to view everything. For the most part I restricted my attention to the posts that garnered a number of up-votes. Students vote posts up and down, and once a post shows 5 or more up-votes, I take that as an indication that the issue may be worth looking at.

The thread with the highest number of up-votes (165) was titled Deadlines way too short. Clearly, the question of deadlines was a hot topic. How, if at all, to respond to such feedback is no easy matter. In a course with tens of thousands of students, even a post with hundreds of up-votes represents just a tiny fraction of the class. Moreover, threads typically include opinions on both sides of an issue.

For instance, in threads about the pace of the course, some students complained that they did not have enough time to complete assignments, and pleaded for more relaxed deadlines, whereas others said they thrived on the pace, which stimulated them to keep on top of the material. For many, an ivy-league MOOC offers the first opportunity to experience an elite university course, and I think some are surprised at the level and pace. (I fact, I did keep the pace down for the first three weeks, but I also do that when I give a transition course in a regular setting, since I know how difficult it is to make that transition from high school math to university level mathematics.)

A common suggestion/request was to simply post the course materials online and let students access them according to their own schedules, much like Khan Academy. This raises a lot of issues about the nature of learning and the role MOOCs can (might? should?) play. But this blog post has already gone on long enough, so I’ll take up that issue next time.

To be continued …

The Crucible: MOOC Planning – Part 8

Published October 6, 2012 Free online courses 23 Comments
Tags: Big Data, Coursera, higher education, mathematical thinking, MOOC, online education, Stanford School of Education, transition course, transition math

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

Well, I have survived the initial three weeks of my first MOOC. Though the bulk of the work (and I mean “bulk”) came before the course launched, it has still taken my TA and me a lot of time to keep things ticking over. There are the in-flight corrections of the inevitable errors that occur in a new course, together with the challenges presented by a completely new medium and a buggy, beta release platform, still under very rapid development.

The course website shows 61,846 registered students, but I suspect many of those have long stopped any kind of connection to the course, and another large group are simply watching the lecture videos. The really pleasing figure is that the number of active users last week (week 3) was 19,298. Based on what I hear about other MOOCs, retaining one student in three is a good number.

Both my hands-on TA, Paul, and the course Research Associate, Molly, are graduate students in Stanford’s School of Education, and besides helping me with aspects of the course design, they are approaching the project as an opportunity to carry out research in learning, particularly mathematics learning. Given the massive amount of data a MOOC generates, the education research world can expect to see a series of papers coming from them in the months ahead.

I’m not trained in education research, but some observations are self-evident when you look over the course discussion forums – something I’ve spent a lot of time doing, both to gauge how the course is going and to look for ways to improve it, either by an in-course modification of for a future iteration of the course.

I’ve always felt that the essence of MOOC learning is community building. There is no hope that the “instructor” can do more than orchestrate events. Without regular close contact with the students, the video-recorded lectures and the various course notes and handouts are like firing off a shotgun on a misty Scottish moor. The shot flies out and disperses into the mist, and you just hope some of it hits a target. (I haven’t actually fired a shotgun on a Scottish moor, or anywhere else for that matter, but I’ve seen it on TV and it seems the right metaphor.) With 60,000 (or 20,000) students, I can’t allow myself to respond to a forum post or an email from any single student. I have to rely on the voting procedure (“Like/Dislike”) of the forums to help me decide which questions to address.

This means the student body has to resolve things among themselves. It was fascinating watching the activity on the discussion forums take shape and develop a profile over the first couple of weeks.

One huge benefit for the instructor is the virtual elimination of the potentially disruptive influence – present in almost any class with more than twenty or so students – of the small number of students for whom nothing is good enough. Even in a totally free course, put on by volunteers, for which no college credential is awarded, there were a few early posts of that kind. But in each case the individual was rapidly put in his or her place by replies from other students, and before long stopped posting, and very likely dropped the course.

(An interesting feature of this was that each time it occurred, a number of students emailed me in private – rather than on the public course forum – to say they did not agree with the complainer, and to tell me they were enjoying the course. Clearly, even with the possibility of anonymous forum posts, which Coursera allows, at least for now, some people prefer to keep their communication totally private.)

Of far greater interest, at least to me, was how the student body rapidly split into two camps, based on how they reacted to the course content. As I’ve discussed in earlier posts to this blog, my course is a high-school to university transition course for mathematics. It’s designed to help students make the difficult (and for most of us psychologically challenging) transition from high school mathematics, with its emphasis on learning to follow procedures to solve highly contrived “math problems”, to developing an ability to think logically, numerically, analytically, quantitatively, and algebraically (i.e., in aggregate, mathematically) about novel problems, including often ill-defined or ambiguous real-world problems.

When I give this kind of course to a traditional class of twenty-five or so entering college students, fresh out of high school, the vast majority of them have a really hard time with it. In my MOOC, in contrast, the student body has individuals of all ages, from late teens into their sixties and seventies, with different backgrounds and experiences, and many of them said they found this approach the most stimulating mathematics class they had ever taken. They loved grappling with the inherent ambiguity and open-ended nature of some of the problems.

Our schools (at least in the US), by focusing on one particular aspect of mathematics – the formal, procedural – I think badly shortchange our students. They send them into the world with a fine scalpel, but life in that world requires a fairly diverse toolkit – including WD40 and a large roll of duct tape.

The real world rarely presents us with neat, encapsulated problems that can be solved in ten minutes. Real world problems are messy, ambiguous, ill-defined, and often with internal contradictions. Yes, precise, formal mathematics can be very useful in helping to solve such problems. But of far broader applicability is what I have been calling “mathematical thinking”, the title of my course.

I suspect the students who seemed to take to my course like ducks to water were people well beyond high school, who had discovered for themselves what is involved in solving real problems. Judging by the forum discussions, they are having a blast.

The others, the ones whose experience of mathematics has, I suspect, been almost entirely the familiar, procedural-skills learning of the traditional K-12 math curriculum, keep searching for precision that simply is not there, or (and I’ve been focusing a lot on this in the first three weeks) where the goal is to learn how to develop that precision in the first place.

The process of starting with a messy, real world problem, where we have little more than our intuitions to guide us, and then slowly distilling some precision to help us deal with that problem, is hugely valuable. Indeed, it is the engine that powered (and continues to power) the entire development of our science and our technology. Yet, in our K-12 system we hardly ever help students to learn how to do that.

Done well, the activities of the traditional math class can be great fun. I certainly found it so, and have spent a large part of my life enjoying the challenges of pure mathematics research. But a lot of that fun comes from working within the precise definitions and clear rules of engagement of the discipline. To me mathematics was chess on steroids. I loved it. Still do, for that matter. But relatively few citizens are interested in making a career in mathematics. An education system that derives its goals from the ivory-towered pursuit of pure mathematics (and I use that phrase in an absolutely non-denigrating way, knowing full well how important it is to society and to our culture that those ivory towers exist) does not well serve the majority of students.

It requires some experience and sophistication in mathematics to see how skill in abstract, pure reasoning plays an important role in dealing with the more messy issues of the real world. There is an onus on those of us in the math ed community to help others to appreciate the benefits available to them by way of improved mathematical ability.

As I have followed the forum discussions in my MOOC, I have started to wonder if one thing that MOOCs can give to mathematics higher education in spades is a mechanism to provide a real bridge between K-12 education and life in the world that follows. By coming together in a large, albeit virtual community, the precision-seeking individuals who want clear rules and guidelines to follow find themselves side-by-side (actually, keyboard-to-keyboard) with others (perhaps with weak formal mathematics skills) more used to approaching open-ended, novel problems of the kind the real world throws up all the time. If so, that would make the MOOC a powerful crucible that would benefit both groups, and thus society at large.

To be continued …

MOOCtalk

Posts Tagged 'Coursera'

MathThink MOOC v4 – Part 6

The MOOC Express – Less Hype, More Hope

How are MOOCs organized?

Here we go again

MOOC reflections – and Coursera’s Business Model

It’s About Time (in Part): MOOC Planning – Part 10

Final Lecture: MOOC Planning – Part 9

The Crucible: MOOC Planning – Part 8

profkeithdevlin

Devlin’s Angle (MAA)

Twitter Updates

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011

Posts Tagged 'Coursera'

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011