MOOC | MOOCtalk

Posts Tagged 'MOOC'

Evaluation rubrics: the good, the bad, and the ugly

Published August 13, 2013 Free online courses 6 Comments
Tags: calibrated peer review, education design, evaluation rubric, mathematical thinking, MOOC, online education, transition course, transition math

A real-time chronicle of a seasoned professor just about to launch the third edition of his massively open online course.

With the third session of my MOOC Introduction to Mathematical Thinking starting on September 2, I am busy putting the final touches to the course materials. As I did when I offered the second session earlier this year, I have made some changes to the way the course is structured. The underlying content remains the same, however – indeed at heart it has not changed since I first began teaching a high school to university “transition” course back in the late 1970s, when I was a young university lecturer just starting out on my career.

With the primary focus on helping students develop an new way of thinking, the course was always very light on “content” but high on internal reflection. A typical assignment question might require four or five minutes to write out the answer; but getting to the point where that is possible might take the student several hours of thought, sometimes days. Students who approach the course thinking it is an introductory course on logic – some of whom likely will, as they have in the past, post on the course forum that they cannot understand why I am proceeding so slowly and making such heavy weather of the material – will, if they don’t walk away in disgust, eventually (by about week four) realize they are completely lost. Habituated to courses that rush through a pile of material that required mostly procedural mastery, they find it challenging, and in many cases impossible, to slow down and adopt the questioning, reflective approach this course requires.

My course uses elementary linguistics and formal logic as a vehicle to help develop new thinking skills that are essential for university mathematics majors, very valuable for STEM majors, and of considerable value for anyone who wants to lead a more rewarding life. But it is definitely not a course in linguistics or logic. It is about thinking.

Starting with an analysis of certain features of ordinary language, as I do, provides a starting point that is accessible to everyone – though because the language I examine is English, students for whom that is a second language are at a disadvantage. That is unavoidable. (A Spanish language version, embedded in Hispanic culture, is currently under development. I hope other deep translations follow.)

And formal logic is so simple and structured, and so accessible to a beginner, that it too is well suited to an introductory level course on analytic, and in particular mathematical, thinking.

Why my course videos are longer than most

The imperative of a student devoting substantial periods of time engaged in sustained contemplation of the course material has led to me making two decisions that go against the current grain in MOOCs. First, the pace is slow. I speak far more slowly than I normally do, and I repeat each point at least once, and often more so. Second, I do not break my “lectures” into the now-almost-obligatory no-longer-than-seven-and-ideally-under-three-minutes snippets. For the course’s second running, I did split the later hour or more long videos into half-hour sections, but that was to make it easier for students without fast broadband access, who have to download the videos overnight to watch them.

Of course, students can speed up or slow down the videos, they can watch them as many times as they want, and they can stop and start them to suit their schedules. But then they are in control and make those decisions based on their own progress and understanding. My course does not come pre-digested. It is slow cooking, not fast food.

Learning by evaluation

The main difference returning students will notice in the new session is the much greater emphasis on developing evaluation skills. Fairy early in the course, students will be presented with purported mathematical proofs that they have to evaluate according to a grading rubric.

At first these will be fairly short arguments, designed by me to illustrate various key features of proofs, and often incorporating common mistakes beginners make. Later on, the complexity increases. For those students who elect to take the final exam (and thereby become eligible to earn a Distinction grade for the course), evaluation will culminate in grading three randomly assigned, anonymized exam submissions from fellow students, followed by grading their own submission.

Peer evaluation is essential in MOOCs that involve work that cannot be machine graded, definitely the category into which my Mathematical Thinking course falls. The method I use for the Final Exam is called Calibrated Peer Review. It has a long history and proven acceptable results. (I describe it in some detail on my MOOC course website – accessible to anyone who signs up for the course.) So adopting peer evaluation for my course was unavoidable.

The first time I offered the course, I delayed peer evaluation until the final couple of weeks, when it was restricted to the final exam. Though things went better than I had feared, there were problems. The main issues, which came as no surprise, were, first, that many students felt very uneasy grading the work of others, second, many of them did not do a good job, and third, the rubric (which I had taken off another university’s Internet shelf) did not work at all well.

On the other hand, many students posted forum comments saying they found they enjoyed that part of the course, and learned more in those final two weeks than in the entire earlier part of the course.

I had in fact expected this would be the case, and had told the class early on that many of them would have that reaction. In particular, evaluating the work of fellow students is a very powerful, known way to learn new material. Nevertheless, it came as a great relief when this actually transpired.

As a result of my experience in the first session, when I gave the course a second time this spring, I increased the number of assignment exercises that required students to evaluate purported proofs. I also altered the rubric to make it better suited to what I see as the main points in the course.

The outcome, as far as I could ascertain from reading the comments student posted on the course discussion forum, was that it went much better. But it was still far from perfect. The two main issues were the rubric itself and how to use it.

Designing a rubric

Designing a good rubric is not at all easy for any course, and I think particularly challenging for a course on more advanced parts of mathematics. Qualitative grading of mathematical arguments, like grading essays or works of art, is a holistic skill that takes years to acquire to a degree it can be used to evaluate performance with some degree of reliability. A beginner attempting evaluation needs guidance, most typically provided by an evaluation rubric. The idea is to replace the holistic application of a lifetime’s acquisition of tacit domain knowledge with a number of categories that the evaluator should look for.

The more fine-grained the rubric, the easier it will be for the novice evaluator, but the more onerous the grading task becomes. The rubric I started with for my course had six factors, which I felt was about right – enough to make the task doable for the student yet not too many to turn it into a dull chore. I have retained that number. But, based on the experiences of students using the rubric, I changed several categories the first time I repeated the course and I have changed one category for the upcoming third session.

In each of the six categories in the rubric, the student must chose between three levels, which I name Novice, Apprentice, and Practitioner. I chose the names to emphasize that we are using evaluation as a way to learn, and the focus is to measure progress along a path of development, not assign summative performance judgments of “poor”, “okay”, and “good”.

The intention in having just three levels is to force a student evaluator to make a decision about the work being assessed. But this can be particularly difficult for a beginner who is, of course, lacking in confidence in their ability to do that. To counter that, in this third session, when the student enters the numerical value that course software will use to track progress, the numerical equivalents to those three categories are not 0, 1, 2, but 0, 2, and 4. The student can enter 1 or 3 as a “middle value” if they are undecided as to which category to assign.

Using the rubric

Even with “middling” grades available for the rubric items, most students will find the evaluation process difficult and very time consuming. A rubric simply breaks a single evaluation task into a number of smaller evaluation tasks, six in my case. In so doing, it guides the student as to what things to look for, but the student still has to make qualitative judgments within each of the categories.

To help them make these judgments, the last time I gave the course, I provided them with tutorial videos that take them through the grading process. I record myself grading the same sample arguments that they have just attempted to evaluate, verbalizing my thinking process as I go, explaining why I make the calls I do. They are not the most riveting of videos, and they can be a bit long (ten minutes for some assignment questions). But I don’t know of any other way of conveying something of the expertise I have built up over a lifetime. It is essentially a modern implementation of the age-old apprentice system of acquiring tacit knowledge by working alongside the expert.

Unfortunately, as an expert, I make calls based on important distinctions that for me jump from the student’s page, but are not even remotely apparent to a beginner. The result last time was, for some questions, considerable frustration on the part of the students.

To try to mitigate this problem (I don’t think it can be eliminated), I changed some aspects of the way the rubric is formulated and described, and decided to introduce the entire evaluation notion much earlier in the course. The result is that evaluation is now a very central component of the course. Indeed, evaluating mathematical arguments now plays a role equal to constructing them.

If it goes well – and based on my previous experience with this course, I think it will go better than last time – I will almost certainly adopt a similar approach if and when I give the course in a traditional classroom setting once again. (A heavy travel schedule associated with running a research lab means I have not taught a regular undergraduate class for several years now, though an attractive offer to spend a term at Princeton early next year will give me a much welcomed opportunity to spend some time in the classroom once again.)

Evaluating to learn, not to grade

One feature of a MOOC – or at least a MOOC like mine that does not offer college credit – is that the focus is on learning, not acquiring a credential. Thus, grading can be used entirely for formative purposes, as a guide to progress, not to provide a summative measure of achievement. As an instructor, I find the separation of the teaching and the grading extremely freeing. For one thing, with the assignment of grades out of the picture, the relationship between teacher and student is changed significantly. Also, it means numerical grades can be used as useful indicators of progress. A grade of 35% can be given for a piece of work annotated as “good” (i.e., good for someone taking an introductory course for the first time). The number indicates how much improvement would be required to take the student to the level of an expert practitioner.

To be sure, students who encounter this use of grades for the first time find it takes some getting used to. They are so habituated to the (nonsensical but widespread) notion that anything less than an A is a “failure” that they can be very discouraged when their work earns them a “mere” 35%. But in order to function as a school-to-university transition course, it has to help them adjust to a world where 35% if often a respectable passing grade.

(A student who regularly scores in the 90% range in advanced undergraduate mathematics courses can likely jump straight into a Ph.D. program – and some have done just that. 35% really can be a good result for a beginner.)

One final point about peer evaluation is an issue I encountered last time that surprised me, though perhaps it should not have, given everything I know about a lot of high school mathematics instruction. Many students approached grading the work of others as a punitive process of looking to deduct points. Some went so far as to complain (sometimes angrily) on the discussion forums about my video-streamed grading as being far too lenient.

In fact, one or two even held the view that if a mathematical argument was not logically correct, the only possible grade to give was 0. This particular perspective worried me on two counts.

Firstly, it assumes a degree of logical infallibility that no living mathematician possesses. I doubt there is a single published mathematical proof of more than a few paragraphs that does not include some minor logical slips, and hence is technically incorrect. (Most of the geometric proofs in Euclid’s Elements would score 0 if logical correctness were the sole metric!)

Second, my course is not a mathematics course, it is about mathematical thinking, and has the clearly stated aim of looking at the many different aspects of mathematical arguments required to make them “good.” Logical correctness is just one item on that six-point rubric. As a result, at most 4 of the possible 24 points available can be deducted in an argument is logically incorrect. (Actually, 8 can be deducted, as the final category is “Overall assessment”, designed to encourage precisely what the phrase suggest.)

To be sure, if my course were a mathematics course, I would assign greater weight to logical correctness. As it is, all six categories carry equal weight. But that is deliberate. Most of my students’ entire mathematical education has been in a world where “getting the right answer” is the holy grail. One other objective of transition courses is to break them of that debilitating default assumption.

Finally, and remember, this is for posterity, so be honest. How do you feel?

I’ve written elsewhere that I think MOOCs as such will not be the cause of a revolution in higher education. Rather they are just part of what is more like to be an evolution, though a major one to be sure. From the point of view of an instructor, though, they are providing us with a wonderful domain to re-examine all of our assumptions about how to teach and how students learn. As you can surely tell, I continue to have a blast in the MOOCasphere.

To be continued …

The MOOC will soon die. Long live the MOOR

Published June 3, 2013 Free online courses 16 Comments
Tags: Khan Academy, mathematical thinking, MOOC, MOOR, online education, Stanford MOOC

A real-time chronicle of a seasoned professor who just completed giving his second massively open online course.

The second running of my MOOC (massive open online course) Introduction to Mathematical Thinking ended recently. The basic stats were:

Total enrollment: 27,930

Number still active during final week of lectures: ca 4,000

Total submitting exam: 870

Number of students receiving a Statement of Accomplishment: 1,950

Number of students awarded a SoA with Distinction: 390

From my perspective, it went better than the first time, but this remains very much a research project, and will do for many more iterations. It is a research project with at least as many “Can we?” questions as “How do we?”

From the start, I took the viewpoint that, given the novelty of the MOOC platform, we need to examine the purpose, structure, and use of all the familiar educational elements: “lecture,” “quiz,” “assignment,” “discussion,” “grading,” “evaluation,” etc. All bets are off. Some changes to the way we use these elements might be minor, but on the other hand, some could be significant.

For instance, my course is not offered for any form of college credit. The goal is purely learning. This could be learning solely for its own sake, and many of my students approached it as such. On the other hand, as a course is basic analytic thinking and problem solving, with an emphasis on mathematical thinking in the second half of the course, it can clearly prepare a student to take (and hopefully do better in) future mathematics or STEM courses that do earn credit – and I have had students taking it with that goal in mind.

Separating learning from evaluation of what has been learned is enormously freeing, both to the instructor and to the student. In particular, evaluation of student work and the awarding of grades can be devoted purely to providing students with a useful (formative) indication of their progress, not a (summative) measure of their performance or ability.

To be sure, many of my students, conditioned by years of high stakes testing, have a hard time adjusting to the fact that a grade of 30% on a piece of work can be very respectable, indeed worth an A in many cases.

My typical response to students who lament their “low” grade is to say that their goal should be that a problem for which they struggle to get 30% in week 2 should be solvable for 80% or more by week 5 (say). And for problems they struggle with in week 8 (the final week of curriculum in my course), they should be able to do them more successfully if they take the course again the next time it is offered – something else that is possible in the brave new world of MOOCs. (Many of the students in my second offering of the course had attempted the first one a few months earlier.)

Incidentally, I think I have to make a comment regarding my statement above that the MOOC platform is novel. A number of commentators have observed that “online education is not new,” and they are right. But they miss the point that even this first generation of MOOC platforms represents a significant phase shift, not only in terms of the aggregate functionality but also the social and cultural context in which today’s MOOCs are being offered.

Regarding the context, not only have many of us grown accustomed to much of our interpersonal interaction being mediated by the internet, the vast majority of people under twenty now interact far more using social media than in person.

We could, of course, spend (I would say “waste”) our time debating whether or not this transition from physical space to cyberspace is a good thing. Personally, however, I think it is more productive to take steps to make sure it is – or at least ends up – a good thing. That means we need to take good education online, and we need to do so for the same reason that it’s important to embed good learning into video games.

The fact is, we have created for the new and future generations a world in which social media and video games are prevalent and attractive – just as earlier generations created worlds of books and magazines, and later mass broadcast media (radio, films, television) which were equally as widespread and attractive in their times. The media of any age are the ones through which we must pass on our culture and our cumulative learning. (See my other blog profkeithdevlin.org for my argument regarding learning in video games.)

Incidentally, I see the points I am making here (and will be making in future posts) as very much in alignment with, and definitely guided by, the views Sir Ken Robinson has expressed in a series of provocative lectures, 1, 2, 3.

Sir Ken’s thoughts influenced me a lot in my thinking about MOOCs. To be sure, there is much in the current version of my MOOC that looks very familiar. That is partly because of my academic’s professional caution, which tells me to proceed in small steps, starting from what I myself am familiar with; but in part also because the more significant changes I am presently introducing are the novel uses I am making (or trying to make) of familiar educational elements.

The design of my course was also heavily influenced by the expectation (more accurately a recognition, given how fast MOOCs are developing) that no single MOOC should see itself as the primary educational resource for a particular learning topic. Rather, those of us currently engaged in developing and offering MOOCs are, surely, creating resources that will be part of a vast smorgasbord from which people will pick and choose what they want or need at any particular time.

Given the way names get assigned and used, we may find we are stuck with the name MOOC (massive open online course), but a better term would be MOOR, for massive open online resource.

For basic, instructional learning, which makes up the bulk of K-12 mathematics teaching (wrongly in my view, but the US will only recognize that when virtually none of our home educated students are able to land the best jobs, which is about a generation away), that transition from course to resource has already taken place. YouTube is littered with short, instructional videos that teach people how to carry out certain procedures.

[By the way, I used the term “mathematical thinking” to describe my course, to distinguish it from the far more prevalent instructional math course that focuses on procedures. Students who did not recognize the distinction in the first three weeks, and approached the material accordingly, dropped out in droves in week four when they suddenly found themselves totally lost.]

By professional standards, many of the instructional video resources you can find on the Web (not just in mathematics but other subjects as well) are not very good, but that does not prevent them being very effective. As a professional mathematician and mathematics educator, I cringe when I watch a Khan Academy video, but millions find them of personal value. Analogously, in a domain where I am not an expert, bicycle mechanics, I watch Web videos to learn how to repair or tune my (high end) bicycles, and to assemble and disassemble my travel bike (a fairly complex process that literally has potential life and death consequences for me), and they serve my need, though I suspect a good bike mechanic would find much to critique in them. In both cases, mathematics and bicycle mechanics, some sites will iterate and improve, and in time they will dominate.

That last point, by the way, is another where many commentators miss the point. Something else that digital technologies and the Web make possible is rapid iteration guided by huge amounts of user feedback data – data obtained with great ease in almost real time.

In the days when products took a long time, and often considerable money, to plan and create, careful planning was essential. Today, we can proceed by a cycle of rapid prototypes. To be sure, it would be (in my view) unwise and unethical to proceed that way if a MOOC were being offered for payment or for some form of college credit, but for a cost-free, non-credit MOOC, learning on a platform that is itself under development, where the course designer is learning how to do it, can be in many ways a better learning experience than taking a polished product that has stood the test of time.

You don’t believe me? Consider this. Textbooks have been in regular use for over two thousand years, and millions of dollars have been poured into their development and production. Yet, take a look at practically any college textbook and ask yourself is you could, or would like to, learn from that source. In a system where the base level is the current college textbook and the bog-standard course built on it, the bar you have to reach with a MOOC to call it an improvement on the status quo is low indeed.

Again, Khan Academy provides the most dramatic illustration. Compared with what you will find in a good math classroom with a well trained teacher, it’s not good. But it’s a lot better than what is available to millions of students. More to the point, I know for a fact that Sal Khan is working on iterating from the starting point that caught Bill Gates’ attention, and has been for some time. Will he succeed? It hardly matters. (Well, I guess it does to Sal and his employees!) Someone will. (At least for a while, until someone else comes along and innovates a crucial step further.)

This, as I see it, is what, in general terms, is going on with MOOCs right now. We are experimenting. Needless to say – at least, it should be needless but there are worrying developments to the contrary – it would be unwise for any individual, any educational institution, or any educational district to make MOOCs (as courses) an important component of university education at this very early stage in their development. (And foolish to the point of criminality to take them into the K-12 system, but that’s a whole separate can of worms.)

Experimentation and rapid prototyping are fine in their place, but only when we all have more experience with them and have hard evidence of their efficacy (assuming they have such), should we start to think about giving them any critical significance in an educational system which (when executed properly) has served humankind well for several hundred years. Anyone who claims otherwise is probably trying to sell you something.

A final remark. I’m not saying that massive open online courses will go away. Indeed, I plan to continue offering mine – as a course – and I expect and hope many students will continue to take it as a complete course. I also expect that higher education institutions will increasingly incorporate MOOCs into their overall offerings, possibly for credit. (Stanford Online High School already offers a for-certificate course built around my MOOC.) So my use of the word “die” in the title involved a bit of poetic license.

But I believe my title is correct in its overall message. We already know from the research we’ve done at Stanford that only a minority of people enroll for a MOOC with the intention of taking it through to completion. (Though that “minority” can comprise several thousand students!) Most MOOC students already approach it as a resource, not a course! With an open online educational entity, it is the entire community of users that ultimately determines what it primarily is and how it fits in the overall educational landscape. According to the evidence, they already have, thereby giving us a new (and more accurate) MOOC mantra: resources, not courses. (Even when they are courses and when some people take them as such.)

In the coming posts to this blog, I’ll report on the changes I made in the second version of my MOOC, reflect on how things turned out, and speculate about the changes I am thinking of making in version 3, which is scheduled to start in September. First topic up will be peer evaluation – something that I regard as key to the success of a MOOC on mathematical thinking.

Those of us in education are fortunate to be living in a time where there is so much potential for change. The last time anything happened on this scale in the world of education was the invention of the printing press in the Fifteenth Century. As you can probably tell, I am having a blast.

To be continued …

Overcoming the legacy of prior education

Published March 20, 2013 Free online courses 25 Comments
Tags: dismal education, education design, higher education, K-12 education, mathematical thinking, MOOC, online education, textbooks

A real-time chronicle of a seasoned professor who is giving his second massively open online course.

We’re now into the third week of the course. The numbers are down on the first edition, almost certainly because the six months that have passed have seen the appearance of hundreds of other MOOCs students have to choose from. But the numbers are still huge. As of today:

Total registration: 27,014

Active students last week: 9,608

Total number of streaming views of lectures: 120,925

Total number of lecture downloads: 35,888

Number of unique videos watched: 87,155

Number of students submitting homework assignments: 5,552

Based on what we (my TA, Paul, and I) learned when I gave the course the first time last fall, I made some changes this time round. Paul and I discussed those changes in a video-recorded discussion we had with media host Angie Coiro just before edition 2 launched, that I referred to in my last blog.

Although the overall numbers are down by about 60%, the profile of the class activity is very similar. The most obvious one, the huge drop in numbers from the total number of enrollments to the number who are still active in week three, has been discussed ad infinitum, often being referred to as “a big problem with MOOCs.” As I observed in a recent blog in the Huffington Post, I don’t think there is a problem at all. The drop off is just a feature of what is a very new form of human experience. Old metrics are simply not appropriate, “retention rate” being one such. (Unless you pay attention to the base for the retention computation, in which case MOOC “retention” is not that different from retention in traditional college education.)

Some of the early research into MOOC participants that has been carried out by my colleagues at Stanford (including studies of my first MOOC) has already demonstrated what we suspected about why so many drop out of MOOCs: many people who register for a MOOC never have any intention of completing the course, or even getting beyond sampling one or two lectures and perhaps attempting one or two of the assignments. Some are motivated by pure curiosity into this new phenomenon, others just want to get a flavor of a particular discipline or topic, and doubtless others have different reasons.

For example, one reason some students enroll that I had not anticipated, reflects the fact that a MOOC offers a large number of eyeballs to be accessed. A very small number of students enrolled for my course in order to advertise products. (At least, that was one reason they enrolled; they may also have wanted to learn how to think mathematically!) In the long run, this may or may not turn out to be a positive thing. Certainly, the products advertised in the discussion forums for my course (at least the ones I saw) were all education related and free. (Moreover, I also included my own course-related textbook in my short list of suggested – but not required – resources.)

Still, the very wide reach of MOOCs means we are likely to see new kinds of activities emerge, some of them purely commercial. The example I cite above, though right now a very isolated one, may be a sign of big things to come – which is why I mention it. There is, after all, a familiar pattern. The Internet, on which MOOCs live, began as a military and educational network, but now it is a major economic platform. And textbooks grew from being a valuable educational support to the present-day mega-profit industry that has effectively killed US K-12 education.

Talking of which (and this brings me to my main focus in this post), the death – or at least the dearth – of good K-12 mathematics education becomes clear when you look through the forum posts in a MOOC such as mine, which assumes only high school knowledge of mathematics.

To be sure, generalizing is always dangerous, particularly so when based on comments in an online forum, which always attracts people with something to complain about. (Case in point: See my Twitter feed when it comes to banks, United Airlines, and bigoted politicians.) But with that caveat in mind, some themes become clear.

First, many forum posters seem to view education as something done to them, by other people who are in control. This is completely wrong, and is the opposite of what you will find in a good university (and a very small number of excellent K-12 schools). “To learn” is an active verb. The focus should be creating an environment where the student can learn, wants to learn, and can obtain the support required to do so. There is no other way, and anyone who claims to do anything more than help you to learn is trying to extract money from you.

Second, there is a common view of education as being primarily about getting grades on tests – generally by the most efficient means (which usually means by-passing real learning). In education, tests are metrics to help the student and the instructor gauge progress. That does not prevent tests being used to assess achievement and provide credentials, but that is something you do after an educational experience is completed. Their use within the learning process is different, and everyone involved in education – students, instructors, parents, bureaucrats, and politicians – needs to be aware of the distinction.

Even worse, is the belief that a test grade of less than 90% is an indication of failure, often compounded by the hopeless misconception that activities like mathematics depend mostly on innate talent, rather than the hours of effort that those of us in the business know is the key. (Check out Carol Dweck’s Mindset research or read Malcolm Gladwell’s book Blink. Better still, read both.)

This is compounded by the expectation that a grade of 90% is possible within just a few days of meeting something new. For example, here is one (slightly edited) forum post from a student in my class:

Right now I want to quit this class. I don’t understand ANY of it. Hell I don’t understand anything regarding to math except basic equations and those barely. When asked to give a theorem on why something (let’s say a right angle) is that way my answer always was “it is because it is”). So now I don’t know what to do. I got 14 out of 40 … 14, and the perfectionist in me is saying might as well give up … you gave it a shot … there is no way to catch up now. The person in me who wants to learn is saying to keep trying you never know what will happen. And the pessimist in me says it doesn’t matter – I dumb and will always be dumb and by continuing I am just showing how dumb I am.

In this case, I looked at other posts from this student and as far as I can tell (this is hard when done remotely over the Internet) she is smart and shows every indication she can do fine in mathematics. In which case, I take her comment as an indication of the total, dismal failure of the education system she has hitherto been subjected to. No first-line education system should ever produce a graduate who feels like that.

Certainly, in learning something new and challenging, getting over 30% in the first test, less than a week after meeting it for the first time, is good. In fact, if you are in a course where you get much more than that so quickly, you are clearly in the wrong course – unless you signed up in order to fine-tune something you had already learned. Learning is a long, hard process that involves repeated “failure”. And (to repeat a point I made earlier) anyone who says otherwise is trying to extract money from you.

Turning to the third theme emerging on the course forums, there is a perception that the most efficient way to learn is to break everything down into the smallest possible morsels. While an important component of learning – if the breaking down is done by, and not for, the student – it is just the first part of a two-part process. The second part, which is by far the most important, and is in fact where the actual learning takes place, is putting it back together into a coherent whole. Textbooks and YouTube videos can provide morselized edubits (I just made that word up), and they do so by the bucketload. What they cannot do, is deliver real learning.

Suitably designed, I see no reason why MOOCs cannot be made to provide good learning, at least up to sophomore college level in many, if not most, disciplines. But a key to doing that is to leverage the power, not of machines, but of people. For fairly well understood evolutionary reasons, human learning is a social activity. We learn best from and with other people. That is how we are built!

Part of the benefit from learning in a social context is that it can offer the learner not just feedback, but also the – at a fundamental level, more important – human support that people need to succeed in education. You can find both of these in a MOOC. Within a short time of the student above posting her feelings, another student responded with this:

Hi. Don’t be discouraged. This course will give you the opportunity to think in a different way. I took the course last year and struggled with most of it. I am taking the course again as I find the subject of mathematical thinking fascinating. My scores this time round are better than the last time which indicates that given enough time even the most mathematically challenged can improve! Only have one caveat for you. If you don’t enjoy the struggle in trying to comprehend and feel that it is not worth the effort then maybe this course is not for you.

With that comment we can see one huge benefit of MOOCs. (At least, all the time they are free.) You can take them as many times as you need or want.

The one essential ingredient in order to take advantage of the huge opportunity MOOCs offer, is knowing how to learn. That should be the main ability graduates of the K-12 system get from their education. Unfortunately, with the current US (and elsewhere) system built around “being taught” and “being tested,” only a few students emerge with that crucial ability, and the ones who do usually say it is in spite of their school education.

The problem, by the way, is not the teachers. Certainly, most of the ones I meet agree with me, and are very clear as to what the problem is: a system that simply does not give them the freedom and support that is necessary for them to really help students learn. (See Jo Boaler’s excellent, well researched book What’s Math Got To Do With It? for a distressing account of how the current, overly micro-regulated system fails our students in the case of mathematics.)

Okay, that’s enough ranting for one post. Let me finish with a couple of examples where MOOCs are already working well. One student in my MOOC posted the following comment:

I have taken this course on a whim to get myself back in gear to return to school in the fall. I always despised the math classes that I was forced to attend in high school and early college. I was frustrated with the endless formulas and cookie cutter style problem solving. If you can solve one you can solve them all so being forced to endlessly solve these equations and proofs over and over seemed to be a futile act of nonsense.

Heading into week three three of this class, my mind has been completely changed. I not only enjoy this more logic based math, but have, in the course of some personal reading and problem solving, discovered i have a knack for it. I have found the challenge of solving more and more difficult problems from a few books i have purchased much more gratifying and interesting than any other area of previous study.

I would like you know that I now plan to switch majors to mathematics. I would like to thank you and your team for an eye-opening experience.

Oh, all right, I admit that included more ranting about US K-12 education. But, heavens, it is bad, and it is likely to remain so all the time that real, knowledgable educators are not part of the conversation, with all the important decision being made by people whose primary interests are profits or political career advancement. (BTW, I have nothing against the profit motive. Heavens, I have two for profit companies of my own and am talking with colleagues about launching a third. But financial ROI is not the same as educational ROI – and again, anyone claiming otherwise, as one head of a major textbook publisher did not long ago, is motivated by the former. I do have something against many politicians, but then I am an American citizen, so after what we have experienced in the past four years, I would.*)

Here’s the other example, this one sent to me in an email, rather than posted on the course discussion forum.

I am enrolled in your course “Introduction to Mathematical Thinking.” It is incredible. You have alleviated my fears that my college professors will have the same attitude towards mathematics that my high school teachers do. Mathematics is beautiful and certainly emotional. I am surrounded at school by people who believe mathematics is systematic. Through all of the videos you have posted so far and your archived NPR clips, I am now confident that mathematics is the direction I want to pursue. I am excitedly awaiting next week’s lectures.

With tears in my eyes and more gratitude than I know how to express,

It’s that kind of feedback that makes teaching one of the most rewarding professions in the world. It’s why people become teachers. If society would just get off teachers’ backs and let them get on with what they were trained to do, what they know how to do, and what they want to do, we’d all be a lot better off. (Check out Finland.)

To be continued …

*ADDED LATER IN RESPONSE TO A QUERY FROM AN OVERSEAS READER: The problem is the complete refusal of the Republican Party to cooperate with a now twice-elected President of the US, in governing the country as they are all elected and paid from public funds to do, choosing instead to drive the country, and with it most of the world, to the brink of financial and thence social disaster.

Here we go again

Published February 20, 2013 Free online courses 17 Comments
Tags: Coursera, mathematical thinking, MOOC, online education, Stanford MOOC

A real-time chronicle of a seasoned professor who is about to give his second massively open online course.

The second offering of my MOOC Introduction to Mathematical Thinking begins on March 4 on Coursera. (The site actually opens on March 2, so students can familiarize themselves with its structure and start to make contact with other students before the first lecture.) So far, 13,000 students have registered. Last time I got 65,000, but back then there was the novelty factor. I’m expecting about 35,000 this time round.

For a quick overview of my current thoughts on MOOCs, see this 13 minute TV interview I did at Tallinn University of Technology in Estonia last November. (As the home of Skype, global-tech-hub Tallinn is particularly interested in MOOCs, of course.)

It’s been almost four months since my first foray into the chaotic new world of MOOCs came to an end, and ten weeks since I posted my last entry on this blog. I have decided that giving a MOOC falls into the same category as running a marathon (I’ve done maybe two dozen), completing the Death Ride (three), and – I am told – having a baby (I played a decidedly minor role in two). At the time you wonder why you are putting yourself through such stress, and that feeling continues for a while after the event is over. But then the strain of it all fades and you are left with feelings of pleasure, accomplishment, and satisfaction. And with that comes the desire to do it all again – better in the case of running, cycling, and MOOCing.

Coursera, we have a problem

It’s important to remember that genuinely massive MOOCs are a mere eighteen months old, and each one is very much a startup operation — as are the various platform providers such as Udacity, edX, Coursera, Venture Labs. and Class2Go (all except edX coming out of Global Startup Central, i.e., Stanford). One of the features of any startup operation is that there will be plenty of missteps along the way. Given the complexity of designing and delivering a university course in real time to tens of thousands of students around the world, it’s amazing that to date there have been just two missteps. The first, when the instructor had to pull the plug on a MOOC on designing online courses (yes, a particularly poignant topic as it turned out) and then more recently when the instructor pulled out, leaving the course to be run by the support staff.

Notice that I did not refer to either as a “failure.” Anyone who views such outcomes as failures has clearly never tried to do anything new and challenging, where you have to make up some of the rules as you go on. We are less than two years into this whole MOOC thing, so it’s worth reminding ourselves what it took (VIDEO) the USA to put a man on the Moon and bring him back alive, and to go on and build the Space Shuttle. The pedagogic fundamental that we gain confidence from our successes but learn from our mistakes, is as true for MOOC platform builders and MOOC instructors as it is for MOOC students.

Fortunately, I survived my first test flight relatively unscathed. I may not be so lucky second time round. I’ve made some changes that are intended to make the course better, but won’t know if they do until the course is underway.

Perhaps the most obvious change is to stretch the course from seven weeks (five weeks of lectures followed by two weeks of final exam work) to ten (8 + 2). Many students in my first course told me that the “standard university pace” with which I covered the curriculum was simply too much for online students who were fitting the course around busy professional and family schedules. I doubt that change will have any negative consequences.

More uncertain in their outcome are the changes I have made to the peer review process, that forms a major component of the course for students who are taking it for a Certificate of Completion (particularly Completion with Distinction).

Give credit where credit is due? Maybe

Talking of which, the issue of credentialing continues to generate a lot of discussion. My course does not offer College Credit (and it is not clear any Stanford MOOC ever will), but just recently, the American Council on Education’s College Credit Recommendation Service (ACE CREDIT) has evaluated and recommended college credit be given for five MOOCs currently offered (by other universities) on Coursera. (Starting this March, it will be possible to take an enhanced version of my MOOC given by Stanford Online High School, for which a credential is awarded, but that course, aimed at high flying high school juniors and seniors, has a restricted enrollment and carries a fee, so it is not a MOOC, rather a course with tutors and assessment, built around my MOOC.)

But I digress. As I observed on a number of occasions in this blog and my MAA blog Devlin’s Angle, I see group work and peer evaluation as the key to making quality mathematics education available in a MOOC. So students who took the first version of my course and are planning on enrolling again (and I know many are) will see some changes there. Not huge ones. Like NASA’s first fumbling steps into space, I think it is prudent to make small changes that have a good chance of being for the better. But I learned a lot from my first trip into MOOC-space, and I expect to learn more, and make further changes, on my second flight.

Finally, if you want to learn more about my reflections on my first MOOC and MOOCs in general, and have a two hour car drive during which you would find listening to a podcast about MOOCs marginally better than searching through an endless cycle of crackly Country and Western radio stations, download the two podcast files from Wild About Math, where host Sol Lederman grills me about MOOCs.

MOOC reflections – and Coursera’s Business Model

Published December 5, 2012 Free online courses 6 Comments
Tags: Coursera, educational tsunami, higher education, Huffington Post, MAA, MOOC, online education

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

Two of my most recent reflections on MOOCs were in many ways reflections about mathematics education in general, so instead of burying them here, where only the MOOC-curious would see them, I submitted them to the Mathematical Association of America as articles in my monthly series “Devlin’s Angle”, and that’s where you will find them.

The first, titled MOOC lessons, focused on the kind of learning that can take place in a MOOC.

In the second, The Darwinization of Higher Education, I looked at the likely effect of MOOCs on the higher education landscape. I originally submitted a shorter, “less-personal-blog” version as a post to my blog in The Huffington Post (Education Section), but much to my surprise, after sitting on it for over a week, they rejected it. Either I am way ahead of the HuffPost’s education editors or else they think I’m off my rocker (maybe both). It probably reflects on the massive uncertainty about where MOOCs are going, that I think there is a 5% chance I am off my rocker on this issue. A week ago I would have put that figure at 10%, but just this morning (a scant week after I had submitted my thoughts to HuffPost, and a mere 24 hours after I had sent the piece instead to the MAA) I received an email Coursera sent out to all past and present students.

It began thus:

Career Services: Finding great job opportunities

Coursera has begun Career Services with the goal of helping Coursera students find great jobs! Meeting great companies just got easier. Just go to <Coursera web page URL> and fill in your profile to opt-in to the service. After you opt-in, we will share your resume and other information you provide with selected partner companies who will introduce themselves if there’s a match.
We’re excited to connect you with great companies and new opportunities! Complete your profile here!

Remember, you read it first here. (As they say.)

Is this really Coursera’s business model (in the sense of the business model)? I am in no position to know. I suspect they don’t yet know either, how (and maybe if) they, or any other MOOC platform, will eventually make sufficient revenue to sustain their activities. Calling it “Coursera’s business model” in my title indicates only that it is a business model that the company has now announced. From my second article listed above, you will gather I think it is a smart move on their part. On the other hand, I can think of at least half a dozen others ways to monetize MOOCs, and at least as many ways for others to build businesses around the MOOC phenomenon.

I know many of my academic colleagues feel uneasy when education is discussed as a for-profit enterprise, but it has never been anything else. Someone has to pay. Usually, it is the student or the student’s family, either directly or indirectly. The novel aspect of the Ivy League MOOCs that I hope those colleagues see as positive is that the one person who does not pay is the student — at least on entry, which means that MOOC education is entirely free.

While on the topic of MOOC upsides, I had lunch recently with three of my fellow pioneer MOOC-instructors, and one substantial student demographic we all noticed was moms with young children (in many cases single moms, without the means to afford child care while they study). Hard to fault that.

We are entering a very different world in terms of access to higher education.

To be continued …

Coming up for air (and spouting off)

Published November 26, 2012 Free online courses 11 Comments
Tags: Coursera, education design, edX, Facebook, Harvard University, higher education, Khan Academy, Larry Cuban, mathematical thinking, MIT, MOOC, online education, Sebastian Thrun, social media, Stanford Graduate School of Education, Stanford MOOC, Udacity

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

Almost a month has passed since I last posted to this blog. Keeping my MOOC running took up so much time that, once it was over, I was faced with a huge backlog of other tasks to complete. Taking a good look at the mass of data from the course is just one of several post-MOOC activities that will have to wait until the New Year. So readers looking for statistics, analyses, and conclusions about my MOOC will, I am afraid, have to wait a little bit longer. Like most others giving these early MOOCs, we are doing so on the top of our existing duties; the time involved has yet to be figured into university workloads.

One issue that came up recently was when I put on my “NPR Math Guy” hat and talked with Weekend Edition host Scott Simon about my MOOC experience.

In the interview, I remarked that MOOCs owed more to Facebook than to YouTube. This observation has been questioned by some people, who believe Kahn Academy’s use of YouTube was the major inspiration. In making this comment, they are echoing the statement made by former Stanford Computer Science professor Sebastian Thrun when he announced the formation of Udacity.

In fact, I made my comment to Scott with my own MOOC (and many like it) in mind. Though I have noted in earlier posts to this blog how I studied Sal Khan’s approach in designing my own, having now completed my first MOOC, I am now even more convinced than previously that the eventual (we hope) success of MOOCs will be a consequence of Facebook (or social media in general) rather than of Internet video streaming.

The reason why I felt sure this would be the case is that (in most disciplines) the key to real learning has always been bi-directional human-human interaction (even better in some cases, multi-directional, multi-person interaction), not unidirectional instruction.

What got the entire discussion about MOOCs off in the wrong direction – and with it the public perception of what they are – is the circumstance of their birth, or more accurately, of their hugely accelerated growth when a couple of American Ivy League universities (one of them mine) got in on the act.

But it’s important to note that the first major-league MOOCs all came out of Stanford’s Computer Science Department, as did the two spinoff MOOC platforms, Udacity and Coursera. When MIT teamed up with Harvard to launch their edX platform a few months later, it too came from their Computer Science Department.

And there’s the rub. Computer Science is an atypical case when it comes to online learning. Although many aspects of computer science involve qualitative judgments and conceptual reasoning, the core parts of the subject are highly procedural, and lend themselves to instruction-based learning and to machine evaluation and grading. (“Is that piece of code correct?” Let the computer run it and see if it performs as intended.)

Instructional courses that teach students how to carry out various procedures, which can be assessed to a large degree by automatic grading (often multiple choice questions) are the low hanging fruit for online education. But what about the Humanities, the Arts, and much of Science, where instruction is only a small part of the learning process, and a decidedly unimportant part at that, and where machine assessment of student work is at best a goal in the far distant future, if indeed it is achievable at all?

In the case of my MOOC, “Introduction to Mathematical Thinking,” the focus was the creative/analytic mathematical thinking process and the notion of proof. But you can’t learn how to think a certain way or how prove something by being told or shown how to do it any more than you can learn how to ride a bike by being told or shown. You have to try for yourself, and keep trying, and falling, until it finally clicks. Moreover, apart from some very special, and atypical, simple cases, neither thinking nor proofs can be machine graded. Proofs are more like essays than calculations. Indeed, one of the things I told my students in my MOOC was that a good proof is a story, that explains why something is the case.

For the vast majority of students, discussion with (and getting feedback from) professors, TAs, and other students struggling to acquire problem solving ability and master abstract concepts and proofs, is an essential part of learning. For those purposes, the online version does not find its inspiration in Khan Academy as it did for Thrun, but in Facebook, which showed how social interaction could live on the Internet.

When the online version of Thrun’s Stanford AI class attracted 160,000 students, he did not start a potential revolution in global higher education, but two revolutions, only the first of which he was directly involved in. The first one is relatively easy to recognize and understand, especially for Americans, who for the most part have never experienced anything other than instruction-based education.

For courses where the goal is for the student to achieve mastery of a set of procedures (which is true of many courses in computer science and in mathematics), MOOCs almost certainly will change the face of higher education. Existing institutions that provide little more than basic, how-to instruction have a great deal to fear from MOOCs. They will have to adapt (and there is a clear way to do so) or go out of business.

If I want to learn about AI, I would prefer to do so from an expert such as Sebastian Thrun. (In fact, when I have time, I plan on taking his Udacity course on the subject!) So too will most students. Why pay money to attend a local college and be taught by a (hopefully competent) instructor of less stature when you can learn from Thrun for free?

True, Computer Science courses are not just about mastery of procedures. There is a lot to be learned from the emphases and nuances provided by a true expert, and that’s why, finances aside, I would choose Thrun’s course. But at the end of the day, it’s the procedural mastery that is the main goal. And that’s why that first collection of Computer Science MOOCs has created the popular public image of the MOOC student as someone watching canned instructional videos (generally of short duration and broken up by quizzes), typing in answers to questions to be evaluated by the system.

But this kind of course occupies the space in the overall educational landscape that McDonalds does in the restaurant business. (As someone who makes regular use of fast food restaurants, this is most emphatically not intended as a denigratory observation. But seeing utility and value in fast food does not mean I confuse a Big Mac with quality nutrition.)

Things are very, very different in the Humanities, Arts, and most of Science (and some parts of Computer Science), including all of mathematics beyond basic skills mastery – something that many people erroneously think is an essential prerequisite for learning how to do math, all evidence from people who really do learn how to do math to the contrary.

[Ask the expert. We don’t master the basic skills; we don’t need them because, early on in our mathematic learning, we acquired one – yes, just one – fundamental ability: mathematical thinking. That’s why the one or two kids in the class who seem to find math easy seem so different. In general, they don’t find math easy, but they are doing something very different from everyone else. Not because they are born with a “math gene”. Rather, instead of wasting their time mastering basic skills, they spent that time learning how to think a certain way. It’s just a matter of how you devote your learning time. It doesn’t help matters that some people managed to become qualified math teachers and professors seemingly without figuring out that far more efficient path, and hence add their own voice to those who keep calling for “more emphasis on basic skills” as being an essential prerequisite to mathematical power.]

But I digress. To get back to my point, while the popular image of a MOOC centers on lecture-videos and multiple-choice quizzes, what Humanities, Arts, and Science MOOCs (including mine) are about is community building and social interaction. For the instructor (and the very word “instructor” is hopelessly off target in this context), the goal in such a course is to create a learning community. To create an online experience in which thousands of self-motivated individuals from around the world can come together for a predetermined period of intense, human–human interaction, focused on a clearly stated common goal.

We know that this can be done at scale, without the requirement that the participants are physically co-located or even that they know one another. NASA used this approach to put a man on the moon. MMOs (massively multiplayer online games – from which acronym MOOCs got their name) showed that the system works when the shared goal is success in a fantasy game world.

Whether the same approach works for higher education remains an open question. And, for those of us in higher education, what a question! A question that, in my case at least, has proved irresistible.

This, then, is the second MOOC revolution. The social MOOC. It’s outcome is far less evident than the first.

The evidence I have gathered from my first attempt at one of these second kinds of MOOC is encouraging, or at least, I find it so. But there is a long way to go to make my course work in a fashion that even begins to approach what can be achieved in a traditional classroom.

I’ll pursue these thoughts in future posts to this blog — and in future versions of my Mathematical Thinking MOOC, of which I hope to offer two variants in 2013.

Meanwhile, let me direct you to a recent article that speaks to some of the issues I raised above. It is from my legendary colleague in Stanford’s Graduate School of Education, Larry Cuban, where he expresses his skepticism that MOOCs will prove to be an acceptable replacement for much of higher education.

To be continued …

Peer grading: inventing the light bulb

Published October 31, 2012 Free online courses 29 Comments
Tags: calibrated peer review, Coursera, Edison, Facebook, mathematical thinking, MIT, MOOC, online education, social media, Stanford MOOC

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

With the deadline for submitting the final exam in my MOOC having now passed, the students are engaging in the Peer Evaluation process. I know of just two cases where this has been tried in a genuine MOOC (where the M means what it says), one in Computer Science, the other in Humanities, and both encountered enormous difficulties, and as a result a lot of student frustration. My case was no different.

Anticipating problems, I had given the class a much simplified version of the process – with no grade points at stake – at the end of Week 4, so they could familiarize themselves with the process and the platform mechanics before they had to do it for real. That might have helped, but the real difficulties only emerged when 1,520 exam scripts started to make their way through the system.

By then the instructional part of the course was over. The class had seen and worked through all the material in the curriculum, and had completed five machine-graded problem sets. Consequently, there were enough data in the system to award certificates fairly if we had to abandon the peer evaluation process as a grading device, as happened for that humanities MOOC I mentioned, where the professor decided on the fly to make that part of the exam optional. So I was able to sleep at night. But only just.

With over 1,000 of the students now engaged in the peer review process, and three days left to the deadline for completing grading, I am inclined to see the whole thing through to the (bitter) end. We need the data that this first trial will produce so we can figure out how to make it work better next time.

Long before the course launched, I felt sure that there were two things we would need to accomplish, and accomplish well, in order to make a (conceptual, proof-oriented) advanced math MOOC work: the establishment (and data gathering from) small study groups in which students could help one another, and the provision of a crowd-sourced evaluation and grading system.

When I put my course together, the Coursera platform supported neither. They were working on a calibrated peer review module, but implementing the group interaction side was still in the future. (The user-base growth of Coursera has been so phenomenal, it’s a wonder they can keep the system running at all!)

Thus, when my course launched, there was no grouping system, nor indeed any social media functionality other than the common discussion forums. So the students had to form their own groups using whatever media they could: Facebook, Skype, Google Groups, Google Docs, or even the local pub, bar, or coffee shop for co-located groups. Those probably worked out fine, but since they were outside our platform, we had no way to monitor the activity – an essential functionality if we are to turn this initial, experimental phase of MOOCs into something robust and useful in the long term.

Coursera had built a beta-release, peer evaluation system for a course on Human Computer Interaction, given by a Stanford colleague of mine. But his needs were different from mine, so the platform module needed more work – more work than there was really time for! In my last post, I described some of the things I had to cope with to get my exam up and running. (To be honest, I like the atmosphere of working in startup mode, but even in Silicon Valley there are still only 24 hours in a day.)

It’s important to remember that the first wave of MOOCs in the current, explosive, growth period all came out of computer science departments, first at Stanford, then at MIT. But CS is an atypical case when it comes to online learning. Although many aspects of computer science involve qualitative judgments and conceptual reasoning, the core parts of the subject are highly procedural, and lend themselves to instruction-based learning and to machine evaluation and grading. (“Is that piece of code correct?” Just see if it runs as intended.)

The core notion in university level mathematics, however, is the proof. But you can’t learn how to prove something by being told or shown how to do it any more than you can learn how to ride a bike by being told or shown. You have to try for yourself, and keep trying, and falling, until it finally clicks. Moreover, apart from some very special, and atypical, simple cases, proofs cannot be machine graded. In that regard, they are more like essays than calculations. Indeed, one of the things I told my students was that a good proof is a story, that explains why something is the case.

Feedback from others struggling to master abstract concepts and proofs can help enormously. Study groups can provide that, along with the psychological stimulus of knowing that others are having just as much difficulty as you are. Since companies like Facebook have shown us how to build platforms that support the creation of groups, that part can be provided online. And when Coursera is able to devote resources to doing it, I know it will work just fine. (If they want to, they can simply hire some engineers from Facebook, which is little more than a mile away. I gather that, like Google before it, the fun period there has long since passed and fully vested employees are looking to move.)

The other issue, that of evaluation and grading, is more tricky. The traditional solution is for the professor to evaluate and grade the class, perhaps assisted by one or more TAs (Teaching Assistants). But for classes that number in the tens of thousands, that is clearly out of the question. Though it’s tempting to dream about building a Wikipedia-like community of dedicated, math-PhD-bearing volunteers, who will participate in a mathematical MOOC whenever it is offered – indeed I do dream about it – it would take time to build up such a community, and what’s more, it’s hard to see there being enough qualified volunteers to handle the many different math MOOCs that will soon be offered by different instructors. (In contrast, there is just one Wikipedia, of course.)

That leaves just one solution: peer grading, where all the students in the class, or at least a significant portion thereof, are given the task of grading the work of their peers. In other words, we have to make this work. And to do that, we have to take the first step. I just did.

Knowing just how many unknowns we were dealing with, my expectations were not high, and I tried to prepare the students for what could well turn out to be chaos. (It did.) The website description of the exam grading system was littered with my cautions and references to being “live beta”. On October 15, when the test run without the grading part was about to launch, I posted yet one more cautionary note on the main course announcements page:

… using the Calibrated Peer Review System for a course like this is, I believe, new. (It’s certainly new to me and my assistants!) So this is all very much experimental. Please approach it in that spirit!

Even so, many of the students were taken aback by just how clunky and buggy the thing was, and the forums sprung alive with exasperated flames. I took solace in the recent release of Apple Maps on the iPhone, which showed that even with the resources and expert personnel available to one of the world’s wealthiest companies, product launches can go badly wrong – and we were just one guy and two part-time, volunteer student assistants, working on a platform being built under us by a small startup company sustained on free Coke and stock options. (I’m guessing the part about the Coke and the options, but that is the prevalent Silicon Valley model.)

At which point, one of those oh-so-timely events occurred that are often described as “Acts of God.” Just when I worried that I was about to witness, and be responsible for starting, the first global, massive open online riot (MOOR) in a math class, Hurricane Sandy struck the Eastern Seaboard, reminding everyone that a clunky system for grading math exams is not the worst thing in the world. Calm, reasoned, steadying, constructive posts started to appear on the forum. I was getting my feedback after all. The world was a good place once again.

Failure (meaning things don’t go smoothly, or maybe don’t work at all) doesn’t bother me. If it did, I’d never have become a mathematician, a profession in which the failure rate in first attempts to solve a problem is somewhere north of 95%. The important thing is to get enough data to increase the chances of getting it right – or far more likely, just getting it better – the second time round. Give me enough feedback, and I count that “failure” as a success.

As Edison is said to have replied to a young reporter about his many failed attempts to construct a light bulb, “Why would I ever give up? I now know definitively over 9,000 ways that an electric light bulb will not work. Success is almost in my grasp.” (Edison supposedly failed a further 1,000 times before he got it right. Please don’t tell my students that. We are just at failure 1.)

If there were one piece of advice I’d give to anyone about to give their first MOOC, it’s this: remember Edison.

To be continued …

Answering the 64,000-Students Questions

Published October 26, 2012 Free online courses 14 Comments
Tags: Big Data, education design, mathematical thinking, MOOC, online education, Paul Lockhart, Stanford Graduate School of Education, Stanford MOOC, transition course

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

With the “instructional” part of the course finished and the remaining students working on the Final Exam (it will be peer graded next week), at last I can sit back and take a short breather. The next step will be to debrief and reflect with my two course assistants (both PhD students in the Stanford Graduate School of Education) and decide where to ride the MOOC beast next.

For sure I’ll offer another version of this course next year, with changes based on the huge amounts of data you get with a global online class of 64,000 students. Despite the enormous effort in designing, preparing, and running such a massive enterprise, there are three very good reasons to pursue this.

First, and this I believe is one of the main reasons why Stanford is supporting the development of MOOCs (I am not part of the central, policy-making administration), designing, running, and analyzing the learning outcomes of MOOCs is a tremendous research opportunity that will almost certainly result in new understandings of how people learn, and as a result very likely will enable the university to improve the learning experience of our regular on-campus students. After just five weeks, my two graduate assistants have enough data to write several dissertations, in addition to the one they need to get their doctorates.

Second, there is a huge, overall, feel-good factor for those of us involved, knowing that we can help to provide life-changing opportunities for people around the world who would otherwise have no access to quality higher education. Is what they get as good as being at Stanford? I very much doubt it, though the scientist in me says we should keep an open mind into the eventual outcomes of what is at present a very novel phenomenon. But if you compare a Stanford MOOC with the alternative of nothing at all, then already you have an excellent reason to continue.

Third, and this is something that anyone in education will acknowledge makes up for our earning a much lower salary than our (often less formally qualified) friends in the business and financial worlds, there is the pleasure of hearing first-hand from some of our more satisfied customers. The following is one of many appreciative emails and forum posts I have received as my course came to and end:

Mr. Devlin and all members of the Introduction To Mathematical Thinking team, I just wanted to say Thank You for everything that you have done to share your knowledge and giving your time and great effort to help others learn. I imagine that this is not an easy project to lead and sustain on a continuous basis. However, you have done a wonderful job in relaying your message. Through your efforts, you have helped many people in the process; especially me. Until this class, I hated math. I hated the idea of learning math or thinking in mathematically analogous methods that are applicable to real world situations. I just didn’t get it. I’m still a little confused about why I am able to comprehend your lessons as effectively as I am (which is saying a lot considering how much I hated math) when I have not been able to do so in the past. Now, I find myself looking forward to your classes everyday! I look forward to using what I have learned from the last video lectures or assignments and using those lessons in situations I did not think possible. And now, I love math! Your instruction has helped me to think more logically and to draw more concise conclusions with issues that I am trying to handle. This is indeed a skill. This is also a skill that you can build upon throughout your lifetime if one chooses to do so. Though I may not be at the level of learning that I should be at, I have learned more in the past three weeks than I have learned throughout my life; and I will continue to learn. I am very serious about this statement. So, thank you All. Thank you, Mr. Devlin. Great Job and Cheers!

Nice!

To be sure, there were trolls on the course discussion forum, for whom nothing we did was right. But one of the benefits of having tens of thousand of students is that within at most an hour of a flame post appearing, tens of others jumped on the offending individual, and within a short while all that was left was a “This comment has been deleted” notice. As the course wore on, the trolls simply dropped away.

Though there was the one individual who, in week four, posted a comment that he hated my teaching style and was learning nothing. Given that this was a free course that no one was under any compulsion to take, and for which no official credential was awarded, one wonders why this person stuck around for so long!

That example provided no more than an amusing anecdote to tell when I start to give talks on “What’s it like to teach 64,000 students?” (Invitations are already coming in.) But there is a somewhat closely related issue that I find far more significant.

Like almost all current MOOCs, there was no real credentialing in my course, so the focus was entirely on learning for its own sake. (As a lifelong math professor, used to teaching classes where many of the students were there because they needed to fulfill a mathematics requirement, having a class of students who were there purely voluntarily added appeal to my giving a MOOC.) To be sure, there were in-lecture quizzes, machine-graded assignments, and a peer evaluated final exam, but the only people who had access to any student’s results were myself, my two course assistants, and the student. Moreover, there was no official certification to back up a good result (the course offered two levels, Completion and Completion with Distinction), and turn it into a form of credential.

Yet many students had an ongoing obsession with their grades, and indeed pleaded with me from time to time to re-grade their work. (Clearly not possible in a 64,000 student MOOC. Besides, I never saw their work. How could I?) As a competitive person myself, I can appreciate the desire to do well. But with literally nothing at stake, I was at first surprised by the degree to which it bothered some of them. When I figured out what was probably going on, I found something that bothered me.

Unlike most MOOCs, mine, being at first-year university level, can be taken by high school students. Indeed, since my primary target audience comprised students entering or about to enter university to study mathematics or a math-related subject, I expected to get high school seniors, and designed my course as much as possible to accommodate them.

I’m guessing that the majority of students who were obsessed with grades were still at high school – indeed, most likely a US high school. That grade obsession I observed is, I suspect, simply a learned behavior that reflects the way our K-12 system turns the learning of a fascinating subject – one of humankind’s most amazing, creative, intellectual achievements – into a seemingly endless sequence of bite-sized pieces that are fed to the student in a mandated hamster-wheel.

No wonder they could not relax and enjoy learning for its own sake. Any natural curiosity and desire to learn – something all humans are born with – had been driven out of them by the very institution that is supposed to encourage and develop that trait. In its place was mere grade hunting.

Do I know this for a fact? No. That’s why I used those hedging words “guess” and “suspect”. But something has to explain that grade obsession in my course, and it certainly brought to mind Paul Lockhart’s wonderful essay A Mathematician’s Lament, which I had the privilege to bring to a wider audience some years ago.

But now I digress. Time to wrap up and check the dashboard on the course website see how many students have submitted the Final Exam so far.

Though this post has dropped the title “MOOC Planning”, I am going to keep posting here, as the project goes forward. Stay tuned.

To be continued …

It’s About Time (in Part): MOOC Planning – Part 10

Published October 22, 2012 Free online courses 6 Comments
Tags: Big Data, calibrated peer review, Coursera, education design, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, Unix Time, WYSIWOSG

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

Well, lectures have ended and the course has now switched gears. For those still left in the course (17% of the final enrollment total of 64,045), the next two weeks are focused on trying to make sense of everything they have learned, and working on the final exam — which in the case of my course involves peer evaluation.

Calibrated Peer Review is not new. A study of its use in the high school system by Sadler and Good, published in 2006, has become compulsory reading for those of us planning and giving MOOCs that cover material that cannot be machine graded. [If you want to see how I am using it, just enroll in the class and read the description of the “Peer Review system”. There is no obligation to do anything more than browse around the site! No one will know you are not simply a dog that can use a computer.]

As I was working on my course, Coursera was still frantically building out their platform to support peer evaluation. There was a lot of just-in-time construction. It’s been a long time since I’ve had to go behind a user-friendly interface and dig into the underlying code to do something on a computer, and the programming languages have all changed since I last did that.

One thing I had to learn was one of the ways networked computers keep time. I now know that at the time of writing these words, 7:00AM Pacific Daylight Time on October 22, 2012, exactly 1,350,914,400 seconds have elapsed since the first second of January 1st, 1970, Eastern Standard Time. That was the start of Unix Time.

I needed to learn to work in Unix Time in order to set the various opening times and completion deadlines for the exam process. I expect that by the time the next instructor puts together a MOOC, she or he will be greeted by a nice, friendly Coursera interface with pulldown menus and boxes to tick — which probably will come as a great relief to any humanities professors reading this, who don’t have any programming in their background.

[By coincidence, Unix was the last programming language I had any proficiency in, but I did not need to know Unix to use Unix Time. I just used an online converter. Unix was developed in 1969 at AT&T Bell Laboratories in New Jersey. Hence the 1970 EST baseline.]

In fact, time conversion issues in general turned out to be a continuing, major headache in a course with students all over the world. One thing we will not do again is have 12:00PM Stanford Time, aka Coursera Time (i.e., PDT), as any of the course deadlines. It might seem a nice clean stopping point, and there are all those memories of Gary Cooper’s deadline in the classic Western movie High Noon, but many students missed the deadline for the first submitted assignment because they thought 12:00PM meant midnight, which in some parts of the world made them a whole day late.

The arbitrary illogicality of the AM/PM distinction is not apparent to those of us who grew up with it. But my course TA and I are now very aware of the problems it can lead to! In future, we’ll stick to unambiguous times that stay away from noon and midnight. But even then, with local computer systems usually working on local time, to say nothing of the different Summer and Winter Times, which change on different dates around the world, timing events in MOOCs is going to remain a problematic issue, just as it is for international travelers and professionals who collaborate globally over Skype and other conferencing services. (When I used the Unix Time conversion app, I had to remember that Unix thinks New Jersey is currently just two hours ahead of California, not the three hours United Airlines uses when it flies me there. Confusing, isn’t it?)

The reason why times are an issue in my course is that it is a course. At first glance, it may look little different from Khan Academy, where there are no time issues at all. But Khan Academy is really just an educational resource. (At least, that’s the part most people are familiar with and use, namely the video library that started it all. People use it as a video version of a textbook — or more precisely a video equivalent to that good old standby Cliffs Notes, which got many of us through an exam in an obligatory subject we were not particularly interested in.)

In contrast, in my case, as I’ve discussed earlier in this blog series (in particular, Part 6), my goal was to take a standard university course (one I’ve given many times over the years, at different universities, including Stanford) and make it available to anyone in the world, for free. To the degree I could make it happen, they would get the same learning experience.

That meant that the main goal would be to build a (short-lived) learning community. The video-recorded lectures and tutorials were simply tools to make that happen and to orchestrate events. Real learning takes place when students work on assignments on their own, when they repeatedly fail to solve a problem, and when they interact (with the professor and with one another) — not when they watch a lecture or read a book.

To achieve that goal, the MOOC would, as I stated in Part 6, involve admissions, lectures, peer interaction, professor interaction, problem-solving, assignments, exams, deadlines, and certification. To use the mnemonic I coined early on in this series, the basic design principle is WYSIWOSG: What You See Is What Our Students Get.

As we go forward, I intend to iterate on the course design, based on the data we collect from the students (and 64,000 students very definitely puts us into the Big Data realm). But my basic principle will remain that of offering a course, not the provision of a video library. And the reason for that should be obvious to anyone who has been following this blog series, as well as some of the posts on my other blogs Devlin’s Angle and profkeithdevlin.org. The focus is not on acquiring facts or mastering basic skills, but on learning to think a certain way (in my case, like a professional mathematician). And that requires both a lot of effort and (for most of us) a lot of interaction with others trying to achieve the same goal.

Our ancestors in the 11th Century started to develop what to this day remains the best way we know to achieve this at scale: the university, where people become members of a learning community in which learning takes place in a hothouse atmosphere that involves periods of intense interaction as deadlines loom, sustained by the rapidly formed social bonds that emerge as a result of that same pressure.

While I will likely experiment with variants of this model that allow for participation by students who have demanding, full-time jobs, I doubt I will abandon that basic model. It has lasted for a thousand years for a good reason. It works.

To be continued …

Final Lecture: MOOC Planning – Part 9

Published October 20, 2012 Free online courses 8 Comments
Tags: calibrated peer review, Coursera, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, transition course, transition math, WYSIWOSG

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I gave my last lecture of the course yesterday (discounting the tutorial session that will go out next week), and we are now starting a two week exam period.

“Giving” a lecture means the video becomes available for streaming. For logistic reasons (high among them, my survival and continued sanity — assuming anyone who organizes and gives a MOOC, for no payment, is sane), I recorded all the lectures weeks ago, well before the course started. The weekly tutorial sessions come the closest to being live. I record them one or two days before posting, so I can use them to respond to issues raised in the online course discussion forum.

The initial course enrollment of 63,649 has dropped to 11,848 individuals that the platform says are still active on the site. At around 20%, that’s pretty high by current MOOC standards, though I don’t know whether that is something to be pleased about, since it’s not at all clear what the right definition of “success” is for a MOOC.

Some might argue that 20% completion indicates that the standards are too low. I don’t think that’s true for my course. Completion does, after all, simply mean that a student is still engaged. The degree to which they have mastered the material is unclear. So having 80% drop out could mean the standard is too high.

In my case, I did not set out to achieve any particular completion rate; rather I adopted a WYSIWOSG approach — “What You See Is What Our Students Get.” I offered a MOOC that is essentially the first half of a ten week course I’ve given at many universities over the years, including Stanford. That meant my students would experience a Stanford-level course. But they would not be subject to passing a Stanford-level exam.

In fact, I could not offer anything close to a Stanford-exam experience. There is a Final Exam, and it has some challenging questions, but it is not taken under controlled, supervised conditions. Moreover, since it involves constructing proofs, it cannot be machine graded, and thus has to be graded by other students, using a crowd sourcing method (Calibrated Peer Review). That put a significant limitation on the kinds of exam questions I could ask. On top of that, the grading is done by as many different people as there are students, and I assume most of them are not expert mathematicians. As a result, it’s at most a “better-than-nothing” solution. Would any of us want to be treated by a doctor whose final exam had been peer graded (only) by fellow students, even if the exam and the grading had been carried out under strictly controlled conditions?

On the other hand, looking at and attempting to evaluate the work of fellow students is a powerful learning experience, so if you view MOOCs as vehicles for learning, rather than a route to a qualification, then peer evaluation has a lot to be said for it. Traditional universities offer both learning and qualifications. MOOCs currently provide the former. Whether they eventually offer the latter as well remains to be seen. There are certainly ways it can be done, and that may be one way that MOOCs will make money. (Udacity already does offer a credentialing option, for a fee.)

In designing my course, I tried to optimize for learning in small groups, perhaps five to fifteen at a time. The goal was to build learning communities, within which students could help one another. Since there is no possibility of regular, direct interaction with the instructor (me) and my one TA (Paul), students have to seek help from fellow students. There is no other way. But, on its own, group work is not enough. Learning how to think mathematically (the focus of my course) requires feedback from others, but it needs to include feedback from people already expert in mathematical thinking. This means that, in order to truly succeed, not only do students need to work in groups (at least part of the time), and subject their attempts to the scrutiny of others, some of those interactions have to be with experts.

One original idea I had turned out not to work, though whether through the idea itself being flawed or the naive way we implemented it is not clear to me. That was to ask students at the start of the course to register if they had sufficient knowledge and experience with the course material to act as “Community TAs”, and be so designated in the discussion forums. Though over 600 signed up to play that role, many soon found they did not have sufficient knowledge to perform the task. Fortunately,a relatively small number of sign-ups did have the necessary background, as well as the interpersonal skills to give advice in a supporting, non-threatening way, and they more or less ensured that the forum discussions met the needs of many students (or so it seems).

Another idea was to assign students to study groups, and use an initial survey to try to identify those with some background knowledge and seed them into the groups. Unfortunately, Coursera does not (yet) have functionality to support the creation and running of groups, apart from the creation of forum threads. So instead, in my first lecture, I suggested to the students that they form their own study groups in whatever way they could.

The first place to do that was, of course, the discussion forums on the course website, which very soon listed several pages of groups. Some used the discussion forum itself to work together, while others migrated offsite to some other location, physical or virtual, with Skype seeming a common medium. Shortly after the course launched, several students discovered GetStudyRoom, a virtual meeting place dedicated to MOOCs, built by a small startup company.

In any event, students quickly found their own solutions. But with students forming groups in so many different ways on different media, there was no way to track how many remained active or how successful they have been.

The study groups listed on the course website show a wide variety of criteria used to bring the groups together. Nationality and location were popular, with groups such as Brazil Study Group, Grupo de Estudo Português, All Students From Asia, and Study Group for Students Located in Karachi, Pakistan. Then there were groups with a more specific focus, such as Musicians, Parents of Homeschooled Children, Older/Retired English Speakers Discussion for Assignment 1, and, two of my favorites, After 8pm (UK time) English speakers with a day job and the delightfully named Just Hanging on Study Group.

The forum has seen a lot of activity: 15,088 posts and 13,622 comments, spread across 2712 different threads, viewed 430,769 times. Though I have been monitoring the forums on an almost daily basis, to maintain an overall sense of how the course is going, it’s clearly not possible to view everything. For the most part I restricted my attention to the posts that garnered a number of up-votes. Students vote posts up and down, and once a post shows 5 or more up-votes, I take that as an indication that the issue may be worth looking at.

The thread with the highest number of up-votes (165) was titled Deadlines way too short. Clearly, the question of deadlines was a hot topic. How, if at all, to respond to such feedback is no easy matter. In a course with tens of thousands of students, even a post with hundreds of up-votes represents just a tiny fraction of the class. Moreover, threads typically include opinions on both sides of an issue.

For instance, in threads about the pace of the course, some students complained that they did not have enough time to complete assignments, and pleaded for more relaxed deadlines, whereas others said they thrived on the pace, which stimulated them to keep on top of the material. For many, an ivy-league MOOC offers the first opportunity to experience an elite university course, and I think some are surprised at the level and pace. (I fact, I did keep the pace down for the first three weeks, but I also do that when I give a transition course in a regular setting, since I know how difficult it is to make that transition from high school math to university level mathematics.)

A common suggestion/request was to simply post the course materials online and let students access them according to their own schedules, much like Khan Academy. This raises a lot of issues about the nature of learning and the role MOOCs can (might? should?) play. But this blog post has already gone on long enough, so I’ll take up that issue next time.

To be continued …

« Previous Page — Next Page »

MOOCtalk

Posts Tagged 'MOOC'

Evaluation rubrics: the good, the bad, and the ugly

The MOOC will soon die. Long live the MOOR

Overcoming the legacy of prior education

Here we go again

MOOC reflections – and Coursera’s Business Model

Answering the 64,000-Students Questions

It’s About Time (in Part): MOOC Planning – Part 10

Final Lecture: MOOC Planning – Part 9

profkeithdevlin

Devlin’s Angle (MAA)

Twitter Updates

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011

Posts Tagged 'MOOC'

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011