A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.
With the deadline for submitting the final exam in my MOOC having now passed, the students are engaging in the Peer Evaluation process. I know of just two cases where this has been tried in a genuine MOOC (where the M means what it says), one in Computer Science, the other in Humanities, and both encountered enormous difficulties, and as a result a lot of student frustration. My case was no different.
Anticipating problems, I had given the class a much simplified version of the process – with no grade points at stake – at the end of Week 4, so they could familiarize themselves with the process and the platform mechanics before they had to do it for real. That might have helped, but the real difficulties only emerged when 1,520 exam scripts started to make their way through the system.
By then the instructional part of the course was over. The class had seen and worked through all the material in the curriculum, and had completed five machine-graded problem sets. Consequently, there were enough data in the system to award certificates fairly if we had to abandon the peer evaluation process as a grading device, as happened for that humanities MOOC I mentioned, where the professor decided on the fly to make that part of the exam optional. So I was able to sleep at night. But only just.
With over 1,000 of the students now engaged in the peer review process, and three days left to the deadline for completing grading, I am inclined to see the whole thing through to the (bitter) end. We need the data that this first trial will produce so we can figure out how to make it work better next time.
Long before the course launched, I felt sure that there were two things we would need to accomplish, and accomplish well, in order to make a (conceptual, proof-oriented) advanced math MOOC work: the establishment (and data gathering from) small study groups in which students could help one another, and the provision of a crowd-sourced evaluation and grading system.
When I put my course together, the Coursera platform supported neither. They were working on a calibrated peer review module, but implementing the group interaction side was still in the future. (The user-base growth of Coursera has been so phenomenal, it’s a wonder they can keep the system running at all!)
Thus, when my course launched, there was no grouping system, nor indeed any social media functionality other than the common discussion forums. So the students had to form their own groups using whatever media they could: Facebook, Skype, Google Groups, Google Docs, or even the local pub, bar, or coffee shop for co-located groups. Those probably worked out fine, but since they were outside our platform, we had no way to monitor the activity – an essential functionality if we are to turn this initial, experimental phase of MOOCs into something robust and useful in the long term.
Coursera had built a beta-release, peer evaluation system for a course on Human Computer Interaction, given by a Stanford colleague of mine. But his needs were different from mine, so the platform module needed more work – more work than there was really time for! In my last post, I described some of the things I had to cope with to get my exam up and running. (To be honest, I like the atmosphere of working in startup mode, but even in Silicon Valley there are still only 24 hours in a day.)
It’s important to remember that the first wave of MOOCs in the current, explosive, growth period all came out of computer science departments, first at Stanford, then at MIT. But CS is an atypical case when it comes to online learning. Although many aspects of computer science involve qualitative judgments and conceptual reasoning, the core parts of the subject are highly procedural, and lend themselves to instruction-based learning and to machine evaluation and grading. (“Is that piece of code correct?” Just see if it runs as intended.)
The core notion in university level mathematics, however, is the proof. But you can’t learn how to prove something by being told or shown how to do it any more than you can learn how to ride a bike by being told or shown. You have to try for yourself, and keep trying, and falling, until it finally clicks. Moreover, apart from some very special, and atypical, simple cases, proofs cannot be machine graded. In that regard, they are more like essays than calculations. Indeed, one of the things I told my students was that a good proof is a story, that explains why something is the case.
Feedback from others struggling to master abstract concepts and proofs can help enormously. Study groups can provide that, along with the psychological stimulus of knowing that others are having just as much difficulty as you are. Since companies like Facebook have shown us how to build platforms that support the creation of groups, that part can be provided online. And when Coursera is able to devote resources to doing it, I know it will work just fine. (If they want to, they can simply hire some engineers from Facebook, which is little more than a mile away. I gather that, like Google before it, the fun period there has long since passed and fully vested employees are looking to move.)
The other issue, that of evaluation and grading, is more tricky. The traditional solution is for the professor to evaluate and grade the class, perhaps assisted by one or more TAs (Teaching Assistants). But for classes that number in the tens of thousands, that is clearly out of the question. Though it’s tempting to dream about building a Wikipedia-like community of dedicated, math-PhD-bearing volunteers, who will participate in a mathematical MOOC whenever it is offered – indeed I do dream about it – it would take time to build up such a community, and what’s more, it’s hard to see there being enough qualified volunteers to handle the many different math MOOCs that will soon be offered by different instructors. (In contrast, there is just one Wikipedia, of course.)
That leaves just one solution: peer grading, where all the students in the class, or at least a significant portion thereof, are given the task of grading the work of their peers. In other words, we have to make this work. And to do that, we have to take the first step. I just did.
Knowing just how many unknowns we were dealing with, my expectations were not high, and I tried to prepare the students for what could well turn out to be chaos. (It did.) The website description of the exam grading system was littered with my cautions and references to being “live beta”. On October 15, when the test run without the grading part was about to launch, I posted yet one more cautionary note on the main course announcements page:
… using the Calibrated Peer Review System for a course like this is, I believe, new. (It’s certainly new to me and my assistants!) So this is all very much experimental. Please approach it in that spirit!
Even so, many of the students were taken aback by just how clunky and buggy the thing was, and the forums sprung alive with exasperated flames. I took solace in the recent release of Apple Maps on the iPhone, which showed that even with the resources and expert personnel available to one of the world’s wealthiest companies, product launches can go badly wrong – and we were just one guy and two part-time, volunteer student assistants, working on a platform being built under us by a small startup company sustained on free Coke and stock options. (I’m guessing the part about the Coke and the options, but that is the prevalent Silicon Valley model.)
At which point, one of those oh-so-timely events occurred that are often described as “Acts of God.” Just when I worried that I was about to witness, and be responsible for starting, the first global, massive open online riot (MOOR) in a math class, Hurricane Sandy struck the Eastern Seaboard, reminding everyone that a clunky system for grading math exams is not the worst thing in the world. Calm, reasoned, steadying, constructive posts started to appear on the forum. I was getting my feedback after all. The world was a good place once again.
Failure (meaning things don’t go smoothly, or maybe don’t work at all) doesn’t bother me. If it did, I’d never have become a mathematician, a profession in which the failure rate in first attempts to solve a problem is somewhere north of 95%. The important thing is to get enough data to increase the chances of getting it right – or far more likely, just getting it better – the second time round. Give me enough feedback, and I count that “failure” as a success.
As Edison is said to have replied to a young reporter about his many failed attempts to construct a light bulb, “Why would I ever give up? I now know definitively over 9,000 ways that an electric light bulb will not work. Success is almost in my grasp.” (Edison supposedly failed a further 1,000 times before he got it right. Please don’t tell my students that. We are just at failure 1.)
If there were one piece of advice I’d give to anyone about to give their first MOOC, it’s this: remember Edison.
To be continued …
MOOCtalk 
Hi Dr. Devlin. I’m one of your students in Mathematical thinking. I know there were many difficulties and problems for you and for us (it’s not the same to take a class online that taking a class “onlive”). I couldn’t get very good grades in some problem sets (in the 3rd and 4th I obtained around 13/20) but I decided to ride anyway the bike and I think I did well in the exam. I study Economics in Ecuador and I thought I knew maths but you showed me that there are many things to learn. But with the final exam and the Peer grading process I can say I learnt something and I feel I can ride the bike. Even if I don’t get a distinction, at least for me taking this course was a great succes!!
Thanks Dr. Devlin.
I’m also a Student of your class, I’m an engineer down here in Mexico, and I thought I knew math, you showed me that I only knew “highschool math”. I’m decided to master the material of this course and pursue with Real Analysis and Abstract Algebra, after this class I finally see them reachable. I just wanted to say thank you for your time and this opportunity.
Dear Sir
i had the privilege of taking your mooc course – though i did not give any exams.- and i really took back a lot from your classes about logical notation
i also read the book “adding it up- helping children learn math” and in my observation one of the most important job for a teacher is to show the student how a particular thing that he is teaching is so very part of his life.
if a learner is convinced of practical importance – he will naturaaly be more receptive and inquiring.
This is described as one of the goals of teaching in the book as “productive disposition”.
Dear Prof. Devlin,
I am a high school senior from India and am taking your course online, and I have to say, it has changed my entire perspective about Mathematics. I was frankly unaware of this kind of Mathematics, and now it seems as revolutionary to Math as Relativity was to Physics or Wittgenstein was to Philosophy. Secondly, I truly admire the fact that you’ve spread out this previously exclusive education to all parts of the world, removing barriers and the elitism that is prevalent in higher education worldwide. It is an absolute pleasure to learn something I knew previously nothing of, and I will assuredly pursue this for a long time to come.
Dr. Devlin,
I enjoyed learning with you for the MOOC. I wasn’t especially bothered by the problems. I did the best I could, and when problems happened, I tried again. I want to thank you for taking the time to attempt to offer such a difficult course under an experimental system. I think there was plenty for all of us to learn here. And I was expecting the biggest challenges to be math-related!
Dear Mr Kelvin,
I am also one of the students in the course (coming from Romania). Perhaps it is my fault (and likely that of others like me) that we did not post enough in the forums in order to counterbalance the stream of “exasperated flames”. I, for one, thoroughly enjoyed the experience of peer assessment. The fact that there were often technical problems (the page would hang for seconds on end, I sometime had to read the raw LaTeX, etc.) did not really matter all that much.
I particularly found the idea of the “calibration training” very useful; it really taught me how to decide on awarding scores. Had I started off only the Grading Rubric alone, I’m sure I would’ve – incorrectly – stuck to it way too tightly.
I am very curios if you are able to obtain statistics about the accuracy of the grading once the process is over. If you could also send such a follow-up report to the students through Coursera some time in the future, it would be much appreciated.
What I noticed, and it seems natural enough, is that it’s very easy to grade the extremes, but not so the half-baked solutions that still have some merit to them. Also, I’m am curious as to the accuracy of the grading awarded by those students that did not sufficiently master the course material. Are they often awarding high scores to incorrect/incomplete solutions because they are not able to spot the mistakes? Well, I’m sure your list of questions is more comprehensive, this is just one of my curiosities.
In any case, I hope your final assessment of the peer grading exercise is not necessarily that it was only a failure due to the technical issues. Even if that failure will have provided you with valuable insight.
Thank you for your wonderful effort. It is appreciated in all the corners of world.
I belatedly signed up for your class just to check out how you teach math proofs in a MOOC. It sounds like V1.0 was a bit rough.
My daughter attends a Los Angeles Metro area Title I school and I believe that around a dozen of the kids would really benefit from exposure to higher math rather than the standardized test training passing for math and science education today. The curricula today is dreadfully dull. Kids are going to hate math by the time they go through years of this. They need something more interesting to get them to take a second look at math as a major and a career. I hope that you don’t give up and try again.
My PhD is in theoretical physics, not Math, but I would be happy to work as a TA for your next iteration. The class could benefit from TAs, that meet in person with students, and report back to you where the students are having conceptual difficulties.
That the word ‘failure’ has even made it onto the page is bordering on the absurd. Speaking for myself – and many many of my peers who’ve actively provided feedback – this was not far off a revelation. All consuming – akin training for your first cycle race – but brilliant. What an experience to have done this course and (the engaging) Professor Devlin. Of course there are big iterations (at what cost by the way?). I plan to be part of the next iteration – I want to witness this mass social experiment first hand again – and of course continue to ascend into advanced maths, beyond what a 7 week skate-through can achieve.
As I’m in the middle of the grading process I’d better stop here…
It is interesting how the name Keith Devlin was reduced to Kelvin in the previous post!
Dear Professor Devlin – you undertook one of the most challenging tasks which consisted of several completely orthogonal goals. One goal was to educate a massive truly global audience. Another was to implement and test out the system for making it happen considering computer grading was inadequate for the final. There were many other challenges as well. Just consider the massive forum activity. To give this forum a structure, making sure it doesn’t become completely chaotic, doing some semblance of moderation was not a joke.
I am an engineer who is comfortable with empirical formulas not questioning ever the basis of such a formula but just mindlessly applying it. Now, I can stare at the formula and try to build a story (as you put it to describe a proof) to imagine why a number of variables appear in the formula the way they do.
Despite the mixed results I strongly believe your MOOC course has had an enormous impact already in making those who persevered to think mathematically. There are just too numerous side benefits as well. To name a few, perseverance, communication, sharing and helping others, understanding and appreciating cultural differences.
I consider this a success for all the participants (you, your staff, and the students) no matter what. I want to thank and congratulate you and your staff (I can only imagine the hard work and sleepless nights for them).
I am in agreement with others before me. As a member of your course, I actually had no problems at all. No hang-ups, no outages, nothing. Basically – everything worked as I expected it to per your instructions. Of course, I am taking the course from Singapore so maybe the different time frame helped that situation out. As a former Software engineer, I found the entire feat- this MOOC- extremely impressive. As a present math teacher, I found the experience extremely relevant and inspirational. Thanks from all of us who took away even more by far than we expected! I was so excited going in and if possibly am even more excited now! Again, thanks!!
Hello Dr. Devlin (KD): I am a student in the class. First of all, like the other posts, thanks for your time and efforts. I learned more than I thought I would. I do have to be honest. When faced with taking part in the peer review process, I declined the option. I am taking another class also. Likewise in the other class I have also declined the same option. For me, personally it feels like trying to put on a pair of shoes that do not fit. This is just my experience, but I hope my feedback helps you in someway as you soldier on.
Dear Dr. Devlin,
This course and platform combination was a beautiful experiment, and despite teething problems, this looks like the future of education to me. It is not often that I can say that something I participated in on my computer has changed my life. I am a senior. I grew up at a time when “girls didn’t do math”. I signed up for your course because, although I was convinced that I could not do math, I have always wanted to know more about how mathematics works. I never found a book that started from where I was. I did get stuck on the proofs later on in your course, but I have made huge gains from where I started. For me, this course has been a small miracle and has now enabled me to keep going further in my exploration of a subject long closed to me. I have ordered some of the books recommended in the forums: that is another great feature of this course–a resource list.
Thank you for going slowly enough to let me truly understand. Thank you for jumping off the cliff to do this course, and for the joy you have brought this old crone!
Professor Devlin, I am taking this MOOC with my wife. We have extremely different educational backgrounds (Engineering and PHD Music performance) and have enjoyed exploring this new realm of Mathematics together, while also exploring the world of MOOCs, and your discovery process via this blog and your comments throughout the course.
The course did encounter a lot of challenges! (With a career leading information technology projects, I can empathize with you). We found this whole process interesting, not a problem. You made it abundantly clear that this was a many layered learning experience. fair warning, so it is hard to understand why people got upset.
Getting MOOCs working effectively is very important, world-changing work. It is far, far better (and bolder) to run the course, knowing there’ll be abundant challenges to overcome and learn from, than to wait for someone else to solve the problems. later. The world needs the MOOC paths to knowledge and we thank you very much for your striving so hard to provide it.
Definitely was not a failure!
I don’t have enough words to appreciate what you and Coursera did. I am one of your virtual students and i think that besides some bugs in the system the ideas and the workflow for the course is right.
I also think that the “calibration training” is needed for having competent graders. Quality calibration training, quality graders is also much in interest of the students and the idea should be enhanced in the future.
Those “exasperated flames” are human and are parts of the human psychology. “Calibration training” was actually presented like a new exam and the people who didn’t pass were angry. Angry that they didn’t pass, don’t get the certificate with distinction, don’t go further to review the work of other students. They felt excluded!
Someone should have expected that this will happen.
It would have been much better if the system, in background, based on some criteria, would have choose the “graders”. Still all other students would have been participate in grading other students. An adaptive machine learning algorithm would then have decided who the graders are.
i can say, without saying big words that this course and Coursera platform have changed a lot in my life. I have discovered you, your book, “The Language of Mathematics: Making the Invisible Visible” and suddenly see mathematics with another eyes. In a way in which i always thought that is possible but to many times disappointed by the way in which it is studied in schools.
Dear Dr. Devlin,
Thanks for what are doing am a first year university in Ghana…i thought i knew maths..lol..apparently that’s not the case your class has really opened my eyes to the light. I appreciate your efforts.
Sir! What a pleasure to have finished the class. Thanks to you and your teaching assistants for a fun and exhilarating time. I want to let you know what the most valuable part of the course was for me: remembering what it feels like to learn math for the first time. I am currently an AP calculusr teacher and often am often stymied by the fact that I have to explain something like the product rule about 50 times to some of my students. Helloooo! It’s the power rule, for crying out loud, not quantum physics. Like you, I try to be patient, an encourage the kids to keep pressing on, but after a while I sometimes find myself wondering if they will ever “get it”.
I first heard of your course this summer and my first thought was that I would take it so as to have something to do with the kids after the AP test in early May. That is still the plan, by the way, although after taking the course I will no doubt have to rewatch the videos and rework the problem sets and assignments about 1000 times before I feel even remotely comfortable teaching even the most basic concepts. And so the class started and I was determined to give it my best, and above all, to finish, which I didn’t think would be that big of a deal.
Except my last analysis course is now 25 years in the past, and as a high school teacher I haven’t messed with any of the topics in a very long time, and as you well know, the AP program is in no way a proof based calculus course, so, yeah, from day one I was being pushed in ways that I simply haven’t been pushed in a very long time. Things that should have been simple weren’t, and for the first time in a long time I had to do what I tell me own students to do when they get bogged down: persevere, persevere, persevere.
And I got my butt kicked – truly and honestly kicked despite giving it my best shot – but that butt kicking did something very significant for me: it reminded me how easy it is to struggle in math, and I doubt very much if I will ever again get exasperated with my own students when they ask me about the product rule for the 50th time.
So, thanks. As teachers we always have our goals for the classes we teach, and I hope that most of your were met. I’m not sure if one of your goals was helping people like me get more in touch with my own students, but it should have been, for that is exactly what you did for me. I wish I would have understood more of what was going on, and will go back to the material at a later date in an attempt to do that, but for now, you have made me a better teacher, and for that I thank you and your team.
Thanks to all who posted above for your comments. I look forward to giving version 2.0 of the course some time next year.
Prof. Devlin.
I really loved your course. I absolutely sure that it has helped me in understanding mathematics a little bit better now. I’ve finished with the peer grading and anxiously waiting for results!
Thanks a ton!
R.
Dear Prof. Devlin,
Like most of the previous responders to this post, I am a student in your course. Unlike many, I have no math background at all; I’m a retired language professor, and I haven’t take a math class since my senior year in high school (1952). For me, the course was wonderful. I learned a lot and enjoyed the experience. There were obviously some glitches; they were interesting, too. As a former teacher, one of my motives for signing up was to see what the technology and the process were like. Thank you for all you’ve done.
Dr. Devlin,
Thank you for blogging about your experience building and running your MOOC. As a college professor who’s been dabbling with online learning, it’s been very helpful to get this info.
Concerning peer grading, I’ve noticed that the MOOCs I’ve participated in (lurked around would be more accurate) have a significant number of ‘students’ who would qualify as TAs or, in some cases, instructors. These expert students seem to play a key role in the online discussions and often provide significant materials to the community. I’m wondering if you have a way to know how many of your peer evaluators are peers and how many are experts? I also wonder if these having the expert participants is sustainable, or if it’s just a startup curiosity phenomenon. This is related to your idea of a body of volunteer TA you mentioned in your post. I’m not sure that’s a sustainable model.
Again, many thanks for you insights. I’m intrigued.
-Bill Kitch
I wouldn’t call the peer grading a failure. The training module could be improved. I made some comments about that in the forum, which are constructive I hope. The rubric also doesn’t apply well to rather short and trivial proofs, but I guess it is the approach that has the best overall coverage. It has at least the advantage of providing some structure and normalisation. There is no getting around peer grading anyway for a course like this, so we better make the best of it.
I’ve also read some of the harsh comments on the forum. The anger in them is misplaced because this course is most and for all a matter of self-deployement. However, I can understand it a bit in the sense that a certificate is issued at the end. Although this does’t equal credit points it has some value. You will see these courses popping up in CVs. The value of such a certificate will increase in the future, where a formal degree may become less important. After all, a degree is some aggregation of courses that is valued as a ticket to the labour market. This is purely by convention. Perhaps one day a certified shorter online course trajectory will be valued as much as a degree. In this trajectory more general courses can move to a parallel broader self-education thread. Another reason such a trajectory can be shorter is that people will never stop following courses. They will just take less at the same time.
If a certificate represents some value then some people may be upset when they fear it is jeopardised by something beyond their control. That may explain why certain reactions on the forums are somewhat hard.
I would like to conclude by saying that this course was really great. It was an honour to be able to follow it. On top of that it was free, so we should be very grateful. The experimental aspect is exiting rather than disturbing. Who knows later we can proud ourselves for participating in pioneering an educational revolution.
Dear Professor Devlin,
Not only your teaching, but your blogs are also so much enjoyable to read. I consider IMT as definitely a “success” story! Keep offering more such MOOCs and keep writing such blogs! Thanks…
Regards,
Pranab
Prof. Devlin,
Thank you so much for your positive and continuous effort to get this first round done til the end and help us learn. I will remember the lightbulb and Edison… thanks for that!
Dani
There is a whole senior sector who benefit from moocs and webinars.
I for one learned a lot and enjoyed the course. I am 80 and the last
formal math course I took was 63 years ago.
Thanks for all your work and willingness to explore.
John Carollo
Prof. Devlin,
Your instruction is first rate, and very clear, so you succeeded well. Peer review became Napster + stickers. As a Class of 1971 Stanford Grad, I’m delighted to see someone start a new paradigm.
Pivot to a freemium test-grading model underwritten by Pepsi. I arrived two days before the end of term, but managed to download all the lectures and assignments. I’m the English Literature student that was curioius about mathematical linguistics and have been converted! Another success.
Thanks for enriching my life.
Mark Richard-Fogg Stanford 1971
Prof Devlin….you are awesome. Even at the closing of this IMT course you still find the time to fwd us all such interesting articles. Although my work commitments did not permit me to finish the course 100%, I have thoroughly enjoyed the course. You are a credit to your profession….may your tribe increase. From Hoe Swee Yoke, a pure math teacher from Penang, Malaysia.
I am taking your course and revisiting math that I did 30 years ago in high school. your lectures and several things about the course were innovative and awesome. I learnt a lot and I feel I will be able to coach my daughters in math which became rusty in my mind. Found this interesting article in NYT that 2012 is the Year of MOOCs .
I would also vote that Prof KD’s IMT is The Math Course of 2012.
Thanks Dr Devlin,
I express to you my sincere congratulations that you and your people have succeeded taking giant steps and accomplishments on this foundational MOOC with 65,000 students, special thanks for your hard work on teaching the crowds and the long hours to make it work, your legacy for us and the future generations on Math-Thinking is and will continue to be really appreciated always, you are a Math Champion Dr Devlin. My deep appreciation to the Coursera.org team and the wonderful organization you represent Standford for such a wonderful work, we as your students are part of the history, we made it together, You made it possible.
-Edgart Gonzalez.
Instituto Tecnologico de Oaxaca, Mexico 1987.
Instituto Tecnologico de Monterrey, Mexico 1994.
Vertical Sciences Labs. California USA 2012.
e-KnowBrary™.com by the vertical Sciences Labs™{vScLb™.com}
Author Book in production:
Computing in Sciences Youth’s Learning™
Youth’s PhD™ Book Series ©2012 vScLb™
Hi Dr Devlin
Just a brief note to says thanks for a most enjoyable course. I have to admit that a lot of what you taught didn’t get properly absorbed. As you said yourself we probably think we understand the answer to problems like why the square root of 2 is irrational when we probably don’t! Still I must have learnt something as I impressed my son-in-law, who is a high school headmaster and physics graduate, with all those mathematical symbols we have been taught. I showed him the “fooling everybody” quote in symbol form and he seemed most impressed. Or perhaps I am just assuming he was!
Anyone who has an interest in mathematics and how mathematicians think would certainly find this course both useful and thought provoking. Thanks again for giving us the opportunity.