Answering the 64,000-Students Questions

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

With the “instructional” part of the course finished and the remaining students working on the Final Exam (it will be peer graded next week), at last I can sit back and take a short breather. The next step will be to debrief and reflect with my two course assistants (both PhD students in the Stanford Graduate School of Education) and decide where to ride the MOOC beast next.

For sure I’ll offer another version of this course next year, with changes based on the huge amounts of data you get with a global online class of 64,000 students. Despite the enormous effort in designing, preparing, and running such a massive enterprise, there are three very good reasons to pursue this.

First, and this I believe is one of the main reasons why Stanford is supporting the development of MOOCs (I am not part of the central, policy-making administration), designing, running, and analyzing the learning outcomes of MOOCs is a tremendous research opportunity that will almost certainly result in new understandings of how people learn, and as a result very likely will enable the university to improve the learning experience of our regular on-campus students. After just five weeks, my two graduate assistants have enough data to write several dissertations, in addition to the one they need to get their doctorates.

Second, there is a huge, overall, feel-good factor for those of us involved, knowing that we can help to provide life-changing opportunities for people around the world who would otherwise have no access to quality higher education. Is what they get as good as being at Stanford? I very much doubt it, though the scientist in me says we should keep an open mind into the eventual outcomes of what is at present a very novel phenomenon. But if you compare a Stanford MOOC with the alternative of nothing at all, then already you have an excellent reason to continue.

Third, and this is something that anyone in education will acknowledge makes up for our earning a much lower salary than our (often less formally qualified) friends in the business and financial worlds, there is the pleasure of hearing first-hand from some of our more satisfied customers. The following is one of many appreciative emails and forum posts I have received as my course came to and end:

Mr. Devlin and all members of the Introduction To Mathematical Thinking team, I just wanted to say Thank You for everything that you have done to share your knowledge and giving your time and great effort to help others learn. I imagine that this is not an easy project to lead and sustain on a continuous basis. However, you have done a wonderful job in relaying your message. Through your efforts, you have helped many people in the process; especially me. Until this class, I hated math. I hated the idea of learning math or thinking in mathematically analogous methods that are applicable to real world situations. I just didn’t get it. I’m still a little confused about why I am able to comprehend your lessons as effectively as I am (which is saying a lot considering how much I hated math) when I have not been able to do so in the past. Now, I find myself looking forward to your classes everyday! I look forward to using what I have learned from the last video lectures or assignments and using those lessons in situations I did not think possible. And now, I love math! Your instruction has helped me to think more logically and to draw more concise conclusions with issues that I am trying to handle. This is indeed a skill. This is also a skill that you can build upon throughout your lifetime if one chooses to do so. Though I may not be at the level of learning that I should be at, I have learned more in the past three weeks than I have learned throughout my life; and I will continue to learn. I am very serious about this statement. So, thank you All. Thank you, Mr. Devlin. Great Job and Cheers!


To be sure, there were trolls on the course discussion forum, for whom nothing we did was right. But one of the benefits of having tens of thousand of students is that within at most an hour of a flame post appearing, tens of others jumped on the offending individual, and within a short while all that was left was a “This comment has been deleted” notice. As the course wore on, the trolls simply dropped away.

Though there was the one individual who, in week four, posted a comment that he hated my teaching style and was learning nothing. Given that this was a free course that no one was under any compulsion to take, and for which no official credential was awarded, one wonders why this person stuck around for so long!

That example provided no more than an amusing anecdote to tell when I start to give talks on “What’s it like to teach 64,000 students?” (Invitations are already coming in.) But there is a somewhat closely related issue that I find far more significant.

Like almost all current MOOCs, there was no real credentialing in my course, so the focus was entirely on learning for its own sake. (As a lifelong math professor, used to teaching classes where many of the students were there because they needed to fulfill a mathematics requirement, having a class of students who were there purely voluntarily added appeal to my giving a MOOC.) To be sure, there were in-lecture quizzes, machine-graded assignments, and a peer evaluated final exam, but the only people who had access to any student’s results were myself, my two course assistants, and the student. Moreover, there was no official certification to back up a good result (the course offered two levels, Completion and Completion with Distinction), and turn it into a form of credential.

Yet many students had an ongoing obsession with their grades, and indeed pleaded with me from time to time to re-grade their work. (Clearly not possible in a 64,000 student MOOC. Besides, I never saw their work. How could I?) As a competitive person myself, I can appreciate the desire to do well. But with literally nothing at stake, I was at first surprised by the degree to which it bothered some of them. When I figured out what was probably going on, I found something that bothered me.

Unlike most MOOCs, mine, being at first-year university level, can be taken by high school students. Indeed, since my primary target audience comprised students entering or about to enter university to study mathematics or a math-related subject, I expected to get high school seniors, and designed my course as much as possible to accommodate them.

I’m guessing that the majority of students who were obsessed with grades were still at high school – indeed, most likely a US high school. That grade obsession I observed is, I suspect, simply a learned behavior that reflects the way our K-12 system turns the learning of a fascinating subject – one of humankind’s most amazing, creative, intellectual achievements – into a seemingly endless sequence of bite-sized pieces that are fed to the student in a mandated hamster-wheel.

No wonder they could not relax and enjoy learning for its own sake. Any natural curiosity and desire to learn – something all humans are born with – had been driven out of them by the very institution that is supposed to encourage and develop that trait. In its place was mere grade hunting.

Do I know this for a fact? No. That’s why I used those hedging words “guess” and “suspect”. But something has to explain that grade obsession in my course, and it certainly brought to mind Paul Lockhart’s wonderful essay A Mathematician’s Lament, which I had the privilege to bring to a wider audience some years ago.

But now I digress. Time to wrap up and check the dashboard on the course website see how many students have submitted the Final Exam so far.

Though this post has dropped the title “MOOC Planning”, I am going to keep posting here, as the project goes forward. Stay tuned.

To be continued …


14 Responses to “Answering the 64,000-Students Questions”

  1. 1 Robert Berkman October 26, 2012 at 7:51 pm

    I wonder if you would consider the idea of students performing a self- assessment at different points in the course. You would be surprised how honest they can be with themselves about the quality of their work, although when I’ve done this with my students, I found the boys consistently overrated their performance, while girls tended to be very hard on themselves. Perhaps this kind of “gender bias” could be built into the assessment algorithm.

    • 2 Keith Devlin October 27, 2012 at 6:22 am

      Robert: We originally planned on having peer and self assessments throughout the course, but when the problems inherent in the system became apparent, we decided to put it all at the end where it would not interfere with the rest of the course. Now we have had some experience with the system and the process, and given that the system will presumably get more robust and user-friendly as Coursera continues to develop it, I expect we will increase its use next time.

  2. 3 jkjacob123 October 26, 2012 at 8:00 pm

    Professor Devlin –

    I have been away from mathematics for quite some time (M.Ed. Math Ed in 1992) but jumped on the opportunity to take your course the minute I saw it on Coursera and I will never forget this experience. (And the beauty of this post is that it CANNOT be construed as “brown-nosing” since my final will be peer-reviewed). I have been asked by several people why I would choose to take a course that should be mostly review for me. To that I answer that it was a unique opportunity to be a part of the future of education and take a course with a respected mathematician and author FOR FREE from Stanford! And I have enjoyed every minute of it. My brain hasn’t worked this hard since graduate school.

    Any chance we’ll be privy to any of the data/statistical analysis coming out of this? I would be very interested to know how many women were signed up and completed. Also – how many women under the age of 30 etc? One of my research focuses was in this area and I’m certain you are aware of the issues here.


    • 4 Keith Devlin October 27, 2012 at 6:23 am

      jkjacob123: You can expect to see me post and reflect on some of the course statistics in this blog during the coming months.

    • 5 Chris Aldrich October 29, 2012 at 3:09 am

      jkjacob123, Assuming that you may possibly be teaching mathematics professionally given your degree, surely the course was of additional benefit aside from the pure review, as in watching someone else’s lecture style in mathematics may surely have the double benefit of helping you to hone your own pedagogical style?!

      • 6 jkjacob123 October 29, 2012 at 4:16 am

        Not teaching anymore (or…for now?) but, absolutely. Observation is enormously helpful/supportive. This MOOC experience has made me completely rethink what is possible and what isn’t however. I was always a huge proponent of in-person teaching/training as the best way to go. Until now… 🙂

      • 7 Keith Devlin October 29, 2012 at 4:28 am

        Yup, giving or participating in a MOOC makes you question all your long held beliefs. And the great thing is, we get data that can inform any changes we want to make. 🙂

  3. 8 Alessandro Capretti October 26, 2012 at 8:33 pm

    Here is one 2-to-the-minus-16-th of the feedback from the (happy, proud) 65,536 guinea pigs in the course.

    People like me who enjoy Coursera courses are probably driven BOTH by love of learning and by need of recognition. That is why we care about grades, at least to some extent. I am an older student(>50) with a couple of (not so fancy) degrees, and yet I really cared about getting full marks in every test in Coursera. Nobody will ever know my marks (except my wife maybe) and yet I care. Probably not to the point of demanding a regrading if I had not gotten full marks in a test, but to the point of being annoyed if I think the question is ambiguous and that can affect my grade.

    Is this a lingering perverse effect of high school education (not US on my case)? I think it is probably more of a personality thing, perhaps exacerbated by prolonged exposure to the school system. Nothing to be proud about (on the contrary), but not just the school system’s fault…

  4. 9 Joao October 27, 2012 at 12:40 am

    We need more people like you Mr. Devlin.

    Thank you for enlightening ~64,000 people on what mathematics really is about.

  5. 10 Nate Kidwell October 28, 2012 at 12:43 am

    I loved the class, and thought everything was perfectly explained. Your passion for the topic really came through.

    Early in the course I actually was inspired to make a bunch of proofs of things/algorithms I wondered about, and almost became obsessed. One of the proofs I wrote for myself in the first weeks, that non-perfect-squares have irrational roots, I was pleased to actually find on the final exam.

    As far as grades go, I think the “with distinction” is a good metric that enough of the course was absorbed that another one can be moved on to. That alone is why I think striving for it is pretty reasonable.

  6. 11 Norm October 28, 2012 at 6:23 am

    Prof. Devlin, I have been enjoying your Mathematical Thinking course very much. It has been very successful in enabling me to explore more of the underlying essence of mathematics that I had long suspected I was missing during the many math courses I took for my engineering degree.

    I would like to offer one additional motivation for ‘grade obsession’ by some MOOC students to be added to the growing list of motivations. Having completed several Coursera and Udacity courses in CS subjects, I’ve noticed a segment of students that seem to take very seriously the prospect of using the course ‘certificates’ as credentials for job applications and/or graduate school applications.

    Although Coursera does not _stress_ that use of certificates IMHO ( I feel that Udacity does stress it more ), Coursera does leave that door open by offering job placement services that encourage the sharing of a student’s Coursera records ( These services have been mentioned in Coursera emails that I have received from some of my Coursera CS courses.

    I am not taking any position on whether this is a good or bad thing, just mentioning it as an additional motivation for ‘grade obsession’.

  7. 12 Jess October 31, 2012 at 12:00 pm

    Professor Devlin, I believe that you’ll find when you review the course statistics that a large number of students in this and other MOOCs are closer to your age than to High School age, with many of us participating through an internal love of learning, not any external requirement. Thank you for the opportunity to learn new ways of thinking.

  8. 13 Hector Flores November 2, 2012 at 8:38 pm

    I am continuously thinking about that necessity: how to change that “obsession for grades”?

    The “problem” is very well illustrated by Ken Robinson (Sir)

    Yes, we don’t need a reform in education but a revolution.
    And for me the closest possible solutions comes from Daniel Pink:

    Because the same happens in the work world between money and work.

    “Work is to Money as Learning is to Grades.”

Comments are currently closed.

I'm Dr. Keith Devlin, a mathematician at Stanford University. I gave my first free, open, online math course in fall 2012, and have been offering it twice a year since then. This blog chronicles my experiences as they happen.

Twitter Updates

New Book 2012

New book 2011

New e-book 2011

New book 2011

October 2012

%d bloggers like this: