My absence became considerably longer because, faced with a need to generate revenue in order to survive as a company, *Coursera* rebuilt their course platform, tailoring it to a more production-line, largely instruction-based approach to education than the “Let’s take a typical, highly interactive Stanford course structure and make it available on the Web” that inspired many of the original MOOC pioneers, of which I was one. (Strictly, the Stanford group were x-MOOC pioneers, in deference to the Canadians who, a few years earlier, had created the very first MOOCs, subsequently renamed c-MOOCs, with the “c” standing for “connectivist”, not “Canada”. See here for the tortuous history.)

*Coursera*‘s shift—I would not go as far as to call it a *pivot*, though I think that Silicon Valley beloved term does apply to pioneering xMOOC-provider *Udacity*‘s move to the corporate training market—was totally understandable, given their need to survive as a for-profit company. Unfortunately, many of the features they eliminated from their original platform left my course completely un-runnable.

It has taken almost a year, working with a Coursera engineer, for me to rebuild my original MOOC into something that will run on the new platform. That new edition of MathThink, as we insiders all called the course, launches on January 9. On the whole, I am pretty happy with it. I think many people will find it useful. But it is no longer the course I originally created (or rather, ported from the real classroom to a virtual one), and after the first run in early 2017, I will no longer play an active role. (That great, 1972 Douglas Trumbull sic-fi movie Silent Running comes to mind, with me corresponding to the Bruce Dern character. I know, I view education in a romantic way. I think most educators do. Why else would we do it?)

At some point, I may find the energy and the enthusiasm to re-create something like the original MathThink on the Open EdX platform, which, as a nonprofit, Open Source academic project, is not subject to the commercial pressures on a for-profit company. (Though in a subsequent post I will indicate why I think it will differ in significant ways from my original MathThink.)

While taking a successful course and removing features I found valuable was a frustrating task, the reason I approached it in a sanguine fashion was that I had long been aware of the numerical realities of my course. Of the 40,000 or so who would typically register, around 5,000 would complete the Basic Course, lasting eight weeks, and of them around 1,000 would continue to complete the two-week capstone experience I called Test Flight, where they were given an opportunity to experience the role of a professional mathematician. Of those, at most maybe 100 (that is *one hundred*) would have actually had the fully interactive experience I had been trying to take from my physical classroom onto the Web, involving regular interactions with me and my small army of volunteer Teaching Assistants.

For reasons I gave in an earlier post on this blog, on January 2, 2014, I was more than happy to put in the effort to reach that one hundred or so students around the world. For me, the “massive” part—being able to “reach” tens of thousands of students—was simply a way to find that one hundred or so I would react with on a daily basis for ten weeks. That was more than ten times the number of students I would *really* interact with in a physical classroom. I was sorry to have to give up that twice-yearly fix of global mathematical outreach, where I was given a real opportunity to change a small number of lives dramatically for the better (and, like all MOOC instructors, I was able to do just that). But there was no way I could feel bitter about it. No company can survive when its core market is one hundred, and moreover, many of that one hundred were unable to pay anything for the experience.

So, the old MathThink is dead. Long live the new MathThink.

Meanwhile, get ready to meet MOLEs: *Massive Online Learning Experiences*. That concept is the (potential) jewel I was able to identify when I was raking through the ashes of my original MOOC. As I continually tell my students (including my MOOC students), “failure” is something to be looked upon, not as an ending, but as a learning opportunity that can lead to starting something new. (Google “Edison + lightbulb + failure”.)

When the students in my Introduction to Mathematical Thinking MOOC encounter a difficulty with an assignment problem, many of them take to the course Discussion Forum to discuss it. By far the longest single thread in the course was for Problem Set 6, Question 5, a couple of weeks ago, which grew rapidly to 193 original student posts, garnering 1,051 views.

The mathematical topic was proofs by mathematical induction. I had given an example in the video-lecture, and then presented the students with a number of purported induction proofs to evaluate according to the course rubric. (See the previous post in this blog for background on the course structure and its rationale, together with a link to the rubric.)

PS6, Q5 presented them with a purported induction proof that in any finite group of Americans, everyone has the same age (and hence all Americans have the same age). Clearly, this is a ludicrously false claim.

The argument I gave in support of the statement was 19 lines long. Each line comprised a single, fairly simple statement. The lines were numbered. The students’ task was to locate the first line where the proof broke down.

The question had a clear and unambiguous correct answer. The logical chain held up for a certain number of steps, and then the logic failed. But I had constructed the argument with the deliberate intent of making the identification of that failure line a tricky task. (You will find variants of this problem all over the Web. I made it particularly fiendish.)

And fiendish is how the students found it. In fact, only 1 in 5 (exactly 20%) got it right. One other (incorrect) line was chosen by slightly more students (23%), while other lines selected ranged widely over many of the lines. Indeed, there were only two lines of the total 19 that no one selected.

Many interesting points were raised and debated – in many cases in heated fashion – in the ensuing forum discussion. For an online course focused on group discussion, this was easily one of the most successful problems I gave them, with learning taking place on many levels.

One of the meta lessons I wanted this particular exercise to provide was the realization that there is a lot more to proofs than whether they are right or wrong. (See the companion post to this in my *profkeithdevlin.org* blog for a lot more on what role proofs play in mathematics.) The argument I had constructed was, with one subtly positioned logical slip, entirely correct. 18 of the 19 lines are fine. Yet, the claim purportedly being proved in so absurd, in a very real sense the entire argument must be nonsense from the getgo. And so it is.

The widespread belief that proofs are primarily about right and wrong is the argumentation analog of the equally widely held belief that mathematics is about “answer getting” that I discussed in my recent post on *Devlin’s Angle* for the Mathematical Association of America. (Yes, that makes three Devlin blogs. Everybody has a blog these days. If you want to stand out from the crowd, you need two or more.)

Both beliefs – math is all about answer getting and proofs are all about truth – are, I believe, a consequence of the way mathematics and proofs are presented in our K-12 system. What is taught is so unrepresentative of mathematics as practiced by professional mathematicians, there surely has to be an explanation.

Presumably, the perception that mathematics is about answer getting came about in the days before we had calculators and computers, when (accurate) answer getting * was* an important part of a useful mathematics education. Its continued survival well into the digital age can probably be ascribed to systemic inertia (of which there is no lack in the world of education), with the additional incentive that right/wrong questions are extremely easy to grade (by machine, if you are an administrator who prefers to buy equipment than pay teachers)!

In contrast, evaluating mathematical thinking and problem solving is much more difficult and requires a lot of time on the part of a skilled teacher.

Similarly, for the simple kinds of proofs encountered in high school, determining whether an argument is correct or not is usually easy, but evaluating it * as a proof* is much more difficult and requires a lot more skill and experience – as the students in my MOOC have been discovering to their continued great frustration.

The idea that proofs are primarily about truth and correctness is very ingrained. When presented with an argument that is extremely well crafted but has an obvious flaw (so this clearly does not include my Americans’ age example), many students find it hard to evaluate *the overall structure* of the argument. Yet proofs are all about structure. As I keep emphasizing, to my MOOC students and anyone else who is willing to listen, in effect, proofs are stories mathematicians tell to convince the intended recipient that a certain statement is true.

If you forget that, and focus entirely, or even almost entirely, on logical validity, you end up with absurdities like my example of a logically correct proof of Fermat’s Last Theorem so small it will fit into a Tweet, let alone the margin of a book:

Thanks to some work by Andrew Wiles and Richard Taylor, that tweeted argument is *logically correct*. Every statement follows logically from the preceding part of the argument. If you want to fault it, you have to examine the *structure*, pointing out that there are some steps missed out that the intended reader may not be able to reconstruct, especially as there are no reasons given. (See here and here for the missing bits.)

The fact is not that logical correctness is not important. It’s that its importance is only in the context of many other features proofs need to have in order to function as intended.

What features? Well, for starters, how about the features of proofs I list in the rubric for my MOOC?

I’ll tell you one thing. Andrew Wiles would not have had his paper accepted for publication if he had not addressed all the points on that rubric!

No, Wiles did not take my course before proving his famous result. The flow is the other way round. I formulated the rubric to try to identify some of the factors professional mathematicians like Wiles make tacit use of all the time when writing up proofs for publication. You would not believe the objection many people have to a rubric that tries to make that skill set available.

And I’m not talking about the strange folks who post “it’s the end of civilized life as we know it” commentaries on the Drexel Math Forum (cc-ing me directly, because they suspect, rightly, that I don’t frequent the site). Many of the good folks who voluntarily spend ten weeks struggling through my MOOC object as well. And not a few of them indicate in Forum posts where they learned to put so much emphasis on logical correctness. A fictional composite of a fair number of posts I’ve seen over the five runs of my MOOC runs thus: *When I was at university, if there was a logical error in my proof, the professor would award zero points*.

As a mathematician who knows how f-ing hard it can be to prove an original result, reading those kinds of comments fills me with more dismay that you can possibly imagine.

To end on a positive note, at least you have now seen a concise, but correct proof of Fermat’s Last Theorem.

]]>At the start of the week, registrations were at 38,221, of whom 24,342 had visited the site at least once, with 2,818 logging on in the previous week. But none of those numbers is significant – by which I mean significant in terms of the course I am offering. (People drop in on MOOCs for a variety of reasons besides taking the course.)

The figure of most interest to me is the number of students who completed and submitted the weekly Problem Set. In my sense, those are the real course students. As of last week, they numbered 1,013, and all of them will almost certainly complete the course. That is a big class. The undergraduate class I taught at Princeton this past spring (using my MOOC as one of several resources) had just 9 students.

My MOOC has two main themes: understanding how mathematicians abstract formal counterparts to everyday notions, and how they make use of those abstractions to extend our cognitive understanding of our world.

For much of the time the focus is on language, since that is the mechanism used to formulate and define abstract concepts and prove results about them.

The heavy focus on language and its use in reasoning gives the course appeal to two different kinds of students: those looking to investigate some issues of language use and sharpen their reasoning skills, and those wanting to develop their analytic problem solving skills for mathematics, science, or engineering. (The latter are the ones who typically do the optional final two weeks of the course.)

The pedagogy underlying the course is Inquiry-Based Learning.

To make that approach work in a MOOC, where many students have no opportunity to interact directly with a mathematics expert, I have to design the course in a way that encourages interaction with other students, either on the course Discussion Forum on the course website or using social media or local meetings.

Early in the course, I identify a few students whose Forum posts indicate good metacognitive skills and appoint them “Community Teaching Assistants”. A badge against their name then tells other students that it is worthwhile paying attention to their posts. The CTAs, there are currently thirteen of them, and I also have a back-channel discussion forum to discuss any problematic issues before posting on the public channel.

It seems to work acceptably well. To date, there have been over 3,700 original posts (from 957 students) and 3,639 response comments on the course Discussion Forum.

Since the only practical form of regular performance evaluation in a MOOC involves machine grading – which boils down to some form of multiple choice questions – it’s not possible to ask students to construct mathematical proofs. The process is far too creative.

Instead, I ask them to evaluate proofs (more precisely, purported proofs). To help them do this, I provide a five-point rubric that requires them to view each argument from different perspectives, assigning a “grade” on a five-point numerical scale. See here for the current version of the evaluation rubric.

Notice that the rubric has a sixth category, where they have to summarize their five individual-category evaluations into a single, overall “grade” on the same five-point scale. How they perform the aggregation is up to them. The overall goal is to help the students come to appreciate the different features of proofs, as used in present-day mathematics. The rubric asks them first to look at the proof from the five different perspectives, then integrate those assessments into a single evaluation.

After the students have completed an evaluation of a purported proof, their (numerical) evaluations are machine graded (more about this in a moment), after which they view a video of me evaluating the same proof so they can compare their assessment to one expert.

The goal in comparing their evaluation to mine is not to learn to assign numerical evaluation marks the way I do. For one thing, evaluation of proofs is a very subjective, holistic thing. For another, having been evaluating proofs by both students and experts for many decades, I have achieved a level of expertise that no beginner could hope to match. Moreover, I almost never evaluate using a rubric.

Rather, the point of the exercise is to help the students come to understand what makes an argument (1) a *proof*, and (2) a *good* proof, by examining it from different perspectives. (For a discussion of the approach to proofs I take, see my most recent post on my other blog, profkeithdevlin.org.)

To facilitate this, the entire process is set up as a game with rules. (Of course, that is true for any organized educational process, but in the case of my MOOC the course design is strongly influenced by video games – see many of the previous posts in that blog for more on game-based learning, starting here.)

In particular, the points they are awarded (by machine grading) for how close they get to my numerical proof-evaluation score are, like all the points the Coursera platform gives out in my course, very much like the points awarded in a typical video game. They are important in the moment, but have no external significance. In particular, success in the course and the award of a certificate does not depend on a student’s points total. My course offers a learning experience, not a paper qualification. (The certificate attests that they had that experience.)

Overall, I’ve been pleased with the results of this way to handle mathematical argumentation in a MOOC. But it is not without difficulties. I’ll say more in my next post, where I will describe some of the observations I have made so far.

*Stay tuned…*

]]>

Having made changes to the course on sessions 2, 3, and 4, this time round I have made only very minor tweaks. Session 4 was, I felt, more or less what I wanted it to be. Sure, I could re-do the entire thing with better video, better sound, and slicker graphics. But I’d prefer to wait another year or two before embarking on what would be considerable expense and a fairly significant amount of work. I think the jury is still out as to whether there will continue to be significant interest in MOOCs (at least as they are at present), in particular a MOOC like mine, which goes against some of the common wisdom.

For one thing, my MOOC is very definitely a *course*, with regular submission deadlines, and the videos are long by MOOC standards, averaging around half an hour each.

The former (the deadlines) are designed to try to ensure that as many students as possible are working on a particular topic or assignment at the same time, thereby facilitating fruitful discussions on the course Discussion Forum or on social media. Only very unusual students are able to master mathematical thinking on their own. Everyone else needs constant interaction with others, either with an instructor familiar with the material (something not possible in a MOOC) or with other students working on the same issues.

The longer videos reflect the fact that mathematical thinking cannot be broken down into bite-sized chunks – indeed, mathematical thinking demands the very opposite of morselization. It’s not about individual steps, rather the construction and carrying out of an overall strategy. To put it bluntly, working in intensive, thirty-minute chunks is the absolute minimum required to solve almost any mathematical problem of merit, and in general it takes a fair number of such half hour sessions, if not more sustained efforts.

Of course, there is nothing to prevent someone from working their way through the course materials in their own time, or from pausing the videos whenever they want. Depending on the individual, that could prove more or less beneficial. But they would then be using my materials to create their own learning experience. The course as I designed it is intended to be experienced in a cohort group, with much of the actual learning emerging during, or as a result of, discussions with other students.

Details aside, though, why am I in the MOOC business at all? I still get asked this from time to time.

I articulated my reasons for originally creating the MOOC in the early posts to this blog, starting with my first post in May 2012. Subsequent, modified sessions of the course were driven by a desire to try to “get it right”. Not in the sense of creating “the perfect course”. There is no such thing. Rather, my goal was to create a MOOC that represented what I felt was my best effort to put into MOOC format *a course by me.* What I wanted to create was the closest I could get in a MOOC format to taking a regular, physical class with me.

But what of now? Why am I offering this course for a fifth time? Well, with 20,000 students already enrolled, a full week before the course opens, there is clearly still considerable demand. But that is only part of my rationale. The fact is, with the initial MOOC euphoria now (thankfully) a thing of the past – I never bought into the hype and said so at the time – I still see MOOCs as offering some benefits over traditional, classroom teaching.

One significant benefit of MOOCs over traditional classroom courses is that they offer the possibility to deliver personal, non-threatening, side-by-side, one-on-one education. The course presenter simply has to design it that way. I believe the huge early success of Khan Academy comes from that one factor. Khan’s pedagogic model is poor (though typical of a lot of classroom teaching) and in some instances his content is flat wrong, but his delivery is superb and he puts people at their ease. The guy has charisma, and it flows at you through the audio channel on your computer. What is more, for many people, what he offers is a lot better than they experienced, or are enduring, in a more traditional education setting. (Disclaimer: I know Sal slightly, and like him a lot, but we are not close friends.)

So, seeing what he had done well, I modeled, as best I could, the videos in my MOOC on Sal’s delivery, modified to work for more advanced, less-procedural mathematics.

The second benefit of a MOOC, for those who can take advantage of it, is that it puts the student in full control of their learning. Timing, pace, number of repeats (of items or of the whole course). True, many students cannot handle that. But for the ones who can, it is wonderfully freeing and empowering.

Certainly, the students at Princeton who took my MOOC as part of their course said they preferred accessing the instructional lectures part of the course in video format rather than actual lectures by me, precisely because they could control the pace and “rewind the tape” whenever they needed to.

Neither of those MOOC pluses has anything to do with the Massive Open part, of course. What is the upside for me to having thousands of students? Well, my major satisfaction on that front is that, by being totally open on a global scale, MOOCs can reach a relatively small number of talented individuals, in various parts of the world, who crave and can benefit from a good education, but have no other access to one. I’ll happily tolerate massive dropout from my MOOC in order to reach those few whose lives I really can change.

Of course, I am hardly alone in seeking my reward in my successes with a few. It’s what motivates math teachers the world over.

How do I know I actually am changing some lives? Some have told me. Like all MOOC instructors, in every course I have given, I have e-chatted with a small number of students whose forum posts catch my eye, and some of them eventually tell me I have absolutely transformed their lives.

So, perhaps 75,000 of my 80,000 registrants drop out, as happened with the first session I gave. Before they pull the plug, they may have gotten something of benefit. It may have just been conformation that they really don’t like math, though I suspect that most gain more benefit than that. Be that as it may, however, what really inspires me, is being able to reach 10, 20, 100 – or maybe only 1 or 2 – students for whom my MOOC was the thing that changed their lives.

I hardly ever have an opportunity to do that in a Stanford or a Princeton classroom. The most I can do there is polish a jewel. Maybe.

]]>The familiar, over-hyped and over-played media story about MOOCs was inspired in large part by (the possibility of) large numbers of students taking the same class, all around the world. And in one sense, the numbers are large. Though enrollments rarely reach six figures, compared with traditional physical courses, even a “tiny MOOC” (a TOOC?) can have several thousand students.

But those numbers can be misleading. In particular they appear to position MOOCs as being off the right-hand edge of the familiar universities maximum-class-size chart, where elite liberal arts colleges attract students (and their paying parents) with claims such as “None of our classes have more than twenty-five students”.

For some MOOCs, such off-the-chart positioning is, at least to some extent, appropriate, particularly the introductory-level computer science courses that dominated the first wave of xMOOCs coming out of Stanford and then MIT, where the pedagogy is largely instructional. Those courses are in many significant ways simply very large versions of their physical counterparts, with an Internet connection separating the instructor from the student, rather than the more traditional thirty feet of air. Indeed, some of the early CS MOOCs were built around recorded and streamed versions of actual physical courses, with real students.

But in many cases, that “large-version-of-the-familiar” picture is just wrong. Rather, for many MOOCs the educational model is *one-on-one, apprenticeship learning*. That is certainly the case for my course. I made that choice for, what were for me, two very compelling reasons.

One was my experience as an upper level undergraduate and then a graduate student, when a typical week was spent struggling for hours on my own or with a group of fellow students, interspersed with one or two private sessions sitting next to a professor, as I sought help with the concepts I had not fully grasped or the problems I had failed to solve. That was when I *really* *learned* mathematics.

The other influence was my experience in writing books and articles for newspapers and magazines, and in radio and television broadcasting. As with MOOCs, the popular perception is that those media are about conveying thoughts and ideas to thousands if not millions of others. But as any successful writer or broadcaster will tell you, in reality they are all *one-on-one*. The trick you need to master to make the communication flow is to imagine you are writing or speaking to just one person and to *connect* with them. (In the case of an interview, such as my “Math Guy” discussions with host Scott Simon on NPR’s *Weekend Edition*, there actually is a single discussant, of course – I am speaking with one person – and the listeners or viewers are essentially silent observers. The secret of being a good interviewer, as Scott is, is to be able to act as a representative of the viewer or listener.)

In both cases, education and mass media, the secret to success is to evoke thousands of years of human evolution in social interaction. Ritualized classroom education and mass media are relatively recent phenomena, but interpersonal communication is as old as humanity itself, and the successful teacher or broadcaster can take advantage of the many instinctive, powerful aspects thereof.

In the case of (basic) mathematics teaching, look at the huge success of Khan Academy. (I certainly did in planning my MOOC.) Salman Khan built his organization, and with it his reputation, around a large library of short, video-recorded instructional lessons. Though much of the content is not good, and in many cases mathematically incorrect, and the pedagogy poor (Khan is trained in neither advanced mathematics nor mathematics education), what he does as well as – I would say better than – almost anyone else in the business is successfully package “side-by-side, one-on-one conversation” and distribute it over the Internet via YouTube. He is every bit the master of his chosen medium as Walter Cronkite was of television news delivery.

In designing my MOOC, I set out to create that same sense of the student sitting alongside me, one-on-one. If you can pull it off, it’s powerful. In particular, if you can create that feeling of intimate human connection, the student will overlook a lot of imperfections and problems. (I rely on that a lot – though the reason I do not edit out my frequent fluffs is that I want to portray mathematics as it is *really* done.)

True, what I deliver is not the same as actually sitting side-by-side with me. In particular, the student is not able to talk back to me, nor can I begin by reading the student’s initial attempts and then comment on them. Other features of the MOOC have to provide, as best they can, equivalents of those important feedback channels in learning.

On the other hand, in a physical class of more than a dozen or so, it is not really possible for any instructor to provide ongoing, one-on-one, close guidance to each student.

In fact, strange as it may seem, I think it might be possible to *better* provide some crucial aspects of one-on-one higher mathematics education by making use of a platform designed to provide unlimited scaling, than can be achieved in a traditional classroom.

This is particularly true, I believe, for a course such as mine, where the focus is developing a new way of thinking, not mastering a toolbox of techniques that can be used to solve particular kinds of problems. Here’s why I think that.

The fact is, we don’t know how we do mathematics, or how we learn it. The people who do learn to think mathematically will tell you that they found it within themselves – ultimately, they had to figure everything out for themselves, just as learning to ride a bike comes down to discovering the ability within yourself. (Remember, I am not talking about mastering and applying procedures, which can largely be done without any deep understanding.)

Some, like the famous Indian mathematician Srinivasa Ramanujan (VIDEO), manage to take this step with no human help, working alone from textbooks borrowed from libraries. But most of us find we need the regular encouragement and feedback from one or more others or from a tutor. (See the full length documentary (52 mins) Ramanujan: Letters from an Indian Clerk.)

But how important is it to be physically co-present with that tutor? Is it the feedback that is crucial or do the encouragement and the provision of explanations and examples suffice?

After all, mathematics is, by its very nature, logical – supremely so – which means that it can be discovered by reflection. Particularly the basics of mathematical thinking.

Whether a particular individual has the desire or persistence to persevere with such reflection is another matter. Personality type presumably plays a big role. So too does innate mental power. And there has to be motivation.

But for those who are of the appropriate personality type and who have enough mental capacity and motivation, is it necessary that they spend a period of time physically co-present with an instructor?

Absent individual feedback, modern social media provide a powerful means for humans to come together. Maybe that is enough.

(The face-to-face continuity pieces in my lecture videos are designed to make that human connection as strong as possible. That was the only part of my MOOC where we spent money, to get high quality video that conveys my *presence*. I recorded the handwriting segments in my home, using a cheap consumer camcorder, and I edit my own videos.)

The fact is, a student taking my MOOC can make a closer connection with me than if they were in a class of more than 25 or so students, and certainly more than in a class of 250.

So let’s take stock of what can be delivered to the students in a MOOC.

Certainly, the streamed lecture video of a MOOC delivers more than they would get if they were sitting in a large lecture hall with me doing my thing at the front. The lecture video delivers me in a way the student has complete control over, making it self-evidently better.

And in a large class, the student is not going to get my individual attention, so there is no loss there in learning in a MOOC.

So a MOOC seems to offer more of me than a student would get in a regular, large class.

But they also get a version of that close, one-on-one instruction that they absolutely do not get in a regular class of any size.

Absent being able to speak back to me – something many students have insufficient confidence to do (unfortunately) – I think there is good reason to believe that human connection through social media may be enough to have whatever effect is provided by the real thing.

For sure, for some students, it may be important to have frequent real contact with someone to work with, especially someone who knows enough about the subject to provide constructive feedback. But that can often be arranged locally on the receiving end.

(Equally, shy students can perform much better in an online environment.)

The bottom line then is this. Though I do not know that the modalities in a MOOC are enough to help people learn how to think mathematically, I have yet to encounter any reason that it cannot be made to work.

Mathematics, with its intrinsic figure-out-able nature, may be a special case.

It would be ironic indeed if the subject that has historically been the one that most people find impossibly difficult, turns out to be the one most suited to MOOC learning. (Again, let me stress that I am at not talking about procedural problem solving.)

I doubt that large numbers of students can become mathematicians by taking a MOOC, and the same is true for physical classes. But I see no reason why a great many cannot gain useful mathematical thinking skills from a MOOC, nor that there is an insurmountable obstacle to people with the talent and the motivation using a MOOC to take the first crucial step towards a professional mathematical career.

In any case, I no more am discouraged by recent media articles claiming the death of the MOOC than I was encouraged by those same writers’ breathless hype just twelve months ago. (The only MOOC death associated with the story the New York Times ran on December 10, 2013 was the demise of its own over-hyped and under-informed coverage of a year earlier.) America, in particular, has a strikingly naive perception of education that would be its undoing were it not for a continuing supply of J1 and H1 visas. I plan on moving ahead.

My total spend so far? Forget all those media stories about MOOCs costing hundreds of thousands of dollars. After an initial outlay from Stanford of, I think, around $40,000, to cover initial video recording and editing and student TA support for the first version, and $9,000 to cover the cost of a course TA in the second version (TA-ships being a form of student financial aid, of course), everything since has been on a budget of $0.

In particular, as I noted above, I now do all my own recording (cheap consumer camcorder) and video editing (cheap consumer editing package) at my home in Palo Alto.

Of course, Stanford does pay my salary, but developing and giving my MOOC is on top of my regular duties, and is essentially viewed as research into teaching methods. So when Oklahoma Senator Tom Coburn looks into me for fodder for his annual Wastebook (see Section 63 if you think innovative mathematics education could not possibly be in his sights), I will be able to counter by saying that no taxpayers were harmed in the making of my MOOC.

By the way, the two panel lights I use when I record my handwriting segments (shown in the earlier photograph) have historical significance in the world of MOOCs. I was given them by Google’s Peter Norvig after he had finished using them to record the first Stanford-Google Artificial Intelligence MOOC that generated all the current interest in the medium. A contemporary equivalent of the Ishango Bone?

**THE END**

Why do I devote so much (unpaid) time working on my MOOC? And it is, to be sure, a lot of time, little of it factored in to my official Stanford workload.

According to least one very good, and highly respected (by me no less than many others) educational writer, it is the prospect of fame, as she recently tweeted thus.

I suppose there may be a professor or two somewhere who sees MOOCs as a pathway to fame, but if so, they should definitely take my *Mathematical Thinking* MOOC to develop good numerical sense. A globally distributed, ten week class of maybe 40,000 students, half of whom will watch at most one video and many of whom would not be able to tell you the name of their MOOC instructor if you asked them (the same is true for regular, physical classes, by the way), is hardly fame.

Fame is epitomized by @KimKardashian, with almost 20 million Twitter followers. If that’s your goal, devoting many years of your life to get a PhD ain’t the optimal path!

What academics tend to seek is *peer recognition*. And, believe me, giving a MOOC will, if anything, *reduce* the status of any scholar within the Academy, possibly to an even greater extent than writing books and magazine articles “for the general reader”. (I’ve done both. As an academic, I was doomed long ago.)

The danger of stepping outside the walls of Academia has been recognized ever since The National Academy of Sciences denied entry to Carl Sagan. As a recipient of the Carl Sagan Award for Science Popularization, I am thus doubly doomed.

No wonder I felt I had nothing to lose by jumping onto the MOOC bandwagon – though at the time I started work on my first MOOC it was not so much a bandwagon as a small Stanford wheelbarrow, yet to be discovered by New York Times columnist Thomas Friedman. (He soon made up for missing the start. Just google “Thomas Friedman MOOC” and you will uncover a host of **M**assively **O**ver-hyped **O**utrageous **C**laims.)

Why do academics give MOOCs? While I surely cannot speak for all MOOC instructors, I can probably speak for the many I have talked with, and by and large they all give the same answer. It comes in two parts.

The first part is educational research. (This is the reason why Stanford, my university, provides some – very modest – support for MOOC development.) The process of designing and giving a MOOC provides a wonderful opportunity for an instructor to find ways to improve their teaching craft, and provides educational researchers with massive amounts of data to help us better understand the learning process. For just one illustration of this, check out this article from a MOOC instructor at Vanderbilt University.

The second part is the same answer you will get if you ask someone why they went into K-12 teaching, a profession that not only pays poorly, but ranks so low in the US psyche that a savvy State governor contemplating a run for President will regard you as fair media game for a finger-wagging, photo-opp tongue-lashing:

Teachers are not seeking fame, or wealth. They do it because they have this deep-seated urge to change lives by teaching.

When I joined the tiny band of Stanford faculty who were designing the first wave of MOOCs, our motto was “Let’s Teach the World”, a slogan that I took for the subtitle to this blog. This is what it is about.

It wasn’t a desire to be famous that we found attractive. Heavens, if you are at Stanford, you probably already have all the academic “fame” you could ever want. Rather, the hook was an opportunity to take something we had been providing regularly to a privileged few and make it available to anyone in the world who had access to the Internet.

It was, in short, an idealistic dream. How to operationalize that dream was another question, and there were at least as many approaches as MOOC instructors.

The Stanford-MOOC-pioneering computer science professors Thrun, Koller, and Ng set their initial sights on large numbers of students around the world being able to take CS courses, 100,000 or more (maybe a lot more) at a time.

Recognizing that (introductory-level) computer science is almost certainly a special case – because it is suited to instruction-based learning and a lot of what is being taught is, by its very nature, machine gradable – instructors in other disciplines set different expectations for their courses.

In my case, I had two clear teaching goals in mind, one very much focused on “the world”, the other “egalitarian elitist”.

As a mathematician who has devoted a *lot* of my career to community outreach, through public talks, newspapers, general-audience books, magazines, radio, television, movies (occasionally), blogs, and podcasts, I saw MOOCs as yet another medium to “spread the gospel of mathematics”, moreover a medium that offered the possibility of taking my audience a lot further down the mathematical path than any of those other media.

Broadly speaking, the first six weeks of my *Mathematical Thinking* MOOC attempt to cater to that general audience. I very definitely want to capture and sustain the interest of as many individuals as possible. *Massive* (the M of MOOC) is the goal. My focus is not so much on getting my students to learn mathematics – there is precious little of it in those first six weeks – but to raise their awareness of the nature and power of mathematics, and to help them come to realize that they actually do have a (creative) mathematical mind, it just needs to be unlocked from the panic-inducing prison that traditional K-12 math education so often drives it into.

[Time for another Ken Robinson video. This one is a doozy. It’s the one that made him world famous – unlike MOOCs, TED talks *can* make you famous. For the evidence that what Sir Ken says applies to mathematics, see my own book The Math Gene: How Mathematical Thinking Evolved and Why Numbers Are Like Gossip.]

In the final weeks of my MOOC, I slowly shift attention to my second audience. That audience is a lot smaller. I am looking for people who, in certain key ways, are very much like I was as a teenager.

Growing up in a working class family in post-Second-World-War England, in the grimy, Northern industrial city and port of Hull, with no ready access to quality education (let alone higher education), and no role models for learning in my family or my neighborhood, my innate talent for mathematics would likely have gone forever un-realized.

(Through to my early teens, my school teachers advised me to focus on writing, since they felt I had no mathematical abilities, as evidenced by the fact that I was always the last person to master each technique, and kept asking pesky “What?” and “Why?” questions when “everyone knew” that doing math was all about “How”. “*Our’s not to reason why, just invert and multiply.”*)

Fortunately, at high school I encountered a math teacher who recognized something else in me, and pulled me out of his regular math class to teach myself, with his occasional guidance, from his own college textbooks.

I also started to pore through every available “popular mathematics book.” (There weren’t many back then, but most were available as cheap paperbacks.)

That got me started on a rewarding and fulfilling mathematical journey I have been following ever since.

I am certainly not unique in having stumbled my way into mathematics through chance. For most of my professional career I have been surrounded by people who are a lot better mathematicians than me, and a lot more accomplished, and many of them can tell similar “humble origins” stories. But they come from all around the world. Not many of them, if any, come from where I grew up. Similar places, but not the same place. (It’s a density issue.)

In fact, I was surprised to discover a few years ago that the official listing of “Famous People of Hull” includes just two mathematicians, John Venn (of Venn diagram fame) and yours truly.

That may or may not be a comprehensive listing (I never knew John Venn was from Hull until I saw that entry), but it does suggest that you may have to extend access to quality mathematical learning to populations in the hundreds of thousands (Hull’s population was about 300,000 when I was growing up there, it’s considerably less today), in order to connect with just *one or two* who have talent.

I want to do just that. Citizen Devlin wants to provide mathematical outreach to millions around the world. Keith Devlin the grown-up kid from Hull, wants to reach those few individuals who have talent for mathematics but neither learning role models nor access to good education, and provide an educational opportunity analogous to the one that changed my life.

If the “Famous People of Hull” data is even remotely correct, I need to reach many hundreds of thousands, and perhaps millions, around the world, to stand any chance of connecting to those talented few who currently do not have a seat at the educational table.

(It’s probably not an issue of raw talent density. I am sure there are many people will significant mathematical ability in every part of the world. Rather the challenge is the density of talented individuals you are able to connect with, and as a result recognize and bring out their talent.)

Large dropout rates in MOOCs? Though I work hard to try to keep everyone in my course for the first half, and put considerable effort into keeping as many as possible through to the end of the *Basic Course* (see earlier posts), as far as my *second* motivator is concerned, those dropout rates are not a problem at all. They are part of the filtering process.

I’m looking for “me” – that talented young person with no access, and probably no hope – to give them a similar opportunity to the one that chance brought my way all those years ago.

MOOCs have given me that dream.

In each of the three iterations of my MOOC I have given, I have seen a small number of students who I think *may* be such individuals. They are the ones for whom I have made an exception to my (obviously essential) rule of not communicating individually to MOOC students. That’s reason enough to continue.

In other words, my involvement in MOOCs is in large part driven by my own educational Selfish Gene. Not to replicate *me*, but to replicate what happened to me. Now you know.

As I’ve noted in previous posts to this blog, for the first version of my Introduction to Mathematical Thinking MOOC, I took the first part of a course I had given many times in regular classroom settings, and ported it to a MOOC platform in what I thought was the most sensible way possible. In particular, I changed only things that clearly had to be changed. It was always going to be an iterative process, whereby each time I gave the course I would make changes based on what I had learned from previous attempts.

Given the significant differences between a physical class of 25 entry-qualified students at a selective college or university and a distributed class of 80,000 students around the globe (the size of my first MOOC class in Fall 2012), of widely different educational backgrounds and ability levels, for whom the only entrance criterion was being able to fill in a couple of personal information boxes in a Website, it made sense to maintain – for the first version – as much as possible the contents and structure of the original classroom course. That way, I could focus on the MOOC-specific issues.

After the first session was completed (*survived* more accurately describes my sensation at the time), all bets would be off, and I would follow where the experience led me. I felt then, and continue to feel now, that there is no reason why a MOOC should resemble anything we are currently familiar with.

I watched as Sebastian Thrun quickly moved Udacity away from his original conception of a highly structured, programmed traditional course – with all that entails – to offering more a smorgasbord of mini-courses, built up from what can be viewed as stand-alone lectures. I asked myself then, and continue to do so, if I should hang on to the central notion of a course, and maybe just tweak it.

So far I have decided I should, the main reason being, as I tried to explain in my last post, the kind of experience I feel best results in the kind of learning I want to provide.

In particular, the primary goal of my course was, and is, to help develop a particular *way of thinking* – certain *habits of mind*. That is best achieved, I believe, by focusing on particular “content”.

I used the quotation marks there, because I think it is not accurate to view learning *experiences* (for experiences are what produce learning) as a certain volume of “content” that is “contained” is some sort of container or vessel. But it seems that everyone else knows what the term (*educational*) *content* means – a shared understanding that provides Silicon Valley entrepreneurs with a nice story to raise investment for developing “platforms” to “deliver” that “content” – so I’ll go with it. (I used the word five times in my last post, and no one wrote in to object or say they did not understand what I meant.)

Anther reason for maintaining a course structure (the indefinite article is intentional) is that I want my course to function as a *transition course*, to help students make the shift from high school to university. And for the foreseeable future, I think universities will continue to carve up “content” into delivery packages called “courses”.

The third reason for having a *course* is our old friend, student expectations. Many of my full-term students tell me that they signed up because they *want* a course, with all that entails: commitment, deadlines, testing, and community.

That third reason likely reflects the self-selection implicit in students who sign up for a MOOC, fully 80% of whom (according to recent MOOC research) already have a college degree, and hence are adapted to – and good at – learning that way.

This implies that, by offering a *course*, I may be reinforcing that emergent trend of primarily providing further college education to individuals who already had one.

That may, in fact, be where MOOCs will end up. For sure, Udacity’s recent pivot appears to reflect Sebastian Thrun’s having decided to direct his (investors’) money toward that audience/market.

If the provision of *continuing* higher education for college graduates does turn out to be the main benefit that MOOCs provide, that will surely be something for we MOOC developers to be proud of, particularly in a world in which everyone will need to learn and re-tool throughout their lives. (Major innovations rarely land where the innovators thought they would, or do what was originally intended.)

But in that case, MOOCs won’t yield the massive cost savings in first-pass, higher education that many politicians and education-system administrators have been thinking they offer.

In fact – and here I am probably about to bring the wrath of Twitter onto me – I think the current goal of “solving the problem” of the rising cost of higher education by finding ways to reduce it, *misunderstands* what is going on. I suspect the costs of providing first-pass higher education will continue to rise, *because quality higher education is becoming ever more important for life in the Twenty-First Century*.

Just as the introduction of the automobile meant society had to adjust to the new – and ever rising – expense of gasoline, so too the shift to knowledge work and the knowledge society means we have to adjust to the cost (high and rising) of a first-pass higher education (the fuel for the knowledge society) that stays in synch with society’s needs.

What MOOCs and other forms of online education *have already been shown* to be capable of – and it is huge – is provide lifelong *educational upgrades* at very low cost.

But based on what I and many of my fellow MOOC pioneers have so far discovered – or at least have started to strongly suspect – the initial “firmware” required to facilitate those continual “software” upgrades is not going to get any cheaper. Because the firmware installation is labor intensive and hence not scalable – indeed, for continuously-learning-intensive Twenty-First Century life, not effectively scalable beyond 25-student class-size limits.

The world we have created simply entails those (new and rising) educational costs every bit as much the growth of the automotive society meant accepting the (new and ever-after rising) cost of automotive fuel.

(Oh, and by the way, we in the US need to realize that the knowledge society requires better teacher preparation in the K-12 system as well. Well-educated humans are the new fuel, and they neither grow on trees nor are found underground.)

Okay, that’s enough editorializing for one post. At the end of my last report, I promised to describe how I structure my course so that, while designed primarily to provide a framework for a community learning experience, it can still be useful to folks who want to use it as a resource.

First, what do I mean by “resource”? I decided that for mathematical thinking, it was not possible to produce Khan Academy style “online encyclopedia” materials, where someone can dive in to a single video or narrowly focused educational resource. You simply have to devote more than ten minutes to gain anything of value in what I am focusing on.

So I set my sights on people who come in and complete one or two “Lectures”, a *Lecture* in my case comprising a single thirty-minute video and some associated problem-solving assignments. So I am not delivering “bite-sized learning.” I am serving up meals. (Restaurant meals, where you have time to savor the food and engage in conversation.)

To facilitate such use, the earlier Lectures focus on everyday human communication, ambiguity resolution, logical reasoning, and *very* basic mathematical ideas (primarily elementary arithmetic – though in a conceptual way, not calculation, for which we have cheap and efficient machines).

Only in Weeks 7 and 8 do I cover more sophisticated mathematical ideas. (Weeks 9 and 10 comprise my new Test Flight process, which I described in Part 6 of this series. That part is specifically for advanced *mathematics* seekers.)

Thus, Weeks 1 through 6 can be accessed as a resource by someone not strongly interested in *mathematics*. At least, that is my current intention.

Admittedly, someone who delves into, say, Week 4 might find they need to go back and start earlier; but that’s true of Khan Academy as well, and is surely unavoidable.

By making the awarding of a Statement of Accomplishment dependent on *completion* of the Basic Course (first eight weeks), not the achievement of a particular grade, I hope to be able to maintain and reward the participation of someone who begins by just “trying out the course” and gets hooked sufficiently to keep going.

To cater for this dual use as much as possible, in addition to changing the course structure, the upcoming new session has four new videos, and I modified four existing ones. (All the time keeping that magic ingredient “content” the same.)

Well, that’s where I am at present. As I noted earlier, this blog series is essentially my lab book – complete with speculative reflections – made public in real time. (I am already deviating from things I said in this blog just a year ago.)

Ah yes, last time I also promised I would say “what motivated me to give a MOOC in the first place – and still does.” The answer is, “Reaching students who do not currently have access to quality higher education.”

That probably seems very much at odds with everything I’ve said above. It’s not. I’ll explain why in my next post.

]]>One issue I keep returning to is whether my MOOC should be a *course*. Or, to put the question a more useful way, what features of a classroom course do I want or need to carry over to a MOOC, what features should I jettison, and what new features should I add?

I raised the issue in my blogpost of August 31, 2012, just before my MOOC launched for the first time. Since then, students’ expectations (as expressed in emails to me and in the discussion forums) have continued to confirm my initial instinct that there are good reasons to carry over a lot of traditional course structure.

Still, the question is not going to go away. I brought it up again in June of 2013 after completing the second version of my MOOC, noting that the majority of my students treated the course as a *resource* rather than a course.

In those early posts, I made a number of references to Khan Academy, an educational resource I now have very mixed feelings about. (In particular, I think Sal’s enthusiasm and undeniable – and hugely valuable – ability to project his personality through his voice, and thereby to remove much of the fear that many of his followers may have toward mathematics, fall well short of what he could achieve, due to poor pedagogy and way too many elementary – but educationally important – factual mistakes.) I made several key choices based on what could be learned from his endeavors.

One thing I did not do was go the route of turning my MOOC into a collection of Khan-like, standalone, bite-sized snippets. Indeed, deliberately ignoring the current buzz that the audience will drop precipitously if your videos run more than seven minutes, I decided to aim for half-hour chunks. Hey, if thirty-minutes works for *Seinfeld* and *Thirty Rock*, why not for *Mathematical Thinking*? (Remember, I’m looking at a highly selective audience who have voluntarily chosen to enroll in an online math course! I haven’t completely lost it – just enough to keep trying to make this free online course thing work in the first place.)

My decision was largely because the material simply cannot be broken up in that way. Unless you are a mathematical genius, when it comes to mathematical thinking, most of us find that thirty-minute chunks is the absolute minimum time commitment to make any progress at all, and a lot is lost if you cannot arrange for much longer periods. The very last “lecture” of the course actually lasts an hour and a half, with the original video cut up into three segments of roughly equal, thirty-minute lengths. And students who have completed the course say they wished I had spent even more time on the one (capstone) topic I covered in that last lecture.

Since approximately 5,000 students have, on average, stayed with the course to the end each time, I definitely want to continue to provide the learning experience they have clearly been looking for. (In my next post I’ll say how, at the same time, I try to cater for those seeking a resource.)

A significant part of that experience is, I believe, being part of a community, where everyone is working toward the same goal, with regular pressure points (deadlines) that force them to keep sufficiently in lockstep so that they can exchange ideas and express community reactions in real time. Though many of them do not post regularly on the community discussion forums, they do (I assume) follow them, finding answers to their questions and surely being encouraged to learn that they are not alone in finding something particularly difficult or confusing.

That sense of community is, to my mind, an important part of my course. In the (necessarily) simplistic terminology introduced to try to explain the conceptual difference between the original Canadian MOOCs originating from Athabasca University and the unrelated MOOCs coming out of Stanford some years later, my course is *a c-MOOC in x-MOOC clothing*. (See the Wikipedia article for the tangled history.)

From the very first lecture, I recommend repeatedly that students try to form small learning communities to work through the weekly problem assignments that are the heart of the course.

And there we have another reason why I have not carved my course into bite-sized instructional videos. It’s not about instruction! The expressed goal is not “*teaching* mathematics” but guiding folks on a process of learning *how to think* a certain way. In particular, learning how to set about solving a novel problem that perhaps only partially resembles one encountered before.

In other words, in my course the devil is very much *not* in the details. It’s in the overall flow of ideas, the swirling cloud that hovers above all those details.

The key for making that transition from “template recognizer and applier of known techniques” to “creative problem solver” is to rise above the details and grasp the meta-cognitive aspects of good problem solving.

Having myself made that transition by sitting next to my senior tutor (a professor) in my senior undergraduate year and then my doctoral adviser for the subsequent three years, and watching and listening to them as they worked through problems with me (a very one-sided “with”!), I knew first-hand that the process works. I also know of no other way that does.

It’s a slow process, to be sure. Many students in my regular classes over the years, and far greater numbers of students in my MOOCs, have not been prepared, and in some cases not willing, to adjust to that different pace.

I lost count of the number of MOOC students who expressed frustration (and more) at how slowly I was moving, how I “rambled” and “repeated myself,” and how “unprepared” I had been when I sat down to record those videos.

My approach was, of course, carefully thought out and deliberate. I never intended to give a slick, prepared presentation. (I do many of them, and there are videos all over the Web. But those presentations are about infotainment, not learning to think a different way.)

My approach was always about providing a window into one person’s (mine) thought processes. Not to mimic me. That would make no sense in terms of learning how to think creatively. Rather, to gain sufficient insight to be able to develop that capacity in themselves.

Of course, I can provide just one example – me. But one example is enough. Because the capacity for original thought is in every one of us. It just has to be unleashed.

Evolution by natural selection has made all of us creative problem solvers. That is *Homo sapiens’* great survival trick. Unfortunately, an educational system developed in the industrial age to turn innately creative humans into compliant cogs in organizations, suppresses that innate creativity, rewarding fast acquisition and retrieval of facts and rapid execution of procedures, a sad turn of events for today’s world, as summarized brilliantly by the provocative and always entertaining Sir Kenneth Robinson in the animated talk I will leave you with.

Creativity is in all of us. You see it in every small child. Despite systemic education’s efforts to suppress it, it remains eager to break out. (Google *dopamine*.) It does not take much of a stimulus to make it (start to) happen. A ten week MOOC may seem very short. But it may be enough to initiate the process. (Google “Prison Break”.)

* * *

Next time I’ll describe how I structure the *course* so that, while designed primarily to provide a framework for a community experience, it can still be useful to folks who want to use it as a resource. I’ll also say what motivated me to give a MOOC in the first place – and still does. Meanwhile, here is Sir Ken:

In the past, when students enrolled for my MOOC, they essentially had three options. One was not to take it as a course at all, but just regard it as a resource to peruse over time or to pick and choose from. A second was to take the entire course, but do so on their own time-scale. Or they could take it as a course, and go through it at the designated pace.

As do many MOOC designers, I tried to make sure my course could be used in all three ways. Though the vast majority of MOOC students fall into the first category, the other two are the ones that require by far the greatest effort by the course designer. They are the learners who have significant ambitions and will put in a lot of effort over several weeks.

The students in the last category will surely gain the most. In particular, they move through the course in lockstep with a cohort of several thousand other students who can all learn from and support one another, as they face each course deadline at the same time. Those students form the core community that is the heart of the course.

When the new class enrolls at the start of February, the ones intending to take an entire course as scheduled will have a new choice. They can take what I am calling the Basic Course, which lasts eight weeks, or the Extended Course, which lasts ten. As I described in my last post, those extra two weeks are devoted to a process I am calling Test Flight.

In the previous two versions of the course, the final weeks nine and ten had been devoted to a Final Exam, one week for completion of the (open book) exam itself, the following week to peer evaluation. In peer evaluation, which started as soon as the class had completed and submitted their exam solutions, each student went through a number of activities:

1. Using a rubric I supplied, each student evaluated three completed examination scripts assembled by me, and then compared their results to mine. (Those three samples were selected by me to highlight particular features of evaluation that typically arise for those problems.)

2. Having thus had some initial practice at evaluation, each student then evaluated three examination scripts submitted by fellow students. (The Coursera platform randomly and anonymously distributed the completed papers.)

3. Each student then evaluated their own completed examination.

This was the system Coursera recommended, and for which they developed their peer evaluation module. (Actually, they suggested that each student evaluated five peer submissions, but at least for my course, that would have put a huge time requirement on the students, so I settled for three.)

Their original goal, and mine, was to provide a means for assigning course grades in a discipline where machine evaluation is not possible. The theory was that, if each student is evaluated by sufficiently many fellow students, each of whom had undergone an initial training period, then the final grade – computed from all the peer grades plus the self-grade – would be fairly reliable, and indeed there is research that supports this assumption. (Certainly, students who evaluate their own work immediately after evaluating that of other students tend to be very objective.)

As far as I could tell, the system worked as intended. If the goal of a MOOC is to take a regular university course and make it widely available on the Internet, then my first three sessions of the course were acceptably successful. But MOOCifying my regular Mathematical Thinking (transition) class was always just my starting point.

Since I was aware from the outset that the MOOC version of my regular classroom course was just a two-dimensional shadow of the real thing, where I interact with my class on a regular basis and give them specific feedback on their work, my intention always was to iteratively develop the MOOC into something that takes maximum advantage of the medium to provide something new of value – whatever that turns out to be.

I expected that, as MOOCs evolve, they would over time come to be structured differently and be used in ways that could be very different from our original design goals. That, after all, is what almost always happens with any new product or technology.

One thing I observed was that, while students often began feeling very nervous about the requirement that they evaluate the work of fellow students, and (justifiably) had significant doubts about being able to do a good job, the majority found the process of evaluating mathematical arguments both enjoyable and a hugely beneficial learning process.

Actually, I need to say a bit more about that “majority” claim. My only regular means of judging the reactions of the class to the various elements of the course was to read the postings on the course discussion forums. I spent at least an hour every day going through those forums, occasionally posting a response of my own, but mostly just reading.

Since the number of regular forum posters is in the hundreds, but the effective (full-term) class was in excess of 5,000 in each of the sessions, forum posters are, by virtue of being forum posters, not representative. Nevertheless, I had to proceed on the assumption that any issue or opinion that was shared (or voted up) by more than one or two forum posters was likely to reflect the views of a significant percentage of the entire (full-term) class.

Since I made gradual changes to the course based on that feedback, this means that over time, my course has been developing in a way that suits the more active forum posters. Arguably that is reasonable, since their level of activity suggests they are the ones most committed, and hence the ones whose needs and preferences the course should try to meet. Still, there are many uncertainties here.

To return to my point about the learning and comprehension benefits evaluators gained from analyzing work of their peers, that did not come as a surprise. I had found that myself when, as a graduate student TA, I first had to evaluate students’ work. I had observed it in my students when I had used it in some of my regular classes. And I had read and heard a number of reports from other instructors who noted the same thing.

It was when I factored the learning benefits of evaluating mathematical arguments in with my ongoing frustration with the degree to which “grade hunting” kept getting in the way of learning, that I finally decided to turn the whole exam part on its head.

While some universities and some instructors may set out to provide credentialing MOOCs, my goal was always to focus on the learning, drawing more on my knowledge of video games and video-game learning (see my blog profkeithdevlin.org) than on my familiarity with university education (see my Stanford homepage).

Most of what I know about giving a university-level course involves significant student-faculty interaction and interpersonal engagement, whereas a well-designed video game maintains the player’s attention and involvement using very different mechanisms. With a MOOC of necessity being absent any significant instructor-student interaction, I felt from the outset that the worlds of television and gaming would provide the key weapons I needed to create and maintain student attention in a MOOC.

[A lot of my understanding of how TV captures the viewer’s attention I learned from my close Stanford colleague, Prof Byron Reeves, who did a lot of the groundbreaking research in that area. He subsequently took his findings on television into the video game business, co-authoring the book Total Engagement: Using Games and Virtual Worlds to Change the Way People Work and Businesses Compete.]

So from the outset of my foray into the world of online education, I was looking to move away from traditional higher-education pedagogic models and structure, and towards what we know about (television and) video games, hopefully ending up with something of value in between.

The idea of awarding a Statement of Accomplishment based on accumulated grade points had to go sooner or later, and along with it the Final Exam. Hence, with Session Four, both will be gone. From now on, it is all about the experience – about trying (and failing!).

The intention for the upcoming session is that a student who completes the Basic Course will have learned enough to be able to make useful, and confident use of mathematical thinking in their work and in their daily lives. Completion of the Test Flight process in the Extended Course will (start to) prepare them for further study in mathematics or a mathematically-dependent discipline – or at least provide enough of a taste of university-level mathematics to help them decide if they want to pursue it further.

At heart, Test Flight is the original Final Exam process, but with a very different purpose, and accordingly structured differently.

As a course culmination activity, building on but separate from the earlier part of the course – and definitely not designed to evaluate what has been learned in the course – Test Flight has its own goal: to provide those taking part with a brief hands-on experience of “life as a mathematician.”

The students are asked to construct mathematical arguments to prove results, and then to evaluate other proofs of the same results. The format is just like the weekly Problem Sets that have met throughout the course, and performance level has no more or less significance.

The evaluation rubric, originally employed to try to guarantee accurate peer grading of the exam, has been modified to guide the evaluator in understanding what factors go into making a good mathematical argument. (I made that change in the previous session.)

After the students have used the rubric to evaluate the three Problem Set solutions supplied by me, they view a video in which I evaluate the same submissions. Not because mine provides the “correct” evaluations. There is usually no single solution to a question and no such thing as the “right” one. Rather, I am providing *examples*, so they can *compare* their evaluations with mine.

After that, they then proceed to evaluate three randomly-assigned, anonymously-presented submissions from other students, and finally they evaluate their own submission.

*Procedurally*, it is essentially the same as the previous Final Exam. But the emphasis has been totally switched from a focus on the person being evaluated (who wants to be evaluated fairly, of course) to the individual doing the evaluation (where *striving* for a reliable evaluation is a *tool to aid learning on the part of the evaluator*).

Though I ran a complete trial of the process last time, the course structure was largely unchanged. In particular, there was still a Final Exam for which performance affected the grade, and hence the awarding of a certificate. As a consequence, although I observed enough to give me confidence the Test Flight process could be made to work, there was a square-peg-in-a-round-hole aspect in what I did then that caused some issues.

I am hoping (and expecting) things will go smoother next time. For sure, further adjustments will be required. But overall, I am happy with the way things are developing. I feel the course is moving in the general direction I wanted to go when I set out. I believe I (and the successive generations of students) are slowly getting there. I just don’t know where “there” is exactly, what “there” looks like, and how far in the future we’ll arrive.

As the man said, “To boldly go …”

]]>**In Part 5, I wrestle with grading, and once again expectations raise their troublesome head.**

** **Like all MOOCs coming out of Stanford, my Introduction to Mathematical Thinking course carries no college credit, nor does it lead to any kind of Stanford certificate. For the first three sessions, students whose aggregate grade was above a certain threshold did receive a Statement of Accomplishment, and if their grade was particularly high, their statement testified that the Accomplishment was “with Distinction”. Those terms were stipulated by Stanford, though it was I, as instructor, who issued them, not my university.

Though some courses on Coursera can, for a small fee, be taken in a fashion that provides a degree of certification that the individual named on the course completion certificate is indeed the person who took the course, that’s not the case for Stanford MOOCs. It is also very much inconsistent with my intent in offering the course, which is to offer a course where the entire focus is on learning, not credentialing.

To me, focusing entirely on learning, and not offering a credential, is particularly suited to a course that aims to provide general purpose thinking skills that can be used in many ways in different walks of life. There are so many ways in which mastery of mathematical thinking can be advantageous in other courses, it can help people get credentials in many subjects, to say nothing of the non-academic benefits it can yield in professional and personal life. In today’s world, mathematical thinking ought to be regarded as on a par with basic literacy, not something to emblaze on a certificate.

In fact, I structured the course to maximize those general benefits, by keeping the mathematical content at a very elementary level for the first six weeks, focusing instead on logical and analytic thinking and the process of bringing precision to issues that are initially vague or ambiguous.

My decision to focus on learning, not the awarding of credit, was also heavily motivated by the fact that, arguably much more than in any other discipline, misguided educational policy has turned mathematics from a creative human endeavor into a relentless and mind-deadening treadmill of test taking. Not in all countries, to be sure, but certainly in the two with which I am intimately familiar as a practitioner, the US and the UK, and many others whose educational systems I am also acquainted with.

The majority of my MOOC students would, I knew, have never encountered a course that focuses on (creative, original) mathematical *thinking* (as opposed to mastering and applying standard procedures – something that can often be done with almost no thought whatsoever, and indeed can be done far more efficiently these days by apps you can run on your smartphone). So why spoil their first taste of something different, something creative and rewarding, by testing them?

Of course, in a discipline that is about problem solving, each student’s work needs to be evaluated and the results transmitted to them, so they know how they are progressing. But I did not want there to be any more significance attached to those grades than that.

As a person who is fiercely competitive, and does not like to lose, I knew that many would seek to score the highest grades they could on each piece of evaluated work. My hope was that they could approach the course much the same way I and my cycling colleagues approach a race. During the event, no quarter is taken as we all fight to win. But the moment we have all crossed the finish line, the final result ceases to be important. (Okay, it can last until we hit the bar that evening and begin to exchange embellished personal stories of the event. But definitely no longer than that.)

Unfortunately, what works easily for amateur bike racing, does not seem to work for taking a math course. For all our ultra-light, carbon-fiber racing machines and skin-tight lycra, I and my two-wheeled buddies know we are not professional cyclists and our event is not the Tour de France. We do not approach our races with any expectations carried over from previous experience.

With a math course, unfortunately, people do come along with expectations. Though some students successfully managed to focus on the content and the learning thereof, and were not put off by continually getting grades down in the 30 – 40% range (results that I kept stressing were as good as could be expected for anyone who had not encountered this kind of mathematical activity before), many could not make that shift. For them, many years of high stakes testing had turned mathematics into fierce competition to “get an A”, and anything less was “failure.”

Calling my course “Mathematical *Thinking*” was not enough to counter those expectations, and there was a lot of forum obsessing with grades.

It is, to be sure, a difficult transition to make. (Courses like mine are often called (high school to university) “transition courses.”) To this day, I remember the trauma of going from being an ace at high school, procedural math to being totally lost in my first-year university courses. The main thing that kept me going was the recognition that all my classmates were having the same difficulties. Getting your work back with a 30% at the top is a lot easier to take when everyone sitting in the same room as you is having the same experience – a support mechanism often missing for students in an online course.

The expectations that color how people view course grade-points also affect how they perceive the course certificate. I assumed from the start that many people would attach personal value to the Statement of Accomplishment, even if it has no street value. (On occasion, it appears it does. See the story half way down this article.)

I definitely wanted to make the SoA as meaningful as possible for the person who earns it. My course is difficult, and anyone who completes it should feel proud of what they have done. Accordingly I set the threshold so that approximately 80% of students who completed the course received a SoA, and 20% of those SoAs were with Distinction.

Current platform limitations meant this was not ideal. (The Coursera platform is still under very active development.) Though the instructor was free to specify the algorithm whereby the final course grade was computed from the student grades the system had assigned for each individual piece of work, the only measure the Coursera platform provided on which to make the certificate decisions was that final grade. This meant that some students who did not complete the course, but who scored highly on what they did do, also got certificates. Some of them were not happy to do so.

My response to those was essentially, “Don’t bother to print off the statement then,” though contexed to make it clear I understood why it bothered them.

Still, after enduring for three iterations of the course what for me, given my goals, was a distraction of grades and certificates, at the end of the most recent session I decided to make the award of a SoA in future dependent purely on completing the course.

For students who sign up for my spring session, starting on February 1, SoAs will be awarded for time spent and effort, not level of performance. The grade points awarded for each individual piece of work will cease to have even minimal significance outside the course, not even by way of the SoA.

Students in the spring session will have a choice of two versions of the course. The *Basic Course* will last for eight weeks, and completion leads to a Statement of Accomplishment. The *Extended Course* will continue for a further two weeks, devoted to a process I am calling Test Flight, with completion resulting in the award of a Statement of Accomplishment with Distinction.

The Basic Course is designed to develop mathematically-based analytical thinking skills having wide applicability. The additional learning provided by the Extended Course is focused on applying those skills to mathematics itself, in particular building on the earlier analysis of mathematical proof to establish some basic properties of whole and real numbers.

The Statements of Accomplishment will be awarded on essentially a “Pass/Fail” basis, and the certificate will not state a grade.

Both courses will use grades points purely as a metric of progress, not a record of achievement, nor as a criterion for awarding a SoA.

Not quite. I will use grade points in each individual piece of submitted work to determine what constitutes “completion” of the course. To *guarantee* a SoA, a student will have to submit at least five of the course’s eight, machine-graded Problem Sets, earning at least 5% for those five Sets. That will determine a lower bound for SoAs.

Again, it means that any student who earns a higher overall grade will get a SoA, even if they complete fewer than five Problem Sets, but as long as everyone knows that system, I have no problem with that. Some commentators say that the lack of a guarantee that the person with a SoA really earned it is a weakness of MOOCs, but that’s true only if you view the goal of education as being evaluation and certification — a problem I have with American education in general.

When Coursera develops a more fine-grained framework for determining the awarding of certificates, I will probably modify the process I use, but frankly I do not regard this as a big issue.

Anyone who “cheats” in my course simply cheats themselves, by not getting the benefit of actually learning. I think education will be a lot better if we separate certification from education. I would have no problem with a third party organization coming along and offering an accreditation service for my course. For sure, an equivalent check will already happen if one of my students uses a SoA from my course to secure a job interview at a large company. The first thing the company is likely to do is ask the applicant to take a short test. As long as that test involves mathematical thinking – and any reasonably well designed test surely will – it will become immediately clear if that individual really did benefit from my course.

I am in the education business, not credentialing. If someone simply wants a SoA for my course, the quickest way to get one is to find an image of a certificate on the Web using Google image search, and use graphics processing software to insert their own name. In doing so, they will be demonstrating useful skill with digital media, of course, but I don’t believe that process will provide them with good mathematical thinking skills.

If you *really* want mathematical thinking, you need to actually work through my course, using the grades to measure your progress, and stay for the full ten weeks, completing the Test Flight process at the end.

So what’s Test Flight? Tune in next time. Meanwhile, here is a clue.

]]>