transition course

Posts Tagged 'transition course'

MathThink MOOC v4 – Part 8

Published December 31, 2013 Free online courses 1 Comment
Tags: course content, distance learning, education design, educational platform, higher education, Khan Academy, mathematical thinking, MOOC, online education, Sebastian Thrun, Stanford MOOC, student expectations, transition course, Udacity

In Part 8, I explain why I believe MOOCs cannot and will not lead to cost savings in higher education – at least in a nation that values its standard of living.

As I’ve noted in previous posts to this blog, for the first version of my Introduction to Mathematical Thinking MOOC, I took the first part of a course I had given many times in regular classroom settings, and ported it to a MOOC platform in what I thought was the most sensible way possible. In particular, I changed only things that clearly had to be changed. It was always going to be an iterative process, whereby each time I gave the course I would make changes based on what I had learned from previous attempts.

Given the significant differences between a physical class of 25 entry-qualified students at a selective college or university and a distributed class of 80,000 students around the globe (the size of my first MOOC class in Fall 2012), of widely different educational backgrounds and ability levels, for whom the only entrance criterion was being able to fill in a couple of personal information boxes in a Website, it made sense to maintain – for the first version – as much as possible the contents and structure of the original classroom course. That way, I could focus on the MOOC-specific issues.

After the first session was completed (survived more accurately describes my sensation at the time), all bets would be off, and I would follow where the experience led me. I felt then, and continue to feel now, that there is no reason why a MOOC should resemble anything we are currently familiar with.

I watched as Sebastian Thrun quickly moved Udacity away from his original conception of a highly structured, programmed traditional course – with all that entails – to offering more a smorgasbord of mini-courses, built up from what can be viewed as stand-alone lectures. I asked myself then, and continue to do so, if I should hang on to the central notion of a course, and maybe just tweak it.

So far I have decided I should, the main reason being, as I tried to explain in my last post, the kind of experience I feel best results in the kind of learning I want to provide.

In particular, the primary goal of my course was, and is, to help develop a particular way of thinking – certain habits of mind. That is best achieved, I believe, by focusing on particular “content”.

I used the quotation marks there, because I think it is not accurate to view learning experiences (for experiences are what produce learning) as a certain volume of “content” that is “contained” is some sort of container or vessel. But it seems that everyone else knows what the term (educational) content means – a shared understanding that provides Silicon Valley entrepreneurs with a nice story to raise investment for developing “platforms” to “deliver” that “content” – so I’ll go with it. (I used the word five times in my last post, and no one wrote in to object or say they did not understand what I meant.)

Anther reason for maintaining a course structure (the indefinite article is intentional) is that I want my course to function as a transition course, to help students make the shift from high school to university. And for the foreseeable future, I think universities will continue to carve up “content” into delivery packages called “courses”.

The third reason for having a course is our old friend, student expectations. Many of my full-term students tell me that they signed up because they want a course, with all that entails: commitment, deadlines, testing, and community.

That third reason likely reflects the self-selection implicit in students who sign up for a MOOC, fully 80% of whom (according to recent MOOC research) already have a college degree, and hence are adapted to – and good at – learning that way.

This implies that, by offering a course, I may be reinforcing that emergent trend of primarily providing further college education to individuals who already had one.

That may, in fact, be where MOOCs will end up. For sure, Udacity’s recent pivot appears to reflect Sebastian Thrun’s having decided to direct his (investors’) money toward that audience/market.

If the provision of continuing higher education for college graduates does turn out to be the main benefit that MOOCs provide, that will surely be something for we MOOC developers to be proud of, particularly in a world in which everyone will need to learn and re-tool throughout their lives. (Major innovations rarely land where the innovators thought they would, or do what was originally intended.)

But in that case, MOOCs won’t yield the massive cost savings in first-pass, higher education that many politicians and education-system administrators have been thinking they offer.

In fact – and here I am probably about to bring the wrath of Twitter onto me – I think the current goal of “solving the problem” of the rising cost of higher education by finding ways to reduce it, misunderstands what is going on. I suspect the costs of providing first-pass higher education will continue to rise, because quality higher education is becoming ever more important for life in the Twenty-First Century.

Just as the introduction of the automobile meant society had to adjust to the new – and ever rising – expense of gasoline, so too the shift to knowledge work and the knowledge society means we have to adjust to the cost (high and rising) of a first-pass higher education (the fuel for the knowledge society) that stays in synch with society’s needs.

What MOOCs and other forms of online education have already been shown to be capable of – and it is huge – is provide lifelong educational upgrades at very low cost.

But based on what I and many of my fellow MOOC pioneers have so far discovered – or at least have started to strongly suspect – the initial “firmware” required to facilitate those continual “software” upgrades is not going to get any cheaper. Because the firmware installation is labor intensive and hence not scalable – indeed, for continuously-learning-intensive Twenty-First Century life, not effectively scalable beyond 25-student class-size limits.

The world we have created simply entails those (new and rising) educational costs every bit as much the growth of the automotive society meant accepting the (new and ever-after rising) cost of automotive fuel.

(Oh, and by the way, we in the US need to realize that the knowledge society requires better teacher preparation in the K-12 system as well. Well-educated humans are the new fuel, and they neither grow on trees nor are found underground.)

Okay, that’s enough editorializing for one post. At the end of my last report, I promised to describe how I structure my course so that, while designed primarily to provide a framework for a community learning experience, it can still be useful to folks who want to use it as a resource.

First, what do I mean by “resource”? I decided that for mathematical thinking, it was not possible to produce Khan Academy style “online encyclopedia” materials, where someone can dive in to a single video or narrowly focused educational resource. You simply have to devote more than ten minutes to gain anything of value in what I am focusing on.

So I set my sights on people who come in and complete one or two “Lectures”, a Lecture in my case comprising a single thirty-minute video and some associated problem-solving assignments. So I am not delivering “bite-sized learning.” I am serving up meals. (Restaurant meals, where you have time to savor the food and engage in conversation.)

To facilitate such use, the earlier Lectures focus on everyday human communication, ambiguity resolution, logical reasoning, and very basic mathematical ideas (primarily elementary arithmetic – though in a conceptual way, not calculation, for which we have cheap and efficient machines).

Only in Weeks 7 and 8 do I cover more sophisticated mathematical ideas. (Weeks 9 and 10 comprise my new Test Flight process, which I described in Part 6 of this series. That part is specifically for advanced mathematics seekers.)

Thus, Weeks 1 through 6 can be accessed as a resource by someone not strongly interested in mathematics. At least, that is my current intention.

Admittedly, someone who delves into, say, Week 4 might find they need to go back and start earlier; but that’s true of Khan Academy as well, and is surely unavoidable.

By making the awarding of a Statement of Accomplishment dependent on completion of the Basic Course (first eight weeks), not the achievement of a particular grade, I hope to be able to maintain and reward the participation of someone who begins by just “trying out the course” and gets hooked sufficiently to keep going.

To cater for this dual use as much as possible, in addition to changing the course structure, the upcoming new session has four new videos, and I modified four existing ones. (All the time keeping that magic ingredient “content” the same.)

Well, that’s where I am at present. As I noted earlier, this blog series is essentially my lab book – complete with speculative reflections – made public in real time. (I am already deviating from things I said in this blog just a year ago.)

Ah yes, last time I also promised I would say “what motivated me to give a MOOC in the first place – and still does.” The answer is, “Reaching students who do not currently have access to quality higher education.”

That probably seems very much at odds with everything I’ve said above. It’s not. I’ll explain why in my next post.

MathThink MOOC v4 – Part 7

Published December 26, 2013 Free online courses 2 Comments
Tags: creativity, distance learning, education design, higher education, Ken Robinson, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, student expectations, transition course

In Part 7, I ask myself (yet again) does it need to be a course?

One issue I keep returning to is whether my MOOC should be a course. Or, to put the question a more useful way, what features of a classroom course do I want or need to carry over to a MOOC, what features should I jettison, and what new features should I add?

I raised the issue in my blogpost of August 31, 2012, just before my MOOC launched for the first time. Since then, students’ expectations (as expressed in emails to me and in the discussion forums) have continued to confirm my initial instinct that there are good reasons to carry over a lot of traditional course structure.

Still, the question is not going to go away. I brought it up again in June of 2013 after completing the second version of my MOOC, noting that the majority of my students treated the course as a resource rather than a course.

In those early posts, I made a number of references to Khan Academy, an educational resource I now have very mixed feelings about. (In particular, I think Sal’s enthusiasm and undeniable – and hugely valuable – ability to project his personality through his voice, and thereby to remove much of the fear that many of his followers may have toward mathematics, fall well short of what he could achieve, due to poor pedagogy and way too many elementary – but educationally important – factual mistakes.) I made several key choices based on what could be learned from his endeavors.

One thing I did not do was go the route of turning my MOOC into a collection of Khan-like, standalone, bite-sized snippets. Indeed, deliberately ignoring the current buzz that the audience will drop precipitously if your videos run more than seven minutes, I decided to aim for half-hour chunks. Hey, if thirty-minutes works for Seinfeld and Thirty Rock, why not for Mathematical Thinking? (Remember, I’m looking at a highly selective audience who have voluntarily chosen to enroll in an online math course! I haven’t completely lost it – just enough to keep trying to make this free online course thing work in the first place.)

My decision was largely because the material simply cannot be broken up in that way. Unless you are a mathematical genius, when it comes to mathematical thinking, most of us find that thirty-minute chunks is the absolute minimum time commitment to make any progress at all, and a lot is lost if you cannot arrange for much longer periods. The very last “lecture” of the course actually lasts an hour and a half, with the original video cut up into three segments of roughly equal, thirty-minute lengths. And students who have completed the course say they wished I had spent even more time on the one (capstone) topic I covered in that last lecture.

Since approximately 5,000 students have, on average, stayed with the course to the end each time, I definitely want to continue to provide the learning experience they have clearly been looking for. (In my next post I’ll say how, at the same time, I try to cater for those seeking a resource.)

A significant part of that experience is, I believe, being part of a community, where everyone is working toward the same goal, with regular pressure points (deadlines) that force them to keep sufficiently in lockstep so that they can exchange ideas and express community reactions in real time. Though many of them do not post regularly on the community discussion forums, they do (I assume) follow them, finding answers to their questions and surely being encouraged to learn that they are not alone in finding something particularly difficult or confusing.

That sense of community is, to my mind, an important part of my course. In the (necessarily) simplistic terminology introduced to try to explain the conceptual difference between the original Canadian MOOCs originating from Athabasca University and the unrelated MOOCs coming out of Stanford some years later, my course is a c-MOOC in x-MOOC clothing. (See the Wikipedia article for the tangled history.)

From the very first lecture, I recommend repeatedly that students try to form small learning communities to work through the weekly problem assignments that are the heart of the course.

And there we have another reason why I have not carved my course into bite-sized instructional videos. It’s not about instruction! The expressed goal is not “teaching mathematics” but guiding folks on a process of learning how to think a certain way. In particular, learning how to set about solving a novel problem that perhaps only partially resembles one encountered before.

In other words, in my course the devil is very much not in the details. It’s in the overall flow of ideas, the swirling cloud that hovers above all those details.

The key for making that transition from “template recognizer and applier of known techniques” to “creative problem solver” is to rise above the details and grasp the meta-cognitive aspects of good problem solving.

Having myself made that transition by sitting next to my senior tutor (a professor) in my senior undergraduate year and then my doctoral adviser for the subsequent three years, and watching and listening to them as they worked through problems with me (a very one-sided “with”!), I knew first-hand that the process works. I also know of no other way that does.

It’s a slow process, to be sure. Many students in my regular classes over the years, and far greater numbers of students in my MOOCs, have not been prepared, and in some cases not willing, to adjust to that different pace.

I lost count of the number of MOOC students who expressed frustration (and more) at how slowly I was moving, how I “rambled” and “repeated myself,” and how “unprepared” I had been when I sat down to record those videos.

My approach was, of course, carefully thought out and deliberate. I never intended to give a slick, prepared presentation. (I do many of them, and there are videos all over the Web. But those presentations are about infotainment, not learning to think a different way.)

My approach was always about providing a window into one person’s (mine) thought processes. Not to mimic me. That would make no sense in terms of learning how to think creatively. Rather, to gain sufficient insight to be able to develop that capacity in themselves.

Of course, I can provide just one example – me. But one example is enough. Because the capacity for original thought is in every one of us. It just has to be unleashed.

Evolution by natural selection has made all of us creative problem solvers. That is Homo sapiens’ great survival trick. Unfortunately, an educational system developed in the industrial age to turn innately creative humans into compliant cogs in organizations, suppresses that innate creativity, rewarding fast acquisition and retrieval of facts and rapid execution of procedures, a sad turn of events for today’s world, as summarized brilliantly by the provocative and always entertaining Sir Kenneth Robinson in the animated talk I will leave you with.

Creativity is in all of us. You see it in every small child. Despite systemic education’s efforts to suppress it, it remains eager to break out. (Google dopamine.) It does not take much of a stimulus to make it (start to) happen. A ten week MOOC may seem very short. But it may be enough to initiate the process. (Google “Prison Break”.)

* * *

Next time I’ll describe how I structure the course so that, while designed primarily to provide a framework for a community experience, it can still be useful to folks who want to use it as a resource. I’ll also say what motivated me to give a MOOC in the first place – and still does. Meanwhile, here is Sir Ken:

MathThink MOOC v4 – Part 6

Published December 23, 2013 Free online courses 2 Comments
Tags: Byron Reeves, calibrated peer review, Coursera, distance learning, education design, higher education, mathematical thinking, MOOC, online education, Stanford MOOC, student engagment, student expectations, Test Flight, transition course, transition math, video game learning

In Part 6, I talk about the new Test Flight process.

In the past, when students enrolled for my MOOC, they essentially had three options. One was not to take it as a course at all, but just regard it as a resource to peruse over time or to pick and choose from. A second was to take the entire course, but do so on their own time-scale. Or they could take it as a course, and go through it at the designated pace.

As do many MOOC designers, I tried to make sure my course could be used in all three ways. Though the vast majority of MOOC students fall into the first category, the other two are the ones that require by far the greatest effort by the course designer. They are the learners who have significant ambitions and will put in a lot of effort over several weeks.

The students in the last category will surely gain the most. In particular, they move through the course in lockstep with a cohort of several thousand other students who can all learn from and support one another, as they face each course deadline at the same time. Those students form the core community that is the heart of the course.

When the new class enrolls at the start of February, the ones intending to take an entire course as scheduled will have a new choice. They can take what I am calling the Basic Course, which lasts eight weeks, or the Extended Course, which lasts ten. As I described in my last post, those extra two weeks are devoted to a process I am calling Test Flight.

In the previous two versions of the course, the final weeks nine and ten had been devoted to a Final Exam, one week for completion of the (open book) exam itself, the following week to peer evaluation. In peer evaluation, which started as soon as the class had completed and submitted their exam solutions, each student went through a number of activities:

1. Using a rubric I supplied, each student evaluated three completed examination scripts assembled by me, and then compared their results to mine. (Those three samples were selected by me to highlight particular features of evaluation that typically arise for those problems.)

2. Having thus had some initial practice at evaluation, each student then evaluated three examination scripts submitted by fellow students. (The Coursera platform randomly and anonymously distributed the completed papers.)

3. Each student then evaluated their own completed examination.

This was the system Coursera recommended, and for which they developed their peer evaluation module. (Actually, they suggested that each student evaluated five peer submissions, but at least for my course, that would have put a huge time requirement on the students, so I settled for three.)

Their original goal, and mine, was to provide a means for assigning course grades in a discipline where machine evaluation is not possible. The theory was that, if each student is evaluated by sufficiently many fellow students, each of whom had undergone an initial training period, then the final grade – computed from all the peer grades plus the self-grade – would be fairly reliable, and indeed there is research that supports this assumption. (Certainly, students who evaluate their own work immediately after evaluating that of other students tend to be very objective.)

As far as I could tell, the system worked as intended. If the goal of a MOOC is to take a regular university course and make it widely available on the Internet, then my first three sessions of the course were acceptably successful. But MOOCifying my regular Mathematical Thinking (transition) class was always just my starting point.

Since I was aware from the outset that the MOOC version of my regular classroom course was just a two-dimensional shadow of the real thing, where I interact with my class on a regular basis and give them specific feedback on their work, my intention always was to iteratively develop the MOOC into something that takes maximum advantage of the medium to provide something new of value – whatever that turns out to be.

I expected that, as MOOCs evolve, they would over time come to be structured differently and be used in ways that could be very different from our original design goals. That, after all, is what almost always happens with any new product or technology.

One thing I observed was that, while students often began feeling very nervous about the requirement that they evaluate the work of fellow students, and (justifiably) had significant doubts about being able to do a good job, the majority found the process of evaluating mathematical arguments both enjoyable and a hugely beneficial learning process.

Actually, I need to say a bit more about that “majority” claim. My only regular means of judging the reactions of the class to the various elements of the course was to read the postings on the course discussion forums. I spent at least an hour every day going through those forums, occasionally posting a response of my own, but mostly just reading.

Since the number of regular forum posters is in the hundreds, but the effective (full-term) class was in excess of 5,000 in each of the sessions, forum posters are, by virtue of being forum posters, not representative. Nevertheless, I had to proceed on the assumption that any issue or opinion that was shared (or voted up) by more than one or two forum posters was likely to reflect the views of a significant percentage of the entire (full-term) class.

Since I made gradual changes to the course based on that feedback, this means that over time, my course has been developing in a way that suits the more active forum posters. Arguably that is reasonable, since their level of activity suggests they are the ones most committed, and hence the ones whose needs and preferences the course should try to meet. Still, there are many uncertainties here.

To return to my point about the learning and comprehension benefits evaluators gained from analyzing work of their peers, that did not come as a surprise. I had found that myself when, as a graduate student TA, I first had to evaluate students’ work. I had observed it in my students when I had used it in some of my regular classes. And I had read and heard a number of reports from other instructors who noted the same thing.

It was when I factored the learning benefits of evaluating mathematical arguments in with my ongoing frustration with the degree to which “grade hunting” kept getting in the way of learning, that I finally decided to turn the whole exam part on its head.

While some universities and some instructors may set out to provide credentialing MOOCs, my goal was always to focus on the learning, drawing more on my knowledge of video games and video-game learning (see my blog profkeithdevlin.org) than on my familiarity with university education (see my Stanford homepage).

Most of what I know about giving a university-level course involves significant student-faculty interaction and interpersonal engagement, whereas a well-designed video game maintains the player’s attention and involvement using very different mechanisms. With a MOOC of necessity being absent any significant instructor-student interaction, I felt from the outset that the worlds of television and gaming would provide the key weapons I needed to create and maintain student attention in a MOOC.

[A lot of my understanding of how TV captures the viewer’s attention I learned from my close Stanford colleague, Prof Byron Reeves, who did a lot of the groundbreaking research in that area. He subsequently took his findings on television into the video game business, co-authoring the book Total Engagement: Using Games and Virtual Worlds to Change the Way People Work and Businesses Compete.]

So from the outset of my foray into the world of online education, I was looking to move away from traditional higher-education pedagogic models and structure, and towards what we know about (television and) video games, hopefully ending up with something of value in between.

The idea of awarding a Statement of Accomplishment based on accumulated grade points had to go sooner or later, and along with it the Final Exam. Hence, with Session Four, both will be gone. From now on, it is all about the experience – about trying (and failing!).

The intention for the upcoming session is that a student who completes the Basic Course will have learned enough to be able to make useful, and confident use of mathematical thinking in their work and in their daily lives. Completion of the Test Flight process in the Extended Course will (start to) prepare them for further study in mathematics or a mathematically-dependent discipline – or at least provide enough of a taste of university-level mathematics to help them decide if they want to pursue it further.

At heart, Test Flight is the original Final Exam process, but with a very different purpose, and accordingly structured differently.

As a course culmination activity, building on but separate from the earlier part of the course – and definitely not designed to evaluate what has been learned in the course – Test Flight has its own goal: to provide those taking part with a brief hands-on experience of “life as a mathematician.”

The students are asked to construct mathematical arguments to prove results, and then to evaluate other proofs of the same results. The format is just like the weekly Problem Sets that have met throughout the course, and performance level has no more or less significance.

The evaluation rubric, originally employed to try to guarantee accurate peer grading of the exam, has been modified to guide the evaluator in understanding what factors go into making a good mathematical argument. (I made that change in the previous session.)

After the students have used the rubric to evaluate the three Problem Set solutions supplied by me, they view a video in which I evaluate the same submissions. Not because mine provides the “correct” evaluations. There is usually no single solution to a question and no such thing as the “right” one. Rather, I am providing examples, so they can compare their evaluations with mine.

After that, they then proceed to evaluate three randomly-assigned, anonymously-presented submissions from other students, and finally they evaluate their own submission.

Procedurally, it is essentially the same as the previous Final Exam. But the emphasis has been totally switched from a focus on the person being evaluated (who wants to be evaluated fairly, of course) to the individual doing the evaluation (where striving for a reliable evaluation is a tool to aid learning on the part of the evaluator).

Though I ran a complete trial of the process last time, the course structure was largely unchanged. In particular, there was still a Final Exam for which performance affected the grade, and hence the awarding of a certificate. As a consequence, although I observed enough to give me confidence the Test Flight process could be made to work, there was a square-peg-in-a-round-hole aspect in what I did then that caused some issues.

I am hoping (and expecting) things will go smoother next time. For sure, further adjustments will be required. But overall, I am happy with the way things are developing. I feel the course is moving in the general direction I wanted to go when I set out. I believe I (and the successive generations of students) are slowly getting there. I just don’t know where “there” is exactly, what “there” looks like, and how far in the future we’ll arrive.

As the man said, “To boldly go …”

MathThink MOOC v4 – Part 2

Published December 15, 2013 Free online courses 6 Comments
Tags: education design, failure, mathematical thinking, MOOC, online education, Sebastian Thrun, Stanford MOOC, Steve Jobs, student expectations, Thomas Edison, transition course, Udacity

In Part 2, I reveal that I share with Steve Jobs, J K Rowling, Sebastian Thrun, Thomas Edison, and a successful Finnish video-game studio head, a strong belief in the power of failure.

This post continues the one posted two days ago about the expectations students being to my MOOC.

One of the problematic expectations many students bring to my course is that I will show them how to solve certain kinds of problems, work through a couple of examples, and then ask them to solve one or two similar ones. When I don’t do that, some of them complain, in some cases loudly and repeatedly.

There are several reasons why I do not simply continue to serve up the pureed (instructional) diet they are familiar with, and instead offer them some raw meat to chew on.

Most importantly, the course is not about mastering yet more, specific procedures; rather the goal is to acquire a new way of thinking that can be used whenever a novel situation is encountered. Tautologically, that cannot be “taught.” It has to be learned. The role of the “instructor” is not to instruct, but to offer guidance and feedback – the latter being feasible in a MOOC by virtue of most beginners having broadly similar reactions and making essentially the same mistakes.

To progress in the course, the student has to grow accustomed to the way professional mathematicians (to say nothing of engineers, business leaders, athletes, and the like) make progress: learn by failing. That’s the raw meat I serve up: failure.

Not global failure that debilitates and marks an end to an endeavor; rather repeated local failures that lead to eventual success. (Though the distinction is really one of our attitude toward a failure – I’ll come back to this in a moment.)

Most of us find it difficult making the adjustment to regarding failing as an integral part of learning, in large part because our school system misguidedly penalizes (all) failures and rewards (every little) success.

Yet, it is only when we fail that we actually learn something. The more we fail, the better we learn; the more often we fail, the faster we learn. A person who tries to avoid failure will neither learn nor succeed. If you take a math test and score more than 75%, then you are taking a test that is too easy for you, and hence does not challenge you to learn. A score of 75% or more says you did not need to take the test! You were not pushing the frontiers of your current abilities.

I should add that I am not talking about tests and exams designed to determine what you have learned, rather those that are an integral part of the learning process – which in my case, giving a course that offers no credential, means all the “graded” work.

In my course, the numbers the system throws out after a machine-graded Problem Set, or the mark assigned by peer evaluation, are merely indicators of progress. A grade between 30% and 60% is very solid; above 60% means you are not yet at the threshold where significant (for you) learning will take place, while a score below 30% tells you either that you need to put more time and effort into mastering the material, or slow down, perhaps working through the remainder of the course at your own pace then trying again the next time it is offered. (Another great advantage of a free MOOC.)

What is important is not whether you fail, but what you do as a result. As I was working on this post, I came across an excellent illustration in an article in FastCompany about the Finnish video game studio Supercell. Though the young company has only two titles in the market – Clash of Clans and Hay Day – it grossed $100 million in 2012 and $179 million in the first quarter of 2013 alone.

Supercell’s developers work in autonomous groups of five to seven people. Each cell comes up with its own game ideas. If the team likes it, the rest of the employees get to play. If they like it, the game gets tested in Canada’s iTunes App store. If it’s a hit there it will be deemed ready for global release.

This approach has killed off several games. But here is the kicker: each dead project is celebrated. Employees crack open champagne to toast their failure. “We really want to celebrate maybe not the failure itself but the learning that comes out of the failure,” says Ilkka Paananen, the company’s 34-year-old CEO.

It’s not just in the PISA scores where Finland shows the world it knows a thing or two about learning; you can find it manifested in the App Store download figures as well!

(And let’s not forget that another Finnish game studio, Rovio, produced over a dozen failed games before they hit the global App Store jackpot with Angry Birds.)

Where I live, in Silicon Valley, one of the oft-repeated mantras is, “Fail fast, fail often.” The folks who say that do pretty well in the App Store too. In fact, some of them own the App Store!

One of my main goals in giving my MOOC is helping people get comfortable with failing. You simply cannot be a good mathematical thinker if you are not prepared to fail – frequently and repeatedly. Failing is what professional mathematicians do maybe 99% of the time. Responding appropriately to failure is a key part of mathematical thinking.

And not just mathematical thinking. It’s definitely true of engineering as well. Remember Thomas Edison, who on being asked how he motivated himself to continue his efforts to build an electric light bulb when a thousand attempts had failed, replied (paraphrase), “They were not failures, I just found a thousand ways it won’t work.”

The metaphor I use regularly in my MOOC is learning to ride a bike. If you think about it, you don’t learn to ride a bike; you learn how not to fall off a bike. And you do that by repeatedly falling off until your body figures out how to avoid falling.

Incidentally, the fact that you really did not learn to ride a bike by learning how to is indicated by the fact that almost no one can correctly answer the question, What direction do you turn the handlebars in order for the bike to turn to the right? Your conscious mind, the one that would have been involved if you had learned how to ride a bike, says you twist the handlebars to the right in order to turn the bike to the right. But, if you are able to ride a bike, your body knows better. You turn the handlebars to the left in order to make the bike turn to the right. Your body figured that out when it learned how not to fall down.

Don’t believe me? Go out and try. Make a conscious attempt to turn right by twisting the handlebars to the right. Most likely, your body will prevent you carrying through. But if you manage to over-ride your body’s instinct, you will promptly fall off. So please, do this on grass, not the hard pavement.

Not surprisingly, six weeks in a MOOC is woefully little to adjust to the professionals’ view of failure. The ones who breezed through my course, unfazed by seeing the system return a grade of 30% on a Problem Set, were in most cases, I suspect (and in a fair number of cases that suspicion was confirmed), professional engineers, business people, or others with a fair bit of post-high-school education under their belts. Those for whom the course was one of their first ventures into collegiate education, often had a hard time of it. (Not a few gave up and dropped the course, sometimes leaving an angry, departing post on the class forum page.)

It’s not called a “transition course” for nothing.

I’ll continue this theme of dealing with student expectations in my next post.

Meanwhile, I’ll leave you with three more examples about the power of failing in the learning process.

The first is Steve Jobs’ 2005 commencement address at Stanford.

The second is J. K. Rowling’s 2008 commencement address at Harvard.

Finally, and very close to home, is Sebastian Thrun’s recent business pivot of his MOOC delivery company Udacity, which I discussed in a commentary in the Huffington Post. Though I would agree with the many commentators that his initial attempt had “failed,” where the tone of many was dismissive, I saw just another instance of someone on the pathway to (for him, yet another) success. It’s all about how you view failure and what you do next.

I’ll continue the theme of dealing with student expectations in my next post.

MathThink MOOC v4 – Part 1

Published December 14, 2013 Free online courses 5 Comments
Tags: education design, Jo Boaler, mathematical thinking, MOOC, online education, Stanford MOOC, student expectations, transition course, transition math

In Part 1 of a series, I focus on the distinction between high school math and university-level mathematics, suggesting they are effectively different subjects that are best learned in different ways.

One of the biggest obstacles in giving an online course on mathematical thinking, which my MOOC is, is coping with the expectations students bring to the course – expectations based in large part on their previous experience of mathematics classes. To be sure, prior expectations are often an issue for regular, physical classes. But there the students have an opportunity to interact directly with the instructor on a regular basis. They also have the benefit of a co-present support group of others taking the same class.

But in a massive open online class, apart from locally configured support groups and text-based discussions on the MOOC platform discussion forum, each student is pretty much on her or his own.

The situation is particularly bad for a course like mine, designed to help students transition from high school mathematics to university-level mathematics. For one thing, the two are so different as to be in many ways completely distinct subjects.

School mathematics tends to be almost exclusively procedural, mastering established methods to solve artificially constructed problems designed to be amenable to such an approach. The student who best masters all the techniques in the syllabus and becomes skillful in pattern-matching problems to solution methods, does well. (I know that first hand; it’s how I got to university to study mathematics!)

In contrast, university mathematics is about learning how to deal with a novel situation of a kind you have not encountered before. (If no one else has encountered it, we call it mathematics research.) Though it certainly can involve pattern matching and the application of established, standard procedures, it usually does so only as components of a novel solution you develop to deal with that particular situation. Moreover, at university level, the problems are typically of a “prove that this is true (or false)” variety, rather than “solve this equation” or “compute the value of that formula.”

What is more, while a school math problem typically has a right answer, university mathematics generally involves much more than mere correctness. Indeed, there may not be a unique “right answer.”

Not only is the subject matter different, so too is the pedagogy. Almost all students’ experience of mathematics learning in school is teacher instruction. The teacher describes a method, does a few worked examples, and then asks the students to do a few similar ones. Rinse and repeat.

It’s a very efficient way to cover a lot of ground when the goal is pattern matching and procedure application. It works for school mathematics. Unfortunately, it does not prepare the graduates for the other kind of mathematics. (It also leaves them without ever having a satisfactory answer to their question “What is this good for?”, a question that leaves anyone versed in mathematics astounded. “What is it not good for?” is a more interesting question. It does not have a simple answer, by the way. It’s a very nuanced question.)

It’s like teaching someone the elements of bricklaying, carpentry, plumbing, and electrical wiring, and then asking them to go out and design and build a house. You need all of those skills to build a house, but on their own they are not enough. Not even close.

In deciding, almost two years ago now (before the New York Times had heard of MOOCs) to develop a MOOC to help people learn the other kind of mathematics, what I call mathematical thinking, I knew I was taking on a big challenge. I’d found it hard to teach that kind of course in a physical classroom with just 25, carefully selected students at elite colleges and universities.

On the other hand, most people go through their entire mathematics education without ever encountering what I and my colleagues would call “real mathematics,” and many of them eventually find they need to be able to handle novel situations that involve – or may involve – or could productively be made to involve – mathematical thinking. So I felt there was a need to have a resource publicly available to help them acquire this valuable ability.

The huge dropout rates in MOOCs did not really bother me. For a mathematical thinking course, it’s possible to gain value from dropping into the course for just a few days – and to keep coming back at future times if required. The focus was not on credentialing, it was developing a valuable mental ability – a powerful way of thinking that our ancestors have developed over three thousand years.

That way of thinking can be utilized profitably in many other courses that do yield a certified credential, so students could approach the course as a low-stress, no-risk way of preparing for subsequent learning.

The course is structured as course for those students who seek an encapsulated experience, and in many ways that yields the greatest benefits, in large part because of the interactions with other students working on the same stuff. But the majority of students who have taken it the three times I have offered it have just taken a part of the course.

Each time I gave the course, I changed it, based on what I had learned. When it launches again in February, it will be different again. This time, in some fairly significant ways. In the coming days, I’ll describe those changes and why I made them.

First out of the gate, I’ll describe what exactly were the problems caused by those expectations many students brought to the course, and how did I try to deal with them. Also, what am I changing in the coming version of the course to try to help more people make what is a very difficult transition: from being taught (i.e., instructed) to being able to learn. The reward for making that one transition is huge. It opens up all of mathematics, and in the process makes it much, much easier.

The traditional, instructional way of teaching procedural mathematics frequently leaves students with the impression (dramatically documented by my Stanford colleague Jo Boaler) that mathematics consists of a large number of rules to be learned. But at the risk of sounding like those weird web advertisements (you know, the ones with a drawing or photo of a strange looking person) promising to teach that “one great trick” that will change your life, let me leave you by telling you the one great trick that all mathematicians learn:

You just have to master, once, a particular way of thinking, and you no longer need all those rules.

That’s what my course focuses on. Stay tuned.

Evaluation rubrics: the good, the bad, and the ugly

Published August 13, 2013 Free online courses 6 Comments
Tags: calibrated peer review, education design, evaluation rubric, mathematical thinking, MOOC, online education, transition course, transition math

A real-time chronicle of a seasoned professor just about to launch the third edition of his massively open online course.

With the third session of my MOOC Introduction to Mathematical Thinking starting on September 2, I am busy putting the final touches to the course materials. As I did when I offered the second session earlier this year, I have made some changes to the way the course is structured. The underlying content remains the same, however – indeed at heart it has not changed since I first began teaching a high school to university “transition” course back in the late 1970s, when I was a young university lecturer just starting out on my career.

With the primary focus on helping students develop an new way of thinking, the course was always very light on “content” but high on internal reflection. A typical assignment question might require four or five minutes to write out the answer; but getting to the point where that is possible might take the student several hours of thought, sometimes days. Students who approach the course thinking it is an introductory course on logic – some of whom likely will, as they have in the past, post on the course forum that they cannot understand why I am proceeding so slowly and making such heavy weather of the material – will, if they don’t walk away in disgust, eventually (by about week four) realize they are completely lost. Habituated to courses that rush through a pile of material that required mostly procedural mastery, they find it challenging, and in many cases impossible, to slow down and adopt the questioning, reflective approach this course requires.

My course uses elementary linguistics and formal logic as a vehicle to help develop new thinking skills that are essential for university mathematics majors, very valuable for STEM majors, and of considerable value for anyone who wants to lead a more rewarding life. But it is definitely not a course in linguistics or logic. It is about thinking.

Starting with an analysis of certain features of ordinary language, as I do, provides a starting point that is accessible to everyone – though because the language I examine is English, students for whom that is a second language are at a disadvantage. That is unavoidable. (A Spanish language version, embedded in Hispanic culture, is currently under development. I hope other deep translations follow.)

And formal logic is so simple and structured, and so accessible to a beginner, that it too is well suited to an introductory level course on analytic, and in particular mathematical, thinking.

Why my course videos are longer than most

The imperative of a student devoting substantial periods of time engaged in sustained contemplation of the course material has led to me making two decisions that go against the current grain in MOOCs. First, the pace is slow. I speak far more slowly than I normally do, and I repeat each point at least once, and often more so. Second, I do not break my “lectures” into the now-almost-obligatory no-longer-than-seven-and-ideally-under-three-minutes snippets. For the course’s second running, I did split the later hour or more long videos into half-hour sections, but that was to make it easier for students without fast broadband access, who have to download the videos overnight to watch them.

Of course, students can speed up or slow down the videos, they can watch them as many times as they want, and they can stop and start them to suit their schedules. But then they are in control and make those decisions based on their own progress and understanding. My course does not come pre-digested. It is slow cooking, not fast food.

Learning by evaluation

The main difference returning students will notice in the new session is the much greater emphasis on developing evaluation skills. Fairy early in the course, students will be presented with purported mathematical proofs that they have to evaluate according to a grading rubric.

At first these will be fairly short arguments, designed by me to illustrate various key features of proofs, and often incorporating common mistakes beginners make. Later on, the complexity increases. For those students who elect to take the final exam (and thereby become eligible to earn a Distinction grade for the course), evaluation will culminate in grading three randomly assigned, anonymized exam submissions from fellow students, followed by grading their own submission.

Peer evaluation is essential in MOOCs that involve work that cannot be machine graded, definitely the category into which my Mathematical Thinking course falls. The method I use for the Final Exam is called Calibrated Peer Review. It has a long history and proven acceptable results. (I describe it in some detail on my MOOC course website – accessible to anyone who signs up for the course.) So adopting peer evaluation for my course was unavoidable.

The first time I offered the course, I delayed peer evaluation until the final couple of weeks, when it was restricted to the final exam. Though things went better than I had feared, there were problems. The main issues, which came as no surprise, were, first, that many students felt very uneasy grading the work of others, second, many of them did not do a good job, and third, the rubric (which I had taken off another university’s Internet shelf) did not work at all well.

On the other hand, many students posted forum comments saying they found they enjoyed that part of the course, and learned more in those final two weeks than in the entire earlier part of the course.

I had in fact expected this would be the case, and had told the class early on that many of them would have that reaction. In particular, evaluating the work of fellow students is a very powerful, known way to learn new material. Nevertheless, it came as a great relief when this actually transpired.

As a result of my experience in the first session, when I gave the course a second time this spring, I increased the number of assignment exercises that required students to evaluate purported proofs. I also altered the rubric to make it better suited to what I see as the main points in the course.

The outcome, as far as I could ascertain from reading the comments student posted on the course discussion forum, was that it went much better. But it was still far from perfect. The two main issues were the rubric itself and how to use it.

Designing a rubric

Designing a good rubric is not at all easy for any course, and I think particularly challenging for a course on more advanced parts of mathematics. Qualitative grading of mathematical arguments, like grading essays or works of art, is a holistic skill that takes years to acquire to a degree it can be used to evaluate performance with some degree of reliability. A beginner attempting evaluation needs guidance, most typically provided by an evaluation rubric. The idea is to replace the holistic application of a lifetime’s acquisition of tacit domain knowledge with a number of categories that the evaluator should look for.

The more fine-grained the rubric, the easier it will be for the novice evaluator, but the more onerous the grading task becomes. The rubric I started with for my course had six factors, which I felt was about right – enough to make the task doable for the student yet not too many to turn it into a dull chore. I have retained that number. But, based on the experiences of students using the rubric, I changed several categories the first time I repeated the course and I have changed one category for the upcoming third session.

In each of the six categories in the rubric, the student must chose between three levels, which I name Novice, Apprentice, and Practitioner. I chose the names to emphasize that we are using evaluation as a way to learn, and the focus is to measure progress along a path of development, not assign summative performance judgments of “poor”, “okay”, and “good”.

The intention in having just three levels is to force a student evaluator to make a decision about the work being assessed. But this can be particularly difficult for a beginner who is, of course, lacking in confidence in their ability to do that. To counter that, in this third session, when the student enters the numerical value that course software will use to track progress, the numerical equivalents to those three categories are not 0, 1, 2, but 0, 2, and 4. The student can enter 1 or 3 as a “middle value” if they are undecided as to which category to assign.

Using the rubric

Even with “middling” grades available for the rubric items, most students will find the evaluation process difficult and very time consuming. A rubric simply breaks a single evaluation task into a number of smaller evaluation tasks, six in my case. In so doing, it guides the student as to what things to look for, but the student still has to make qualitative judgments within each of the categories.

To help them make these judgments, the last time I gave the course, I provided them with tutorial videos that take them through the grading process. I record myself grading the same sample arguments that they have just attempted to evaluate, verbalizing my thinking process as I go, explaining why I make the calls I do. They are not the most riveting of videos, and they can be a bit long (ten minutes for some assignment questions). But I don’t know of any other way of conveying something of the expertise I have built up over a lifetime. It is essentially a modern implementation of the age-old apprentice system of acquiring tacit knowledge by working alongside the expert.

Unfortunately, as an expert, I make calls based on important distinctions that for me jump from the student’s page, but are not even remotely apparent to a beginner. The result last time was, for some questions, considerable frustration on the part of the students.

To try to mitigate this problem (I don’t think it can be eliminated), I changed some aspects of the way the rubric is formulated and described, and decided to introduce the entire evaluation notion much earlier in the course. The result is that evaluation is now a very central component of the course. Indeed, evaluating mathematical arguments now plays a role equal to constructing them.

If it goes well – and based on my previous experience with this course, I think it will go better than last time – I will almost certainly adopt a similar approach if and when I give the course in a traditional classroom setting once again. (A heavy travel schedule associated with running a research lab means I have not taught a regular undergraduate class for several years now, though an attractive offer to spend a term at Princeton early next year will give me a much welcomed opportunity to spend some time in the classroom once again.)

Evaluating to learn, not to grade

One feature of a MOOC – or at least a MOOC like mine that does not offer college credit – is that the focus is on learning, not acquiring a credential. Thus, grading can be used entirely for formative purposes, as a guide to progress, not to provide a summative measure of achievement. As an instructor, I find the separation of the teaching and the grading extremely freeing. For one thing, with the assignment of grades out of the picture, the relationship between teacher and student is changed significantly. Also, it means numerical grades can be used as useful indicators of progress. A grade of 35% can be given for a piece of work annotated as “good” (i.e., good for someone taking an introductory course for the first time). The number indicates how much improvement would be required to take the student to the level of an expert practitioner.

To be sure, students who encounter this use of grades for the first time find it takes some getting used to. They are so habituated to the (nonsensical but widespread) notion that anything less than an A is a “failure” that they can be very discouraged when their work earns them a “mere” 35%. But in order to function as a school-to-university transition course, it has to help them adjust to a world where 35% if often a respectable passing grade.

(A student who regularly scores in the 90% range in advanced undergraduate mathematics courses can likely jump straight into a Ph.D. program – and some have done just that. 35% really can be a good result for a beginner.)

One final point about peer evaluation is an issue I encountered last time that surprised me, though perhaps it should not have, given everything I know about a lot of high school mathematics instruction. Many students approached grading the work of others as a punitive process of looking to deduct points. Some went so far as to complain (sometimes angrily) on the discussion forums about my video-streamed grading as being far too lenient.

In fact, one or two even held the view that if a mathematical argument was not logically correct, the only possible grade to give was 0. This particular perspective worried me on two counts.

Firstly, it assumes a degree of logical infallibility that no living mathematician possesses. I doubt there is a single published mathematical proof of more than a few paragraphs that does not include some minor logical slips, and hence is technically incorrect. (Most of the geometric proofs in Euclid’s Elements would score 0 if logical correctness were the sole metric!)

Second, my course is not a mathematics course, it is about mathematical thinking, and has the clearly stated aim of looking at the many different aspects of mathematical arguments required to make them “good.” Logical correctness is just one item on that six-point rubric. As a result, at most 4 of the possible 24 points available can be deducted in an argument is logically incorrect. (Actually, 8 can be deducted, as the final category is “Overall assessment”, designed to encourage precisely what the phrase suggest.)

To be sure, if my course were a mathematics course, I would assign greater weight to logical correctness. As it is, all six categories carry equal weight. But that is deliberate. Most of my students’ entire mathematical education has been in a world where “getting the right answer” is the holy grail. One other objective of transition courses is to break them of that debilitating default assumption.

Finally, and remember, this is for posterity, so be honest. How do you feel?

I’ve written elsewhere that I think MOOCs as such will not be the cause of a revolution in higher education. Rather they are just part of what is more like to be an evolution, though a major one to be sure. From the point of view of an instructor, though, they are providing us with a wonderful domain to re-examine all of our assumptions about how to teach and how students learn. As you can surely tell, I continue to have a blast in the MOOCasphere.

To be continued …

Answering the 64,000-Students Questions

Published October 26, 2012 Free online courses 14 Comments
Tags: Big Data, education design, mathematical thinking, MOOC, online education, Paul Lockhart, Stanford Graduate School of Education, Stanford MOOC, transition course

A real-time chronicle of a seasoned professor who has just completed giving his first massively open online course.

With the “instructional” part of the course finished and the remaining students working on the Final Exam (it will be peer graded next week), at last I can sit back and take a short breather. The next step will be to debrief and reflect with my two course assistants (both PhD students in the Stanford Graduate School of Education) and decide where to ride the MOOC beast next.

For sure I’ll offer another version of this course next year, with changes based on the huge amounts of data you get with a global online class of 64,000 students. Despite the enormous effort in designing, preparing, and running such a massive enterprise, there are three very good reasons to pursue this.

First, and this I believe is one of the main reasons why Stanford is supporting the development of MOOCs (I am not part of the central, policy-making administration), designing, running, and analyzing the learning outcomes of MOOCs is a tremendous research opportunity that will almost certainly result in new understandings of how people learn, and as a result very likely will enable the university to improve the learning experience of our regular on-campus students. After just five weeks, my two graduate assistants have enough data to write several dissertations, in addition to the one they need to get their doctorates.

Second, there is a huge, overall, feel-good factor for those of us involved, knowing that we can help to provide life-changing opportunities for people around the world who would otherwise have no access to quality higher education. Is what they get as good as being at Stanford? I very much doubt it, though the scientist in me says we should keep an open mind into the eventual outcomes of what is at present a very novel phenomenon. But if you compare a Stanford MOOC with the alternative of nothing at all, then already you have an excellent reason to continue.

Third, and this is something that anyone in education will acknowledge makes up for our earning a much lower salary than our (often less formally qualified) friends in the business and financial worlds, there is the pleasure of hearing first-hand from some of our more satisfied customers. The following is one of many appreciative emails and forum posts I have received as my course came to and end:

Mr. Devlin and all members of the Introduction To Mathematical Thinking team, I just wanted to say Thank You for everything that you have done to share your knowledge and giving your time and great effort to help others learn. I imagine that this is not an easy project to lead and sustain on a continuous basis. However, you have done a wonderful job in relaying your message. Through your efforts, you have helped many people in the process; especially me. Until this class, I hated math. I hated the idea of learning math or thinking in mathematically analogous methods that are applicable to real world situations. I just didn’t get it. I’m still a little confused about why I am able to comprehend your lessons as effectively as I am (which is saying a lot considering how much I hated math) when I have not been able to do so in the past. Now, I find myself looking forward to your classes everyday! I look forward to using what I have learned from the last video lectures or assignments and using those lessons in situations I did not think possible. And now, I love math! Your instruction has helped me to think more logically and to draw more concise conclusions with issues that I am trying to handle. This is indeed a skill. This is also a skill that you can build upon throughout your lifetime if one chooses to do so. Though I may not be at the level of learning that I should be at, I have learned more in the past three weeks than I have learned throughout my life; and I will continue to learn. I am very serious about this statement. So, thank you All. Thank you, Mr. Devlin. Great Job and Cheers!

Nice!

To be sure, there were trolls on the course discussion forum, for whom nothing we did was right. But one of the benefits of having tens of thousand of students is that within at most an hour of a flame post appearing, tens of others jumped on the offending individual, and within a short while all that was left was a “This comment has been deleted” notice. As the course wore on, the trolls simply dropped away.

Though there was the one individual who, in week four, posted a comment that he hated my teaching style and was learning nothing. Given that this was a free course that no one was under any compulsion to take, and for which no official credential was awarded, one wonders why this person stuck around for so long!

That example provided no more than an amusing anecdote to tell when I start to give talks on “What’s it like to teach 64,000 students?” (Invitations are already coming in.) But there is a somewhat closely related issue that I find far more significant.

Like almost all current MOOCs, there was no real credentialing in my course, so the focus was entirely on learning for its own sake. (As a lifelong math professor, used to teaching classes where many of the students were there because they needed to fulfill a mathematics requirement, having a class of students who were there purely voluntarily added appeal to my giving a MOOC.) To be sure, there were in-lecture quizzes, machine-graded assignments, and a peer evaluated final exam, but the only people who had access to any student’s results were myself, my two course assistants, and the student. Moreover, there was no official certification to back up a good result (the course offered two levels, Completion and Completion with Distinction), and turn it into a form of credential.

Yet many students had an ongoing obsession with their grades, and indeed pleaded with me from time to time to re-grade their work. (Clearly not possible in a 64,000 student MOOC. Besides, I never saw their work. How could I?) As a competitive person myself, I can appreciate the desire to do well. But with literally nothing at stake, I was at first surprised by the degree to which it bothered some of them. When I figured out what was probably going on, I found something that bothered me.

Unlike most MOOCs, mine, being at first-year university level, can be taken by high school students. Indeed, since my primary target audience comprised students entering or about to enter university to study mathematics or a math-related subject, I expected to get high school seniors, and designed my course as much as possible to accommodate them.

I’m guessing that the majority of students who were obsessed with grades were still at high school – indeed, most likely a US high school. That grade obsession I observed is, I suspect, simply a learned behavior that reflects the way our K-12 system turns the learning of a fascinating subject – one of humankind’s most amazing, creative, intellectual achievements – into a seemingly endless sequence of bite-sized pieces that are fed to the student in a mandated hamster-wheel.

No wonder they could not relax and enjoy learning for its own sake. Any natural curiosity and desire to learn – something all humans are born with – had been driven out of them by the very institution that is supposed to encourage and develop that trait. In its place was mere grade hunting.

Do I know this for a fact? No. That’s why I used those hedging words “guess” and “suspect”. But something has to explain that grade obsession in my course, and it certainly brought to mind Paul Lockhart’s wonderful essay A Mathematician’s Lament, which I had the privilege to bring to a wider audience some years ago.

But now I digress. Time to wrap up and check the dashboard on the course website see how many students have submitted the Final Exam so far.

Though this post has dropped the title “MOOC Planning”, I am going to keep posting here, as the project goes forward. Stay tuned.

To be continued …

Final Lecture: MOOC Planning – Part 9

Published October 20, 2012 Free online courses 8 Comments
Tags: calibrated peer review, Coursera, Khan Academy, mathematical thinking, MOOC, online education, Stanford MOOC, transition course, transition math, WYSIWOSG

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I gave my last lecture of the course yesterday (discounting the tutorial session that will go out next week), and we are now starting a two week exam period.

“Giving” a lecture means the video becomes available for streaming. For logistic reasons (high among them, my survival and continued sanity — assuming anyone who organizes and gives a MOOC, for no payment, is sane), I recorded all the lectures weeks ago, well before the course started. The weekly tutorial sessions come the closest to being live. I record them one or two days before posting, so I can use them to respond to issues raised in the online course discussion forum.

The initial course enrollment of 63,649 has dropped to 11,848 individuals that the platform says are still active on the site. At around 20%, that’s pretty high by current MOOC standards, though I don’t know whether that is something to be pleased about, since it’s not at all clear what the right definition of “success” is for a MOOC.

Some might argue that 20% completion indicates that the standards are too low. I don’t think that’s true for my course. Completion does, after all, simply mean that a student is still engaged. The degree to which they have mastered the material is unclear. So having 80% drop out could mean the standard is too high.

In my case, I did not set out to achieve any particular completion rate; rather I adopted a WYSIWOSG approach — “What You See Is What Our Students Get.” I offered a MOOC that is essentially the first half of a ten week course I’ve given at many universities over the years, including Stanford. That meant my students would experience a Stanford-level course. But they would not be subject to passing a Stanford-level exam.

In fact, I could not offer anything close to a Stanford-exam experience. There is a Final Exam, and it has some challenging questions, but it is not taken under controlled, supervised conditions. Moreover, since it involves constructing proofs, it cannot be machine graded, and thus has to be graded by other students, using a crowd sourcing method (Calibrated Peer Review). That put a significant limitation on the kinds of exam questions I could ask. On top of that, the grading is done by as many different people as there are students, and I assume most of them are not expert mathematicians. As a result, it’s at most a “better-than-nothing” solution. Would any of us want to be treated by a doctor whose final exam had been peer graded (only) by fellow students, even if the exam and the grading had been carried out under strictly controlled conditions?

On the other hand, looking at and attempting to evaluate the work of fellow students is a powerful learning experience, so if you view MOOCs as vehicles for learning, rather than a route to a qualification, then peer evaluation has a lot to be said for it. Traditional universities offer both learning and qualifications. MOOCs currently provide the former. Whether they eventually offer the latter as well remains to be seen. There are certainly ways it can be done, and that may be one way that MOOCs will make money. (Udacity already does offer a credentialing option, for a fee.)

In designing my course, I tried to optimize for learning in small groups, perhaps five to fifteen at a time. The goal was to build learning communities, within which students could help one another. Since there is no possibility of regular, direct interaction with the instructor (me) and my one TA (Paul), students have to seek help from fellow students. There is no other way. But, on its own, group work is not enough. Learning how to think mathematically (the focus of my course) requires feedback from others, but it needs to include feedback from people already expert in mathematical thinking. This means that, in order to truly succeed, not only do students need to work in groups (at least part of the time), and subject their attempts to the scrutiny of others, some of those interactions have to be with experts.

One original idea I had turned out not to work, though whether through the idea itself being flawed or the naive way we implemented it is not clear to me. That was to ask students at the start of the course to register if they had sufficient knowledge and experience with the course material to act as “Community TAs”, and be so designated in the discussion forums. Though over 600 signed up to play that role, many soon found they did not have sufficient knowledge to perform the task. Fortunately,a relatively small number of sign-ups did have the necessary background, as well as the interpersonal skills to give advice in a supporting, non-threatening way, and they more or less ensured that the forum discussions met the needs of many students (or so it seems).

Another idea was to assign students to study groups, and use an initial survey to try to identify those with some background knowledge and seed them into the groups. Unfortunately, Coursera does not (yet) have functionality to support the creation and running of groups, apart from the creation of forum threads. So instead, in my first lecture, I suggested to the students that they form their own study groups in whatever way they could.

The first place to do that was, of course, the discussion forums on the course website, which very soon listed several pages of groups. Some used the discussion forum itself to work together, while others migrated offsite to some other location, physical or virtual, with Skype seeming a common medium. Shortly after the course launched, several students discovered GetStudyRoom, a virtual meeting place dedicated to MOOCs, built by a small startup company.

In any event, students quickly found their own solutions. But with students forming groups in so many different ways on different media, there was no way to track how many remained active or how successful they have been.

The study groups listed on the course website show a wide variety of criteria used to bring the groups together. Nationality and location were popular, with groups such as Brazil Study Group, Grupo de Estudo Português, All Students From Asia, and Study Group for Students Located in Karachi, Pakistan. Then there were groups with a more specific focus, such as Musicians, Parents of Homeschooled Children, Older/Retired English Speakers Discussion for Assignment 1, and, two of my favorites, After 8pm (UK time) English speakers with a day job and the delightfully named Just Hanging on Study Group.

The forum has seen a lot of activity: 15,088 posts and 13,622 comments, spread across 2712 different threads, viewed 430,769 times. Though I have been monitoring the forums on an almost daily basis, to maintain an overall sense of how the course is going, it’s clearly not possible to view everything. For the most part I restricted my attention to the posts that garnered a number of up-votes. Students vote posts up and down, and once a post shows 5 or more up-votes, I take that as an indication that the issue may be worth looking at.

The thread with the highest number of up-votes (165) was titled Deadlines way too short. Clearly, the question of deadlines was a hot topic. How, if at all, to respond to such feedback is no easy matter. In a course with tens of thousands of students, even a post with hundreds of up-votes represents just a tiny fraction of the class. Moreover, threads typically include opinions on both sides of an issue.

For instance, in threads about the pace of the course, some students complained that they did not have enough time to complete assignments, and pleaded for more relaxed deadlines, whereas others said they thrived on the pace, which stimulated them to keep on top of the material. For many, an ivy-league MOOC offers the first opportunity to experience an elite university course, and I think some are surprised at the level and pace. (I fact, I did keep the pace down for the first three weeks, but I also do that when I give a transition course in a regular setting, since I know how difficult it is to make that transition from high school math to university level mathematics.)

A common suggestion/request was to simply post the course materials online and let students access them according to their own schedules, much like Khan Academy. This raises a lot of issues about the nature of learning and the role MOOCs can (might? should?) play. But this blog post has already gone on long enough, so I’ll take up that issue next time.

To be continued …

The Crucible: MOOC Planning – Part 8

Published October 6, 2012 Free online courses 23 Comments
Tags: Big Data, Coursera, higher education, mathematical thinking, MOOC, online education, Stanford School of Education, transition course, transition math

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

Well, I have survived the initial three weeks of my first MOOC. Though the bulk of the work (and I mean “bulk”) came before the course launched, it has still taken my TA and me a lot of time to keep things ticking over. There are the in-flight corrections of the inevitable errors that occur in a new course, together with the challenges presented by a completely new medium and a buggy, beta release platform, still under very rapid development.

The course website shows 61,846 registered students, but I suspect many of those have long stopped any kind of connection to the course, and another large group are simply watching the lecture videos. The really pleasing figure is that the number of active users last week (week 3) was 19,298. Based on what I hear about other MOOCs, retaining one student in three is a good number.

Both my hands-on TA, Paul, and the course Research Associate, Molly, are graduate students in Stanford’s School of Education, and besides helping me with aspects of the course design, they are approaching the project as an opportunity to carry out research in learning, particularly mathematics learning. Given the massive amount of data a MOOC generates, the education research world can expect to see a series of papers coming from them in the months ahead.

I’m not trained in education research, but some observations are self-evident when you look over the course discussion forums – something I’ve spent a lot of time doing, both to gauge how the course is going and to look for ways to improve it, either by an in-course modification of for a future iteration of the course.

I’ve always felt that the essence of MOOC learning is community building. There is no hope that the “instructor” can do more than orchestrate events. Without regular close contact with the students, the video-recorded lectures and the various course notes and handouts are like firing off a shotgun on a misty Scottish moor. The shot flies out and disperses into the mist, and you just hope some of it hits a target. (I haven’t actually fired a shotgun on a Scottish moor, or anywhere else for that matter, but I’ve seen it on TV and it seems the right metaphor.) With 60,000 (or 20,000) students, I can’t allow myself to respond to a forum post or an email from any single student. I have to rely on the voting procedure (“Like/Dislike”) of the forums to help me decide which questions to address.

This means the student body has to resolve things among themselves. It was fascinating watching the activity on the discussion forums take shape and develop a profile over the first couple of weeks.

One huge benefit for the instructor is the virtual elimination of the potentially disruptive influence – present in almost any class with more than twenty or so students – of the small number of students for whom nothing is good enough. Even in a totally free course, put on by volunteers, for which no college credential is awarded, there were a few early posts of that kind. But in each case the individual was rapidly put in his or her place by replies from other students, and before long stopped posting, and very likely dropped the course.

(An interesting feature of this was that each time it occurred, a number of students emailed me in private – rather than on the public course forum – to say they did not agree with the complainer, and to tell me they were enjoying the course. Clearly, even with the possibility of anonymous forum posts, which Coursera allows, at least for now, some people prefer to keep their communication totally private.)

Of far greater interest, at least to me, was how the student body rapidly split into two camps, based on how they reacted to the course content. As I’ve discussed in earlier posts to this blog, my course is a high-school to university transition course for mathematics. It’s designed to help students make the difficult (and for most of us psychologically challenging) transition from high school mathematics, with its emphasis on learning to follow procedures to solve highly contrived “math problems”, to developing an ability to think logically, numerically, analytically, quantitatively, and algebraically (i.e., in aggregate, mathematically) about novel problems, including often ill-defined or ambiguous real-world problems.

When I give this kind of course to a traditional class of twenty-five or so entering college students, fresh out of high school, the vast majority of them have a really hard time with it. In my MOOC, in contrast, the student body has individuals of all ages, from late teens into their sixties and seventies, with different backgrounds and experiences, and many of them said they found this approach the most stimulating mathematics class they had ever taken. They loved grappling with the inherent ambiguity and open-ended nature of some of the problems.

Our schools (at least in the US), by focusing on one particular aspect of mathematics – the formal, procedural – I think badly shortchange our students. They send them into the world with a fine scalpel, but life in that world requires a fairly diverse toolkit – including WD40 and a large roll of duct tape.

The real world rarely presents us with neat, encapsulated problems that can be solved in ten minutes. Real world problems are messy, ambiguous, ill-defined, and often with internal contradictions. Yes, precise, formal mathematics can be very useful in helping to solve such problems. But of far broader applicability is what I have been calling “mathematical thinking”, the title of my course.

I suspect the students who seemed to take to my course like ducks to water were people well beyond high school, who had discovered for themselves what is involved in solving real problems. Judging by the forum discussions, they are having a blast.

The others, the ones whose experience of mathematics has, I suspect, been almost entirely the familiar, procedural-skills learning of the traditional K-12 math curriculum, keep searching for precision that simply is not there, or (and I’ve been focusing a lot on this in the first three weeks) where the goal is to learn how to develop that precision in the first place.

The process of starting with a messy, real world problem, where we have little more than our intuitions to guide us, and then slowly distilling some precision to help us deal with that problem, is hugely valuable. Indeed, it is the engine that powered (and continues to power) the entire development of our science and our technology. Yet, in our K-12 system we hardly ever help students to learn how to do that.

Done well, the activities of the traditional math class can be great fun. I certainly found it so, and have spent a large part of my life enjoying the challenges of pure mathematics research. But a lot of that fun comes from working within the precise definitions and clear rules of engagement of the discipline. To me mathematics was chess on steroids. I loved it. Still do, for that matter. But relatively few citizens are interested in making a career in mathematics. An education system that derives its goals from the ivory-towered pursuit of pure mathematics (and I use that phrase in an absolutely non-denigrating way, knowing full well how important it is to society and to our culture that those ivory towers exist) does not well serve the majority of students.

It requires some experience and sophistication in mathematics to see how skill in abstract, pure reasoning plays an important role in dealing with the more messy issues of the real world. There is an onus on those of us in the math ed community to help others to appreciate the benefits available to them by way of improved mathematical ability.

As I have followed the forum discussions in my MOOC, I have started to wonder if one thing that MOOCs can give to mathematics higher education in spades is a mechanism to provide a real bridge between K-12 education and life in the world that follows. By coming together in a large, albeit virtual community, the precision-seeking individuals who want clear rules and guidelines to follow find themselves side-by-side (actually, keyboard-to-keyboard) with others (perhaps with weak formal mathematics skills) more used to approaching open-ended, novel problems of the kind the real world throws up all the time. If so, that would make the MOOC a powerful crucible that would benefit both groups, and thus society at large.

To be continued …

The Challenges of Online Education: MOOC planning – Part 1

Published July 17, 2012 Free online courses 10 Comments
Tags: Coursera, flipped classroom, mathematical thinking, Stanford MOOC, textbooks, transition course

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I’ve been pretty quiet on this blog since launching it on May 5.

Partly that is due to summer vacation and the start of great cycling weather. But a lot of my time got swallowed up planning and developing my fall MOOC. It’s now scheduled to start on September 17, and the registration page just went live on Coursera, the Stanford spin-off MOOC platform now offering online courses from a number of the nation’s best universities.

All my Stanford colleagues who gave courses in the first round earlier this year reported how much time it takes to create such a course, no matter how long you have been teaching at university level. Knowing that you won’t be in the same room as the students, where there is ongoing interaction and constant, instant feedback, means that the entire course has to be planned down to the finest detail, before the first day. In addition to the usual course planning, lectures have to be recorded written materials prepared, and interactive quizzes constructed well in advance, with the knowledge that for some students, you may be their only connection to the material.

In my case, my fall term was already pretty full, before counting the MOOC, so I knew I could not rely on having the opportunity to record material once the course begins. That meant I had to try to anticipate well before the course launch, the difficulties the students might have.

Of course, I would not have chosen my topic (introduction to mathematical thinking) if I had not taught it many times before. Many colleges and universities ask their incoming mathematics students to take a “transition course” to develop the all-important skill of mathematical thinking. I helped pioneer such courses back in the 1970s. So I did start out with a good idea of the kinds of difficulties students would encounter on meeting the material for the first time.

But the challenges I faced (and still face) in trying to provide such a course in a MOOC format were, and are, formidable. To be honest, I am not sure it can really be done, but the only way to find out is to try – and not just once either. (Like the Coursera platform itself, my fall MOOC will be very much a beta release.)

An obvious problem is that learning to think like a mathematician, which is what transition courses are about, is not something that can be achieved by instruction. In that respect, the learning process is similar to learning to ride a bicycle. There is no avoiding a lengthy, and often painful process of trying and failing (i.e., falling) until, one day everything drops into place and you find you can ride. At that point, you wonder why it took you so long. Instruction helps, though only in retrospect can you see how. During the learning period, riding seemed impossible – something others could miraculously do but that you were not capable of.

(As someone who came to serious road biking and mountain biking later in life, I can recall vividly that the same is true for “advanced cycling.” For instance, being instructed – many times – how to corner fast on a downhill did not prevent me having to go through a lengthy process of learning how to do it. And while the broken collarbone I sustained in the process was a result of a rear-tire blowout on a sharp corner descending Mt Hamilton outside San Jose, California, it is possible that with more experience I could have kept control. But I am getting off track, which is what happened on Mt Hamilton as well.)

The challenge facing anyone trying to help students learn how to think mathematically by way of a MOOC, is that the communication channel is one way, from the instructor to the student. The sheer number of students (likely into the thousands) prevents any reliance on even the highly impoverished forms of student-faculty interaction that are possible with distance education for a class of no more than thirty students.

The only option (at least the only one I could see) is to try to create an environment where the students can help one another, by forming small study-groups and working together. In particular, I felt the students in my transition mathematics MOOC would benefit greatly by having regular transition course instructors use my MOOC in a flipped classroom model, so that my MOOC students working alone would be able to interact with other MOOC students who in turn were interacting in-person with a professor in a regular class, and perhaps on occasion interact directly with one of those professors online.

This is why I decided to offer my MOOC at the same time (the start of the US academic year) as many US colleges and universities offer their own transition courses. If instructors of those courses get their students to take my MOOC as part of their own learning process, their participation in study groups and the online discussion forums could ensure that every student in the MOOC is at most just one step removed from an expert. For the students in regular transition courses, using my MOOC in a flipped classroom experience, there is the added benefit that we all learn very efficiently when we try to teach others.

Another advantage of trying to involve instructors and students from regular transition classes, is that those instructors could critique my teaching in their class. Contrary to popular belief, “experts” are not infallible beings who know everything. They are just regular people who have more experience in a particular domain than most others. Analyzing and critiquing expert performance is another powerful way to learn. (So feel free to tear me apart. I can take it; I brought up two daughters through childhood and adolescence to adulthood, and after that I was a department chair and then a dean.)

To make my course attractive to regular transition course instructors, I had to make it very short, and focus on the very core of such courses, so those instructors would have plenty of time to take their own courses in whatever direction they choose.

Once I made that decision, I decided to write a companion book for the course. My Stanford colleagues who were giving the first MOOCs reported that some students wanted a physical book to read to support the online learning. People learn in different ways, and we instructors should accommodate them as much as possible.

There are many transition mathematics textbooks on the market, but they are all fairly pricey (ranging from $60 to $140) and cover much more ground than was possible in a mere five weeks of MOOC instruction. Definitely outside the spirit of free learning for all. I decided to write a companion book rather than a textbook (insofar as there is a distinction), since my view is that MOOCs are actually twenty-first century replacements of textbooks.

(I don’t think there is any chance that MOOCs can effectively replace regular university education, by the way, and a school district, state, or nation that decides to go that route will be just a single generation away from becoming a new third-world economy. But if I were a major textbook publisher, I would see MOOCs as the impending end of that business.)

To remain close to the ideal of free education, I decided to make my text a cheap, print-on-demand book. I typeset it myself in LaTeX, paid for an experienced mathematics textbook editor to edit the manuscript, and sent it off as a PDF file to Amazon’s self-publishing CreateSpace service to turn it into a book that can be ordered from Amazon. It’s called Introduction to Mathematical Thinking, and it should be available by August 1. It costs $9.99 and comes in at 102 pages. (There is no e-book option. Given the necessity of mathematical typesetting, an acceptable e-book not possible – at least for e-books that can display on any e-reader. Besides, as I mentioned already, to my mind, the MOOC itself is the true digital equivalent of a textbook.)

Incidentally, the process of self publication on CreateSpace is so simple and efficient, I suspect that low-cost, print-on-demand publishing is the future of academic textbooks.

So add writing a book to the other tasks involved in creating a MOOC.

Still, the book-writing part was easy. Though many of my colleagues find writing books a major challenge – an insurmountable challenge for some of them – I have always found it relatively painless, indeed pleasurable.

In any event, books are an ancient medium that academics and teachers have long been familiar with. Pretty well everything else about the MOOC process was new. I wrote the book before I designed the course; indeed, the book constituted the curriculum. The only new twist for me was that in writing the book I was conscious of using it as the basis for a MOOC.

With the book written, the next question was, how do I present the lectures? After experimenting with a number of formats, I finally settled on the one I’ll use this fall. It’s not the one Sal Khan uses for Khan Academy. Given his success, I started out trying his format, but I found it just did not work for the kind of material I was dealing with. I’ll say more in my next posting. There were other surprises ahead as well.

To be continued …

MOOCtalk

Posts Tagged 'transition course'

MathThink MOOC v4 – Part 8

MathThink MOOC v4 – Part 7

MathThink MOOC v4 – Part 6

MathThink MOOC v4 – Part 2

MathThink MOOC v4 – Part 1

Evaluation rubrics: the good, the bad, and the ugly

Answering the 64,000-Students Questions

Final Lecture: MOOC Planning – Part 9

The Crucible: MOOC Planning – Part 8

The Challenges of Online Education: MOOC planning – Part 1

profkeithdevlin

Devlin’s Angle (MAA)

Twitter Updates

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011

Posts Tagged 'transition course'

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Archives

New Book 2012

New book 2011

New e-book 2011

New book 2011