Posts Tagged 'Khan Academy'



It’s About Time (in Part): MOOC Planning – Part 10

 A real-time chronicle of a seasoned professor embarking on his first massively open online course.

Well, lectures have ended and the course has now switched gears. For those still left in the course (17% of the final enrollment total of 64,045), the next two weeks are focused on trying to make sense of everything they have learned, and working on the final exam — which in the case of my course involves peer evaluation.

Calibrated Peer Review is not new. A study of its use in the high school system by Sadler and Good, published in 2006, has become compulsory reading for those of us planning and giving MOOCs that cover material that cannot be machine graded. [If you want to see how I am using it, just enroll in the class and read the description of the “Peer Review system”. There is no obligation to do anything more than browse around the site! No one will know you are not simply a dog that can use a computer.]

As I was working on my course, Coursera was still frantically building out their platform to support peer evaluation. There was a lot of just-in-time construction. It’s been a long time since I’ve had to go behind a user-friendly interface and dig into the underlying code to do something on a computer, and the programming languages have all changed since I last did that.

One thing I had to learn was one of the ways networked computers keep time. I now know that at the time of writing these words, 7:00AM Pacific Daylight Time on October 22, 2012,  exactly 1,350,914,400 seconds have elapsed since the first second of January 1st, 1970, Eastern Standard Time. That was the start of Unix Time.

I needed to learn to work in Unix Time in order to set the various opening times and completion deadlines for the exam process. I expect that by the time the next instructor puts together a MOOC, she or he will be greeted by a nice, friendly Coursera interface with pulldown menus and boxes to tick — which probably will come as a great relief to any humanities professors reading this, who don’t have any programming in their background.

[By coincidence, Unix was the last programming language I had any proficiency in, but I did not need to know Unix to use Unix Time. I just used an online converter. Unix was developed in 1969 at AT&T Bell Laboratories in New Jersey. Hence the 1970 EST baseline.]

In fact, time conversion issues in general turned out to be a  continuing, major headache in a course with students all over the world. One thing we will not do again is have 12:00PM Stanford Time, aka Coursera Time (i.e., PDT), as any of the course deadlines. It might seem a nice clean stopping point, and there are all those memories of Gary Cooper’s deadline in the classic Western movie High Noon, but many students missed the deadline for the first submitted assignment because they thought 12:00PM meant midnight, which in some parts of the world made them a whole day late.

The arbitrary illogicality of the AM/PM distinction is not apparent to those of us who grew up with it. But my course TA and I are now very aware of the problems it can lead to! In future, we’ll stick to unambiguous times that stay away from noon and midnight. But even then, with local computer systems usually working on local time, to say nothing of the different Summer and Winter Times, which change on different dates around the world, timing events in MOOCs is going to remain a problematic issue, just as it is for international travelers and professionals who collaborate globally over Skype and other conferencing services. (When I used the Unix Time conversion app, I had to remember that Unix thinks New Jersey is currently just two hours ahead of California, not the three hours United Airlines uses when it flies me there. Confusing, isn’t it?)

The reason why times are an issue in my course is that it is a course. At first glance, it may look little different from Khan Academy, where there are no time issues at all. But Khan Academy is really just an educational resource. (At least, that’s the part most people are familiar with and use, namely the video library that started it all. People use it as a video version of a textbook — or more precisely a video equivalent to that good old standby Cliffs Notes, which got many of us through an exam in an obligatory subject we were not particularly interested in.)

In contrast, in my case, as I’ve discussed earlier in this blog series (in particular, Part 6), my goal was to take a standard university course (one I’ve given many times over the years, at different universities, including Stanford) and make it available to anyone in the world, for free. To the degree I could make it happen, they would get the same learning experience.

That meant that the main goal would be to build a (short-lived) learning community. The video-recorded lectures and tutorials were simply tools to make that happen and to orchestrate events. Real learning takes place when students work on assignments on their own, when they repeatedly fail to solve a problem, and when they interact (with the professor and with one another) — not when they watch a lecture or read a book.

To achieve that goal, the MOOC would, as I stated in Part 6, involve admissions, lectures, peer interaction, professor interaction, problem-solving, assignments, exams, deadlines, and certification. To use the mnemonic I coined early on in this series, the basic design principle is WYSIWOSG: What You See Is What Our Students Get.

As we go forward, I intend to iterate on the course design, based on the data we collect from the students (and 64,000 students very definitely puts us into the Big Data realm). But my basic principle will remain that of offering a course, not the provision of a video library. And the reason for that should be obvious to anyone who has been following this blog series, as well as some of the posts on my other blogs Devlin’s Angle and profkeithdevlin.org. The focus is not on acquiring facts or mastering basic skills, but on learning to think a certain way (in my case, like a professional mathematician). And that requires both a lot of effort and (for most of us) a lot of interaction with others trying to achieve the same goal.

Our ancestors in the 11th Century started to develop what to this day remains the best way we know to achieve this at scale: the university, where people become members of a learning community in which learning takes place in a hothouse atmosphere that involves periods of intense interaction as deadlines loom, sustained by the rapidly formed social bonds that emerge as a result of that same pressure.

While I will likely experiment with variants of this model that allow for participation by students who have demanding, full-time jobs, I doubt I will abandon that basic model. It has lasted for a thousand years for a good reason. It works.

To be continued …

Advertisements

Final Lecture: MOOC Planning – Part 9

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

I gave my last lecture of the course yesterday (discounting the tutorial session that will go out next week), and we are now starting a two week exam period.

“Giving” a lecture means the video becomes available for streaming. For logistic reasons (high among them, my survival and continued sanity — assuming anyone who organizes and gives a MOOC, for no payment, is sane), I recorded all the lectures weeks ago, well before the course started.  The weekly tutorial sessions come the closest to being live. I record them one or two days before posting, so I can use them to respond to issues raised in the online course discussion forum.

The initial course enrollment of 63,649 has dropped to 11,848 individuals that the platform says are still active on the site. At around 20%, that’s pretty high by current MOOC standards, though I don’t know whether that is something to be pleased about, since  it’s not at all clear what the right definition of “success” is for a MOOC.

Some might argue that 20% completion indicates that the standards are too low. I don’t think that’s true for my course. Completion does, after all, simply mean that a student is still engaged. The degree to which they have mastered the material is unclear. So having 80% drop out could mean the standard is too high.

In my case, I did not set out to achieve any particular completion rate; rather I adopted a WYSIWOSG approach — “What You See Is What Our Students Get.” I offered a MOOC that is essentially the first half of a ten week course I’ve given at many universities over the years, including Stanford. That meant my students would experience a Stanford-level course. But they would not be subject to passing a Stanford-level exam.

In fact, I could not offer anything close to a Stanford-exam experience. There is a Final Exam, and it has some challenging questions, but it is not taken under controlled, supervised conditions. Moreover,  since it involves constructing proofs, it cannot be machine graded, and thus has to be graded by other students, using a crowd sourcing method (Calibrated Peer Review). That put a significant limitation on the kinds of exam questions I could ask. On top of that, the grading is done by as many different people as there are students, and I assume most of them are not expert mathematicians. As a result, it’s at most a “better-than-nothing” solution. Would any of us want to be treated by a doctor whose final exam had been peer graded (only) by fellow students, even if the exam and the grading had been carried out under strictly controlled conditions?

On the other hand, looking at and attempting to evaluate the work of fellow students is a powerful learning experience, so if you view MOOCs as vehicles for learning, rather than a route to a qualification, then peer evaluation has a lot to be said for it. Traditional universities offer both learning and qualifications. MOOCs currently provide the former. Whether they eventually offer the latter as well remains to be seen. There are certainly ways it can be done, and that may be one way that MOOCs will make money. (Udacity already does offer a credentialing option, for a fee.)

In designing my course, I tried to optimize for learning in small groups, perhaps five to fifteen at a time. The goal was to build learning communities, within which students could help one another. Since there is no possibility of regular, direct interaction with the instructor (me) and my one TA (Paul), students have to seek help from fellow students. There is no other way. But, on its own, group work is not enough. Learning how to think mathematically (the focus of my course) requires feedback from others, but it needs to include feedback from people already expert in mathematical thinking. This means that, in order to truly succeed, not only do students need to work in groups (at least part of the time), and subject their attempts to the scrutiny of others, some of those interactions have to be with experts.

One original idea I had turned out not to work, though whether through the idea itself being flawed or the naive way we implemented it is not clear to me. That was to ask students at the start of the course to register if they had sufficient knowledge and experience with the course material to act as “Community TAs”, and be so designated in the discussion forums. Though over 600 signed up to play that role, many soon found they did not have sufficient knowledge to perform the task. Fortunately,a relatively small number of sign-ups did have the necessary background, as well as the interpersonal skills to give advice in a supporting, non-threatening way, and they more or less  ensured that the forum discussions met the needs of many students (or so it seems).

Another idea was to assign students to study groups, and use an initial survey to try to identify those with some background knowledge and seed them into the groups. Unfortunately, Coursera does not (yet) have functionality to support the creation and running of groups, apart from the creation of forum threads. So instead, in my first lecture, I suggested to the students that they form their own study groups in whatever way they could.

The first place to do that was, of course, the discussion forums on the course website, which very soon listed several pages of groups. Some used the discussion forum itself to work together, while others migrated offsite to some other location, physical or virtual, with Skype seeming a common medium. Shortly after the course launched, several students discovered GetStudyRoom, a virtual meeting place dedicated to MOOCs, built by a small startup company.

In any event, students quickly found their own solutions. But with students forming groups in so many different ways on different media, there was no way to track how many remained active or how successful they have been.

The study groups listed on the course website show a wide variety of criteria used to bring the groups together. Nationality and location were popular, with groups such as Brazil Study Group, Grupo de Estudo Português, All Students From Asia, and Study Group for Students Located in Karachi, Pakistan. Then there were groups with a more specific focus, such as Musicians, Parents of Homeschooled Children, Older/Retired English Speakers Discussion for Assignment 1, and, two of my favorites, After 8pm (UK time) English speakers with a day job and the delightfully named Just Hanging on Study Group.

The forum has seen a lot of activity: 15,088 posts and 13,622 comments, spread across 2712 different threads, viewed 430,769 times. Though I have been monitoring the forums on an almost daily basis, to maintain an overall sense of how the course is going, it’s clearly not possible to view everything. For the most part I restricted my attention to the posts that garnered a number of up-votes. Students vote posts up and down, and once a post shows 5 or more up-votes, I take that as an indication that the issue may be worth looking at.

The thread with the highest number of up-votes (165) was titled Deadlines way too short. Clearly, the question of deadlines was a hot topic. How, if at all, to respond to such feedback is no easy matter. In a course with tens of thousands of students, even a post with hundreds of up-votes represents just a tiny fraction of the class. Moreover, threads typically include opinions on both sides of an issue.

For instance, in threads about the pace of the course, some students complained that they did not have enough time to complete assignments, and pleaded for more relaxed deadlines, whereas others said they thrived on the pace, which stimulated them to keep on top of the material. For many, an ivy-league MOOC offers the first opportunity to experience an elite university course, and I think some are surprised at the level and pace. (I fact, I did keep the pace down for the first three weeks, but I also do that when I give a transition course in a regular setting, since I know how difficult it is to make that transition from high school math to university level mathematics.)

A common suggestion/request was to simply post the course materials online and let students access them according to their own schedules, much like Khan Academy. This raises a lot of issues about the nature of learning and the role MOOCs can (might? should?) play. But this blog post has already gone on long enough, so I’ll take up that issue next time.

To be continued …

Why MOOCs Look Unprofessional: MOOC planning – Part 4

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

From an educational perspective, my goal in offering a MOOC on mathematical thinking is very modest. I have not approached the task as one of developing a whole new pedagogic model. That is a future goal — for me or for others. Rather I set out to see how much we can take current university teaching (of transition mathematics material) and make it available to a wide audience. Indeed, almost all the Stanford MOOCs currently being offered are free, online versions of regular Stanford courses, in many cases running concurrently with a physical class on campus. (As I noted in an earlier post, the technology that supports these MOOCs was actually developed at Stanford in order to facilitate flipped-classroom learning in on-campus classes.)

The underlying assumption of university education — at least at major research universities (as Stanford is) — is that the principle value for the student comes from studying with a world expert in a particular domain. Though many professors at research universities do in fact put enormous effort into their teaching, what is really being offered (sold) to students is the expertise (and reputations) of the faculty. (Other parts of the value proposition, such as the prestige of the university, stem from the faculty, both past and present.) It’s a method that works well for very bright, well-prepared, and highly motivated students, but it is not ideal for everyone.

In fact, even at less prestigious universities, where there are fewer leading research faculty, and at liberal arts colleges, where the primary focus is on undergraduate education, field-content knowledge hugely outweighs pedagogical content knowledge — how to teach the subject and how students learn it. (A Ph.D. is usually required for a faculty position.) That makes universities and colleges very different from high schools.

One of the implicit purposes of  a math transition course, such as mine (as well as many other first-year courses in different disciplines), is to help incoming students adjust to the different approach to teaching. More precisely, it is to help them adjust to not being “taught”, but having someone help them learn. This is particularly significant in mathematics — at least in the US — because of the hugely formulaic, procedures-focused nature of K-12 mathematics education in this country.

My challenge then, like that facing most of my colleagues offering their first MOOC, is to figure out how to take an existing educational model, hitherto used to teach (or help to learn) twenty-five or so students in a classroom, and make it available to thousands, spread around the world.

Since my topic is mathematical thinking, the biggest, and most obvious challenge is how to compensate for the complete absence of regular interaction between the students and me, the instructor. Sure, I give lectures when I teach a physical transition class, but the lectures are one of the least significant components. They really just set the agenda for learning. In order to help the students develop the ability for mathematical thinking, I need to see them in action at the board, to read their work, and to discuss their attempts face-to-face. Learning to think mathematically is more like learning to drive or to play tennis than soaking up knowledge. You have to do it alongside an expert or coach.

It’s a challenge I think cannot be completely overcome in a MOOC. The question is, is it possible to get part-way there? I suspect it is, but we’ll only find out for sure by making the attempt. So here we are.

One thing a MOOC does offer that is not possible in a physical class — and hence is a plus — is that all the instruction and professorial-learning-assistance can be on a one-to-one basis. Sure, it’s all one way, but if you set it up right (and if your voice/personality/whatever work over an ethernet cable), then the student can get that sense of working alongside the instructor — the expert.

Though by no means the first to discover that, Salman Khan, by virtue of his huge following at Khan Academy, demonstrated just how powerful is that sense of “working together, side-by-side”. Though I share the dismay of many of my colleagues at his less-than-expert content knowledge and his almost non-existent pedagogical content knowledge (neither of which he could be expected to have, given his background), where I seem to part company with many of them is the huge significance I attach  to the way he pulls off that human-connect. For online learning, I suspect it trumps almost all other factors.

(BTW, in developing my MOOC, I soon lost track of the number of times I made a decision based on a “suspicion” — or a “guess” or  “hunch”. MOOCs are generating enough research questions to sustain several generations of doctoral dissertations in education research.)

Based on that suspicion (admittedly a suspicion comfortingly buttressed by a Khan Academy user base that numbers in the millions), Khan’s format was my starting point, as I observed in my last post. Not just the physical aspect of “sitting alongside in a one-on-one tutorial” but the associated human connect (and with it reassurance and encouragement) that Khan delivers.

In Khan’s case, his now widely familiar format originated with him informally helping his school-age relatives (who lived a long way away) with their math homework. What the viewer gets on their computer screen is, well, just “Uncle Sal”, doing what he would have done if he were really sitting alongside one of his relatives. For my MOOC, I wanted to achieve a similar outcome. Not a slick show, not a polished, rehearsed performance. Just me doing math.

Of course, the logistics of putting together a complete course that has to run automatically, and be scalable to many thousands of students around the world, many of them not native English speakers, meant that there had to be a lot of detailed advanced planning. Everything had to be scripted. But when it comes to the bits where I explain some mathematics, I put the script to one side and just start to work through the material as if I am sitting next to a student.

You might not like it. It might not work for you. You will surely despair at my handwriting. You might hate my accent. (I did cut down drastically on my jokes and puns, in deference to a multilingual audience.) But as far as I can make it, absent being physically in the same room, it’s what you would get if you were taking the course with me here at Stanford.  [Some time spent in a campus video-editing studio made my into-camera segments look a lot smoother than they were when we recorded them! If it’s digital, it’s plastic. But the goal there was to reduce the length of those segments.]

Which brings me back to my starting point: seeing the extent to which we can take existing university education and make it available to the world.

Once we can do that — and it will surely take several iterations to iron out all the kinks and make an altogether better job of it — we can look at how to change the underlying model. In addition to MOOCs making accessible to the world some aspects of university education, I think that the act of designing them, mounting them, and analyzing the results, will lead to changes in the way we organize learning within our universities.

It is because the current goal is to see how well we can deliver (current) real university education to the world for free that most of the MOOCs being offered have an unpolished, unrehearsed look. By deliberate choice, to the greatest degree we can achieve, what you see is what our (on-campus) students get. (I think this WYSIWOSG philosophy — I just made up that term —  is also one of the reasons for the success of Salman Khan — including the fact that in his case, unlike university MOOCs, he does not even lesson-plan his instruction sessions.)

So much for the most visible part of the MOOC: the instruction. But instruction is still just instruction. As I’ve said before, the learning takes place elsewhere, through other mechanisms, none of which we understand very well. So where is that educational  meat?

Now we are about to really enter speculative territory.

To be continued …

COMMENTS: As always, comments are welcome, provided they remain on topic.

Khan Academy Meets Vi Hart: MOOC planning – Part 3

A real-time chronicle of a seasoned professor embarking on his first massively open online course.

The ideal way to learn mathematical thinking (and a great many other things that involve understanding, not just doing) is in a small physical group with an expert. That provides frequent opportunities to interact one-on-one with the expert, during which the expert can observe you work in real time (on paper or at a board) and can give you direct feedback on written work you have done and handed in for evaluation. It also provides frequent opportunities to discuss what is being learned with other students at the same stage of their learning, sometimes with the expert present, other times with the expert absent.

Sometimes, the expert will provide instruction. Though there have been successful instances of mathematics professors who largely avoid instruction (R L Moore being the most notable example), most of us (i.e., university mathematics educators) find that instruction has a valuable place in mathematics education. But many of us view it as just one part of mathematics education.

Anyone who has experienced highly interactive mathematics teaching will know how different it is from mere instruction, and how much more effective. I wrote about this last March in my Devlin’s Angle column for the MAA. Unfortunately, it seems clear that a great many Americans have never experienced good mathematics teaching. If they had, you would not have thousands of Khan Academy users (including famous figures such as Bill Gates) declaring Salman Khan is the best math teacher ever. You can say a number of good things about Sal Khan (I am going to say some of them in just a moment), but being a great math teacher is not one of them. To say that he is, simply reflects on the miserable math ed diet that many millions of American have been fed, for whom Khan Academy offers something far better than they were ever exposed to.

I bring up Khan Academy for a couple of reasons, one being that it set the stage for the MOOC explosion. Indeed, former Stanford CS professor Sebastian Thrun stated publicly last January that it was Khan Academy that inspired him to give his first MOOC in fall 2011, and then to leave Stanford and launch his own MOOC service Udacity at the start of this year.

It’s not merely the wide reach that Khan Academy demonstrated. As I discussed in a recent article for the MAA, Sal Khan managed to tap into the power of the Web medium to achieve a critical element of good teaching that not all teachers can offer: a strong teacher-student bond. Moreover, he did so using just his voice and the electronic trail of a digital pen on the viewer’s computer screen. Yes, some of the math is wrong, and the pedagogy is so poor, experienced teachers tear their hair out, but the very success of Khan Academy shows how important is the teacher-student connection.

Khan Academy is not a MOOC, of course, but it does provide a model for online mathematics instruction. In starting to plan my MOOC, I began by trying variants of Sal’s approach for the instructional part. Like him, I have a voice that works on the radio (or a Web audio channel) — an accident of birth — which makes such an approach feasible.

I soon concluded that his approach would not work. It is fine for presenting short instructional mini-lectures on how to follow a particular mathematical procedure, but it is woefully impoverished for trying to help students understand a mathematics idea or a proof, and to form the right mental concepts. For that, the huge importance in mathematics teaching of physical gestures, in particular the hand(s), cannot be ignored.

There is an old challenge in which you ask someone to describe a helix while keeping their hands clasped firmly behind their back. (Try it!) But it’s not just helices. Explaining almost any mathematical concept without using at the very least hand and arm gestures, and in many cases full body motion, is difficult if not impossible. There is masses written about this topic, based on many years of research. For example, take a look at this summary, or this one, or this forthcoming book. Or Google on the terms “mathematics + learning + hand + gesture” or variants thereof to see a lot more.

Since MOOC students access the material on a wide range of devices, with different screen sizes, I felt that a full body recording of me working at (and in front of) a blackboard or whiteboard would not be ideal. Besides, I love the sense of intimacy Khan Academy offers. You get a strong sense of sitting next to a friendly relative who is personally instructing you. I wanted to create that environment.

But trying to follow an explanation of a mathematical concept or proof Khan-style, where the visual channel consists only of a digital pen trace, was impossible — at least, it was given my educational style. At the very least, I needed my writing hand to direct the student’s focus. The simplest way to achieve that was to have a video camera mounted above my desk and record me working through the material in the time-honored fashion of paper-and-pencil. That seemed to work.

Having decided on the basic modality, the next issue was one of style and tone. After playing with some variants of the basic format, I came down in favor of a very informal look, where I simply slap down a sheet of paper on the desk in front of me and the student, and work through the material. (Marking the exact position of the paper on the desk and letting it totally fill the screen looked too artificial — though at this stage the issue was largely one of taste, and this is a decision I may change based on the experience I get from this first course. I did have to tape down the paper, but the initial placement was fairly casual, and the taping was sufficiently loose that the paper could still move a little — it takes effort to create “informality” on video.)

To counter the inevitable sense of frustration when watching a pen write something out in real time, I decided to speed up a lot of the writing during the video editing phase. (Though not to the speed of the wonderful Vi Hart, whose purpose is informative entertainment.) So at that stage I found myself with a “Sal Khan meets Vi Hart” look. A great place to start, given the success both have achieved!

For standalone Web instruction, that would likely be enough, but a MOOC involves a lot more. It is, after all, a course — a structured experience over several weeks, with a professor. Regular connection to the instructor is important — at least, I think it is. (It was for me when I was a student.) To achieve that “human connection,” many of my Stanford colleagues who have given MOOCs have put a small head-and-shoulders video of themselves speaking in one corner of the screen, as the material being discussed occupies the rest of the display. I tried that, and found it did not work for me, with my material. The face was a distraction. I wanted to keep as much of the Khan Academy sense as possible — you don’t ignore success unless there is good reason! So I opted to keep video of me separate from the hand-writing part.

I’ve posted a short sample from Lecture 1 on YouTube. Given the low resolution of YouTube video encoding, this does not display well in terms of content, but the Coursera platform uses far higher resolution video.

I doubt much of this material will survive to a second iteration of the course next year. At the very least, I’d want to go back and pay more attention to lighting and audio levels and consistency.  But it does have the overall look and feel I was trying to achieve. This is live beta, folks.

But as I have already indicated in this blog series, I don’t see the video lectures as the heart of the course. They merely set the agenda for learning. The real learning takes place elsewhere. I’ll turn to that topic in a future post.

Meanwhile, my Stanford MOOC Introduction to Mathematical Thinking is scheduled to begin on September 17 on Coursera. If you want to do some preliminary reading, there is my low-cost course textbook by the same name. Though written to align to the course, it is not required in order to complete the course. (Indeed, I noted  above that I see MOOCs as replacing textbooks — though some MOOCs may have required textbooks, so it would be unwise to predict the imminent death of the printed textbook!)

To be continued …

NOTE: I mentioned Khan Academy to indicate its role in the MOOC explosion and acknowledge its role in guiding the design of the instructional videos in my MOOC. But the focus of this blog is on MOOCs in general and mathematics MOOCs in particular. Comments discussing the merits or demerits of Khan Academy are off topic and hence will not be published; there are many other venues for such discussions.


I'm Dr. Keith Devlin, a mathematician at Stanford University. I gave my first free, open, online math course in fall 2012, and have been offering it twice a year since then. This blog chronicles my experiences as they happen.

Twitter Updates

New Book 2012

New book 2011

New e-book 2011

New book 2011

March 2019
M T W T F S S
« Dec    
 123
45678910
11121314151617
18192021222324
25262728293031
Advertisements

%d bloggers like this: