The blind leading the blind: adventures in online education at duolingo

We interrupt our regular programming for a bit of amusement. The background for this is that I am considering developing an online calculus class (not a MOOC for now, rather something more modest), and so I have been thinking a lot about online education lately. In particular I am curious to see what is already out there.

Now in general I have some qualms about excessively computerized education. I think that education is about more than just optimizing the process of loading the student’s mind with a set collection of facts and rote skills. Rather, there is a certain give and take between the teacher and the student (which works best face to face), and both learn together.

That said, there are some areas where one cannot get around memorizing a certain number of facts and rote skills. A prime example of such an area is elementary language learning. Thus I was quite interested when I heard about a relatively new website called duolingo.com. The idea of this site is that students learn languages for free, and in the process of learning they translate websites in a crowdsourced manner (and the site is funded by translation fees). Now this sounds a bit preposterous, and I can’t find any details about how it works, but after playing with it for a bit, here is what I think is going on.

The language learner sees a series of quizzes. It is like a game and potentially addictive. Quizzes can either review what you are already supposed to know or introduce new material. Quiz questions ask you to translate a short phrase or sentence either from the language you are learning to your native language or vice versa. It is very engaging, constantly asking you to produce output with instant feedback, which I think is an excellent feature for any kind of online education.

That is the good part. Now the bad part is that it is full of errors and inconsistencies, especially at the higher levels. I’ll show you some examples in a minute. Here is my speculation on what is going on. There is some website that duolingo is being paid to translate. The software breaks up the texts into little bits and quizzes language learners on how to translate these bits. (Never mind that it is questionable whether one can get good translations by breaking up sentences into little bits. Just try this with a long German sentence in which a verb is split into two pieces three lines apart…) The quiz answers are initially graded using some software along the lines of Google Translate (and maybe also some algorithm which compares passages in two different languages and decides whether or not they are saying the same thing). Thus, when a question is first introduced into the quiz system, it may have errors. After a while, the grading is adjusted depending on what the users do. For example, if the same supposedly wrong answer is given some number of times, then this answer is reclassified as correct. There is also a button where you can complain if you think you were graded incorrectly (I pressed this button many times when I was trying it out).

I tested this out saying that I am a native English speaker trying to learn German. (I know enough German to feel competent to evaluate the program in this area.) I tested out of the basic levels and then did some review quizzes. Here are some of the errors I found. (If I am the one who was wrong in any of these cases, please correct me.)

  • It asked me to translate a sentence which began “If we didn’t exist…” I started with “Wenn wir nicht existierten…”. It marked this wrong and said that I should have written “Wenn wir nicht existieren würden…” As far as I know these mean exactly the same thing, but mine is less clumsy. In general it doesn’t deal well with the fact that there can be different equivalent ways to say the same thing, especially in longer sentences.
  • It asked me to translate “I drank wine.” I wrote “Ich habe Wein getrunken”. It marked this wrong and wanted “Ich trank Wein.” Another case of equivalent ways to say the same thing (although here you prefer one or the other depending on whether you are speaking or writing).
  • It asked me to translate “Mit zwölf Jahren, ist ein Hund alt”. I answered “A dog is old when it is twelve”. It wanted “a dog is old with twelve” or “with twelve years, a dog is old.” Neither of these makes much sense in English. It seemed to be insisting on the word for word translation of “mit” as “with”, but you can’t just replace German prepositions by similar English prepositions or you will often get nonsense.
  • It asked me to translate “I go every year”. I answered “Ich gehe jedes Jahr hin”. It marked that wrong and wanted “Ich gehe jedes Jahr”. Another example where word for word translation gives you bad results if you don’t have an idea of the overall meaning.
  • It asked me to translate “He hears notes”. I wrote “Er hört Töne”. It marked this wrong and wanted “Er hört Noten”. While “Note” means “note” in some cases, according to my German dictionary the things that you here are “Töne”, not “Noten”. Correct me if I’m wrong.
  • It asked me to translate “Grandmother is making tea”. I wrote “Großmutter kocht Tee”. It marked this wrong and wanted “Großmutter macht Tee”. I wasn’t sure about this, and since I didn’t have a native German speaker handy to ask, I used a little trick: a Google search for “Tee kochen” (in quotes) gives three times as many hits as “Tee machen”. I conclude from this that mine is probaby better, and in any case acceptable.
  • It asked me to translate “Das ist bei uns ganz klar der Diesel”. It was multiple choice, and the only option remotely close to this was “That is for us clearly the diesel”, which is not something a native English speaker would ever say.
  • It asked me to translate “Wir bleiben in Kontakt”. I answered “We will stay in touch”. It marked this wrong wanted “We stay in touch.” First of all, it doesn’t deal well with the fact that there is not a bijection between verb tenses in the two languages; the present tense in German can correspond to the present or future tense in English. Second, “We stay in touch” is somewhat unnatural English. I can think of situations in which one might say this, but in general the program has a bad tendency to translate things into the “habitual present” tense in English when this does not make sense. For example a correct answer to another question was “The elephant eats an apple.” This is (usually) not a correct English sentence. (I guess you could say it while narrating a story in the present tense, but that is uncommon.) You could say “The elephant is eating an apple” (to refer to what the elephant is doing right now) or “The elephant eats apples” (to indicate that apples are part of the elephant’s diet). But “The elephant eats an apple” does not make sense as a standalone sentence. It could make sense as part of a sentence, e.g. “If the elephant eats an apple, it will get indigestion”. But it looked like we were supposed to translate a complete sentence… The program also seemed to have a lot of trouble with the Konjunktiv, but I don’t have examples handy.
  • It asked me to translate “Ein Auftrag aus der Zukunft”. Ignoring the fact that this is a weird thing to say, I translated it as “An assignment from the future”. It marked this wrong and wanted “A assignment from the future”. Really? Surely it would be trivial to program it to know when to use “a” versus “an”.
  • It asked me to translate “I do not go on vacation without a pocket book”. It was multiple choice. The only close choice was “Ich fahre nicht in den Urlaub nicht ohne Taschenbuch”. There is one “nicht” too many there.
  • It asked me to translate “Does he have his insurances?”. I don’t think that’s correct English, nor do I know what it is supposed to mean.

There is more like this (and there were some hilarious examples much worse than these on the discussion forum), but you get the idea. I would estimate that in the more advanced levels of the German section, about 20 percent of the translation questions have significant issues.

The sentences that look like garbage are obviously the output of computer translation software which we are supposed to be correcting. This is not good for the language learner. We can recognize when our own language is wrong, but if we are learning a new language we don’t know. There are discussion forums where you can ask about questions you have, but it might take a while to get an answer, and the person answering your question might not know any more than you.

The worst part of this is that after a while you are trained to give answers which you know are wrong but the computer will accept in order to avoid losing points. (It is like a game and if you lose too many points you have to backtrack to the beginning of the quiz and get some of the same annoying wrong questions again.) In this way, flaws in the translation software are reinforced and amplified into systematic errors in the program.

Despite all of the above criticism, it is a fun and valuable program. If you need a little break from work, go try it and see what you think. (Hopefully some of the above problems will be fixed. “A” versus “an” should take five minutes.) And then think about how we can teach math online in an engaging manner without propagating errors.

P.S. I don’t know if they are actually doing real translation work yet or if it the system is still in an experimental stage. As I said I couldn’t find any details about what they are doing. There are also some texts that you can try translating yourself, or edit other people’s translations of. However I suspect that at least some of this is not for pay but rather for data gathering purposes. For example some of the texts are classic works (e.g. fairy tales) which have been translated many times before.

P.P.S. Another amusing English sentence which turned up was “You are swimming better than he does.” At least the foreign language sentences seem to have fewer errors, or maybe I just don’t notice them.

This entry was posted in Uncategorized. Bookmark the permalink.

8 Responses to The blind leading the blind: adventures in online education at duolingo

  1. Paolo says:

    Regarding your first example, as far as I know (but I’ve only spoken German in Munich, and this might be Bavarian usage), “Wenn wir nicht existierten” is not used in practice: the analytic form of subjunctive II is used only for few verbs. Chris can give you more reliable information.

    • OK, I am most likely wrong (but I should be able to trust a language teaching program). “Wenn wir nicht existierten” gets 9 hits on google, and “wenn wir nicht existeren würden” gets 53,800 results, which looks bad. On the other hand, “wenn es nicht existierte” gets 305,000 hits and “wenn es nicht existieren würde” gets 161,000 hits. So I don’t know what’s going on here.

      They also have Italian which could maybe give you a few laughs; on the other hand I think the Romance languages (Spanish and French anyway, I don’t know anything about Italian) are closer to being isomorphic to English so maybe there are fewer problems there.

  2. Ronnie Biggs says:

    the same thing happens with spanish and portuguese when i’m trying to translate questions, in order to get them right i have to think in a romance language and invert the order like yoda and it still gets it wrong frequently, i’m surprised that a computer can beat a grand master in chess yet still can’t get language translation right, however it does create opportunities for human professionals and people creating interesting solutions in the software field

    • Yes indeed, it is a very interesting problem to try to create good translation software. Duolingo is supposed to be such a solution, but it is still a mystery to me how it works and I can’t find any details. At first I thought that they were translating documents by breaking them up into short phrases, quizzing language learners on how to translate these phrases, and using the users’ answers (and complaints about wrong answers) to find good translations. That may be part of it, but it is not enough, because even if you go through all the quizzes you have only covered around 2000 words. So one possibility is that they are applying some kind of transformations to get down to their set of 2000 base words. Another possibility is that this will slowly expand. Another possibility is that they are using the users’ behavior in response to the quizzes to create some vast data set which they hope to use to eventually create better translation software.

  3. Update: When I tried this in July, I pressed the button to complain about a number of things which I thought were errors. I then started getting emails from “Duolingo Feedback” saying that my proposed translation for this or that sentence was now accepted. Interestingly, I am still getting some of these emails two months later! My conjecture is that there is some threshhold, so that when the same new translation is suggested by a sufficient number of users, a human administrator checks whether the proposed translation is correct and then approves it or not. (I have never received feeback saying that one of my proposed translations was rejected, even though I surely made some mistakes, presumably because they have a policy of never sending negative feedback.)

  4. david farris says:

    my pal ramesh pointed out the following duolingo translation project: http://www.duolingo.com/translation/fb22c050d21d4d612fa84a9e9d835efa

    • Yes, if you are feeling generous with your time you can try to translate documents directly on duolingo. It’s an interesting challenge with more difficult texts. However the crowdsourcing aspect can be frustrating, because some people either just copy the output of Google Translate, or clearly do not grasp essential grammar points which affect the meaning of the sentence to be translated, and then when you correct it, they correct it back! Ugh.

  5. Jeremy Gates says:

    Great article! Being an advanced Spanish speaker I began using Duolingo to improve, but got frustrated when translations I knew were correct where marked wrong. I, too, found myself ‘adjusting’ my answers to sentences that, intuitively, made no sense to me in order to get the question right! I knew I wasn’t going crazy.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s