We interrupt our regular programming for a bit of amusement. The background for this is that I am considering developing an online calculus class (not a MOOC for now, rather something more modest), and so I have been thinking a lot about online education lately. In particular I am curious to see what is already out there.
Now in general I have some qualms about excessively computerized education. I think that education is about more than just optimizing the process of loading the student’s mind with a set collection of facts and rote skills. Rather, there is a certain give and take between the teacher and the student (which works best face to face), and both learn together.
That said, there are some areas where one cannot get around memorizing a certain number of facts and rote skills. A prime example of such an area is elementary language learning. Thus I was quite interested when I heard about a relatively new website called duolingo.com. The idea of this site is that students learn languages for free, and in the process of learning they translate websites in a crowdsourced manner (and the site is funded by translation fees). Now this sounds a bit preposterous, and I can’t find any details about how it works, but after playing with it for a bit, here is what I think is going on.
The language learner sees a series of quizzes. It is like a game and potentially addictive. Quizzes can either review what you are already supposed to know or introduce new material. Quiz questions ask you to translate a short phrase or sentence either from the language you are learning to your native language or vice versa. It is very engaging, constantly asking you to produce output with instant feedback, which I think is an excellent feature for any kind of online education.
That is the good part. Now the bad part is that it is full of errors and inconsistencies, especially at the higher levels. I’ll show you some examples in a minute. Here is my speculation on what is going on. There is some website that duolingo is being paid to translate. The software breaks up the texts into little bits and quizzes language learners on how to translate these bits. (Never mind that it is questionable whether one can get good translations by breaking up sentences into little bits. Just try this with a long German sentence in which a verb is split into two pieces three lines apart…) The quiz answers are initially graded using some software along the lines of Google Translate (and maybe also some algorithm which compares passages in two different languages and decides whether or not they are saying the same thing). Thus, when a question is first introduced into the quiz system, it may have errors. After a while, the grading is adjusted depending on what the users do. For example, if the same supposedly wrong answer is given some number of times, then this answer is reclassified as correct. There is also a button where you can complain if you think you were graded incorrectly (I pressed this button many times when I was trying it out).
I tested this out saying that I am a native English speaker trying to learn German. (I know enough German to feel competent to evaluate the program in this area.) I tested out of the basic levels and then did some review quizzes. Here are some of the errors I found. (If I am the one who was wrong in any of these cases, please correct me.)
- It asked me to translate a sentence which began “If we didn’t exist…” I started with “Wenn wir nicht existierten…”. It marked this wrong and said that I should have written “Wenn wir nicht existieren würden…” As far as I know these mean exactly the same thing, but mine is less clumsy. In general it doesn’t deal well with the fact that there can be different equivalent ways to say the same thing, especially in longer sentences.
- It asked me to translate “I drank wine.” I wrote “Ich habe Wein getrunken”. It marked this wrong and wanted “Ich trank Wein.” Another case of equivalent ways to say the same thing (although here you prefer one or the other depending on whether you are speaking or writing).
- It asked me to translate “Mit zwölf Jahren, ist ein Hund alt”. I answered “A dog is old when it is twelve”. It wanted “a dog is old with twelve” or “with twelve years, a dog is old.” Neither of these makes much sense in English. It seemed to be insisting on the word for word translation of “mit” as “with”, but you can’t just replace German prepositions by similar English prepositions or you will often get nonsense.
- It asked me to translate “I go every year”. I answered “Ich gehe jedes Jahr hin”. It marked that wrong and wanted “Ich gehe jedes Jahr”. Another example where word for word translation gives you bad results if you don’t have an idea of the overall meaning.
- It asked me to translate “He hears notes”. I wrote “Er hört Töne”. It marked this wrong and wanted “Er hört Noten”. While “Note” means “note” in some cases, according to my German dictionary the things that you here are “Töne”, not “Noten”. Correct me if I’m wrong.
- It asked me to translate “Grandmother is making tea”. I wrote “Großmutter kocht Tee”. It marked this wrong and wanted “Großmutter macht Tee”. I wasn’t sure about this, and since I didn’t have a native German speaker handy to ask, I used a little trick: a Google search for “Tee kochen” (in quotes) gives three times as many hits as “Tee machen”. I conclude from this that mine is probaby better, and in any case acceptable.
- It asked me to translate “Das ist bei uns ganz klar der Diesel”. It was multiple choice, and the only option remotely close to this was “That is for us clearly the diesel”, which is not something a native English speaker would ever say.
- It asked me to translate “Wir bleiben in Kontakt”. I answered “We will stay in touch”. It marked this wrong wanted “We stay in touch.” First of all, it doesn’t deal well with the fact that there is not a bijection between verb tenses in the two languages; the present tense in German can correspond to the present or future tense in English. Second, “We stay in touch” is somewhat unnatural English. I can think of situations in which one might say this, but in general the program has a bad tendency to translate things into the “habitual present” tense in English when this does not make sense. For example a correct answer to another question was “The elephant eats an apple.” This is (usually) not a correct English sentence. (I guess you could say it while narrating a story in the present tense, but that is uncommon.) You could say “The elephant is eating an apple” (to refer to what the elephant is doing right now) or “The elephant eats apples” (to indicate that apples are part of the elephant’s diet). But “The elephant eats an apple” does not make sense as a standalone sentence. It could make sense as part of a sentence, e.g. “If the elephant eats an apple, it will get indigestion”. But it looked like we were supposed to translate a complete sentence… The program also seemed to have a lot of trouble with the Konjunktiv, but I don’t have examples handy.
- It asked me to translate “Ein Auftrag aus der Zukunft”. Ignoring the fact that this is a weird thing to say, I translated it as “An assignment from the future”. It marked this wrong and wanted “A assignment from the future”. Really? Surely it would be trivial to program it to know when to use “a” versus “an”.
- It asked me to translate “I do not go on vacation without a pocket book”. It was multiple choice. The only close choice was “Ich fahre nicht in den Urlaub nicht ohne Taschenbuch”. There is one “nicht” too many there.
- It asked me to translate “Does he have his insurances?”. I don’t think that’s correct English, nor do I know what it is supposed to mean.
There is more like this (and there were some hilarious examples much worse than these on the discussion forum), but you get the idea. I would estimate that in the more advanced levels of the German section, about 20 percent of the translation questions have significant issues.
The sentences that look like garbage are obviously the output of computer translation software which we are supposed to be correcting. This is not good for the language learner. We can recognize when our own language is wrong, but if we are learning a new language we don’t know. There are discussion forums where you can ask about questions you have, but it might take a while to get an answer, and the person answering your question might not know any more than you.
The worst part of this is that after a while you are trained to give answers which you know are wrong but the computer will accept in order to avoid losing points. (It is like a game and if you lose too many points you have to backtrack to the beginning of the quiz and get some of the same annoying wrong questions again.) In this way, flaws in the translation software are reinforced and amplified into systematic errors in the program.
Despite all of the above criticism, it is a fun and valuable program. If you need a little break from work, go try it and see what you think. (Hopefully some of the above problems will be fixed. “A” versus “an” should take five minutes.) And then think about how we can teach math online in an engaging manner without propagating errors.
P.S. I don’t know if they are actually doing real translation work yet or if it the system is still in an experimental stage. As I said I couldn’t find any details about what they are doing. There are also some texts that you can try translating yourself, or edit other people’s translations of. However I suspect that at least some of this is not for pay but rather for data gathering purposes. For example some of the texts are classic works (e.g. fairy tales) which have been translated many times before.
P.P.S. Another amusing English sentence which turned up was “You are swimming better than he does.” At least the foreign language sentences seem to have fewer errors, or maybe I just don’t notice them.