Evaluation procedure: 1) Normalise punctuation for all submissions + gold-standard Moses script (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/normalize-punctuation.perl): perl normalize-punctuation.perl -l de < file > file.norm 2) Lowercase all submissions which were not lowercases + gold-standard Moses script (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/lowercase.perl): perl lowercase.perl < file.norm > file.norm.lc 3) Detokenize all submissions which were tokenized Moses script (https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/detokenizer.perl): + nonbreaking_prefix.de (https://github.com/moses-smt/mosesdecoder/tree/master/scripts/share/nonbreaking_prefixes) perl detokenizer.perl -l de < file.norm.lc > file.norm.lc.det 4) Run Meteor (http://www.cs.cmu.edu/~alavie/METEOR/) Meteor (primary): java -Xmx2G -jar meteor-1.5/meteor-*.jar submission gold-standard -l de Meteor (tokenized – secondary): java -Xmx2G -jar meteor-1.5/meteor-*.jar submission gold-standard -l de -norm