Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for initializing language models for automatic speech recognition. In one aspect, a method includes receiving logged speech recognition results from an existing corpus that is specific to a given language and a target context, generating a target corpus by machine-translating the logged speech recognition results from the given language to a different, target language, and estimating a language model that is specific to the different, target language and the same, target context, using the target corpus.