This specification describes technologies relating to generating custom language models for audio content. In some implementations, a computer-implemented method is provided that includes the actions of receiving a collection of source texts; identifying a type from a collection of types for each source text, each source text being associated with a particular type; generating, for each identified type, a type-specific language model using the source texts associated with the respective type; and storing the language models.