A processing device receives a multi-dimensional input associated with audio data representing an utterance received by a speech-detection device, the multi-dimensional input comprising a speaker identifier (ID) of a speaker of the utterance and a device ID of the speech-detection device. The processing device processes the multi-dimensional input using a first machine learning model associated with at least one of the device ID or the speaker ID to determine a first anomaly rating for the utterance. The processing device determines that the first anomaly rating is above a first anomaly threshold. The processing device determines that the utterance is classified as an anomaly and performs a remedial action.