Examples for augmenting user recognition via speech are provided. One example method comprises, on a computing device, monitoring a use environment via one or more sensors including an acoustic sensor, detecting utterance of a key phrase via data from the acoustic sensor, and based upon the selected data from the acoustic sensor and also on other environmental sensor data collected at different times than the selected data from the acoustic sensor, determining a probability that the key phrase was spoken by an identified user. The method further includes, if the probability meets or exceeds a threshold probability, then performing an action on the computing device.