Systems and methods relying on recognition of a pattern in a data stream, such as detecting a hotword in an audio data stream are sensitive to latency (e.g., response time). To reduce power consumption, a low power processor may be used in combination with a higher power speech recognition device. When the hotword is detected by the low power signal processor, the primary speech recognition device is signaled to wake up and begin emptying a buffer storing the hotword and subsequent audio data. Latency is the delay incurred to recognize the hotword and begin emptying the buffer. To catch-up and reduce the latency, the buffer is drained at a faster rate than the buffer is filled until a latency reduction trigger is received. The latency reduction trigger is generated when the latency has been reduced to a predetermined level.