Technologies are described herein for buffer management during real-time streaming. A video frame buffer stores video frames generated by a real-time streaming video capture device. New video frames received from the video capture device are stored in the video frame buffer prior to processing by a video processing pipeline that processes frames stored in the video frame buffer. A buffer manager determines whether a new video frame has been received from the video capture device and stored in the video frame buffer. When the buffer manager determines that a new video frame has arrived at the video frame buffer, it then determines whether the video processing pipeline has an unprocessed video frame. If the video processing pipeline has an unprocessed video frame, the buffer manager discards the new video frame stored in the video frame buffer or performs other processing on the new video frame.