A method and apparatus to improve motion prediction in video processing systems is introduced. When a motion prediction cache completes requesting data for a current macroblock and enters an into idle state, data comprising one or more reference frames is speculatively requested, with the hope that the requested data are will be needed in a subsequent macroblock. If the speculative data is needed, then it is consumed. However, if the speculative data is not needed, then the correct data must be requested and a price is paid for an extra memory read bandwidth. In case the speculative data is the correct data for the subsequent macroblock, the effective memory read latency is reduced and the decode performance increases. The video decoder becomes more immune to memory read latency.