A method and apparatus for processing a current macro-block of a current frame for motion compensation based on reference data from a reference frame using a sliding window cache to cache the reference data are disclosed. The method steps comprise processing a current sub-block of the current macro-block for motion compensation, determining the reference data for the current sub-block based on a reference sub-block in the reference frame, determining whether the reference data is within a sliding window, if the reference data is within the sliding window, checking for a valid tag corresponding to the reference data, responsive to the valid tag, retrieving the reference data from within the sliding window cache and processing the reference data for motion compensation.