A method of decoding a video bitstream is disclosed. The method generally includes the steps of (A) buffering the video bitstream in a main memory, the video bitstream comprising a first residual block based on a first motion compensated block, the first motion compensated block having been generated by a subpixel motion compensation using an N-tap interpolation on a plurality of first reference samples, (B) copying a first subset of the first reference samples from the main memory to an internal memory, (C) generating a first decode block using the subpixel motion compensation with an M-tap interpolation on the first subset in the internal memory, and (D) reconstructing a first original block by adding the first residual block to the first decode block.