A system and method of compressing a video signal can include the steps of: receiving a video signal, the video signal including frames; analyzing, for each frame, the video signal on a macroblock-by-macroblock level; determining whether to downsample a macroblock residual for each of the macroblocks; selectively downsampling a macroblock residual for some of the macroblocks; and coding the macroblocks. A system and method of decompressing a video signal can include the steps of receiving a compressed video signal, the video signal including frames; analyzing, for each frame, the video signal on a macroblock-by-macroblock level; determining whether to upsample a macroblock residual for each of the macroblocks; selectively upsampling a macroblock residual for some of the macroblocks; and decoding the macroblocks.