Various approaches for motion search refinement in a processing element are discussed. A k/2+L+k/2 register stores an expanded row of an L×L macro block. A k-tap filter horizontally interpolates over the expanded row generating horizontal interpolation results. A transpose storage unit stores the interpolated results generated by the k-tap filter for k/2+L+k/2 entries, wherein rows or columns of data may be read out of the transpose storage unit in pipelined register stages. A k-tap filter vertically interpolates over the pipelined register stages generating vertical interpolation results.