An example video encoder uses a self-adaptive dictionary that only depends on the block location, without requiring real time design/redesign of the dictionary. A mean-removed block may be directly represented using a redundant dictionary including all possible inter-prediction candidates with integer motion vectors (mean-removed). In general the dictionary may also contain some intra-prediction candidates and some pre-designed fixed dictionary atoms. The coefficients may be determined by minimizing the L0 norm of the coefficients subject to a constraint on the sparse approximation error. Using such a self-adaptive dictionary can lead to a very sparse representation, with significantly fewer non-zero coefficients than using the DCT transform on the prediction error. Some example embodiments consistent with the present invention use a modified orthogonal matching pursuit (“OMP”) algorithm to orthonormalize each new chosen atom with respect to all previously chosen and orthonormalized atoms. Each image block is represented by the quantized coefficients corresponding to the orthonormalized atoms.