A prediction mode for encoding data is selected using a reduced number of rate-distortion cost computations. This reduction in rate-distortion cost computations is caused by filtering the number of potential intra prediction modes based on two criteria. First, the number of potential prediction modes is reduced based on at least one characteristic of a quantization procedure performed during the encoding procedure. Second, the number of potential prediction modes is reduced based on an error value calculated for each of the potential prediction modes.