Methods for determining reference type of each image unit and adjusting the corresponding lambda table of the reference type are disclosed. Embodiments according to the invention are used to improve the quality of video compression or reduce the requirement of memory buffer, memory power or computation power. The reference type is determined based on the encoder system information or image unit information. The frame/slice type structure of a video sequence is adjusted according to the image unit information of encoded frames or together with input frames. By fine-tuning the mode decision process, the coding efficiency can be improved. The mode decision process is modified by adaptively adjusting the lambda table. The lambda table is adaptively determined according to the conventional image unit type (such as Intra coded, predicted or bi-directional predicted type) and the reference type determined.