A method of encoding a video signal representing a sequence of pictures, which comprises receiving a current picture for encoding, forming a temporal prediction of the current picture or a part thereof from a first reference picture or a part thereof associated with the current picture, and further comprising associating a second reference picture or a part thereof with the current picture for forming a temporal prediction of the current picture or a part thereof based on a similarity measure. The similarity measure is selected from a group which includes a measure indicative of the similarity between the first reference picture or a part thereof and the second reference picture or a part thereof and a measure indicative of a similarity between the current picture or a part thereof and the second reference picture or a part thereof. A corresponding video decoding method, video encoder and video decoder are presented.