A scalable video bitstream may have an H.264/AVC compatible base layer (BL) and a scalable enhancement layer (EL), where scalability refers to color bit depth. The SVC standard allows spatial inter-layer prediction, wherein a residual in the EL is generated which is then intra coded. Another spatial intra-coding mode for EL is pure intra coding (I_N×N). The invention discloses a new intra-coding mode and two new inter coding modes, particularly for bit depth scalability. The new intra coding mode uses encoding of the residual between upsampled reconstructed BL and original EL, using mode selection. Two possible modes are residual prediction from BL and additional intra-coding of this residual. The new inter coding modes use also prediction of EL from reconstructed BL. In a first inter coding mode, the residual is encoded using Motion Estimation based on this residual. In a second inter coding mode, the residual is encoded using upsampled motion information from the BL.