A Scalable Video Coding (SVC) process is provided for scalable video coding that takes into account color gamut primaries along with spatial resolution. The process provides for re-sampling using video color data obtained from an encoder or decoder process of a base layer (BL) in a multi-layer system to enable improved encoding and decoding in an enhancement layer (EL) or higher layers taking into account color conversion between layers. Examples of applicable SVC include MPEG-4 Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC). With the SVC process, video data expressed in one color gamut space can be used for prediction in encoding with a possibly different color space, and accommodation for different spatial resolution and bit-depth can be made as well.