Strategies are described for processing image information using a data structure which conveys color information. The color information describes color-related formatting applied to the image information. The data structure can be passed down a video processing pipeline, where each functional component in the pipeline can draw color information from the data structure to improve the accuracy of its processing. Further, each component can supply previously unknown color information to the data structure for use by other (downstream) components in the pipeline. Exemplary fields of the data structure can include one or more of: a video transfer function field; a color primaries field; an image light field; a transfer matrix field; a nominal range field; and a video chroma sampling field. The transfer matrix field can be used to convert image information from one color space to another, such as a luma-related color space to an RGB-related color space. Processing operations can be performed on progressive and linear RGB image information.