In a video processing system, a method and system for applying transforms larger than 8×8 and non-rectangular transforms, and generating transform size syntax elements indicative of the transforms for video decoding are provided. The transform size syntax element may be generated by an encoder based on a prediction block size of a video block and the contents of the video block. Further, the transform size syntax element may be generated according to a set of rules to select from 4×4, 8×8, and larger transform sizes during an encoding process. A decoder may perform an inverse transform based on the transform size syntax element and the rules used by the encoder. The transform size syntax element may be transmitted to the decoder as part of the encoded video bitstream.