A method and apparatus for Intra prediction of non-square chroma blocks are disclosed. The system uses a block partitioning structure for partitioning a chroma block into one or more final sub-blocks including at least one non-square final sub-block using block partition recursively. The block partition either splits each given block into intermediate sub-blocks using one or more split modes including one binary split mode or applies no splitting to each given block. The block is treated as an initial given block and each intermediate sub-block is treated as one given block in a next recursion. Any given block that is not further split becomes a final sub-block. After the final sub-blocks are determined, non-square Intra prediction is applied to each non-square final sub-block.