A method and apparatus of video coding using block partitioning process including a binary tree partitioning process are disclosed. The block partitioning structure corresponding to the block partitioning process for a block of video data is derived by parsing a first indicator associated with a current block and determining whether the binary tree partitioning process applies a binary tree partition to the current block based on a value of the first indicator. When it is determined that the binary tree partition is applied to the current block, the current block is always split into two primary sub-blocks with same size or two primary sub-blocks with different sizes. The block partitioning structure represents partitioning the block of video data into final sub-blocks. The block of video data is decoded based on the final sub-blocks decoded according to the block partitioning structure derived.