A method and apparatus for transform coefficient coding of image and video data for a video encoder or decoder are disclosed. The method determines a CG (coding group) size adaptively based on the current TU. The current TU is divided into one or more current CGs (coding groups) according to the CG size. Bins associated with the coefficients of the current TU are then encoded or decoded according to a selected CG scan path through the current TU and one or more coefficient scan paths within each CG. If TU width is larger or smaller than TU height for the current TU, then CG width is selected to be larger or smaller than CG height accordingly. According to another method, all bypass-coded bins associated with coefficients in the current TU are grouped to improve the parsing throughput rate.