A method and apparatus for decoding two-level scanned transform coefficients corresponding to a transform unit (TU) are disclosed. The TU is divided into sub-blocks and the transform coefficients of the TU are scanned across the sub-blocks according to a first scan pattern, and each sub-block is scanned according to a second scan pattern. In one embodiment, the sub-blocks of the transform coefficients received from the variable length decoding are stored in an inverse scan buffer (or TC buffer) and the transform coefficients are retrieved from the inverse scan buffer row-by-row or column-by-column in a selected direction after a corresponding row or column of the transform coefficients is fully received. In a system incorporating an embodiment of the present invention, at least a leading row or a leading column of the transform coefficients is available in the selected direction before a last sub-block of the transform coefficients arrives.