A method and apparatus for significance map coding for 4×4 TUs (transform units) and 8×8 TUs of video data are disclosed. The method comprises receiving at least one context selection table for a first TU associated with a first color component and generating a derived context selection table for a second TU associated with a second color component from said at least one context selection table. The first TU has a first size comprising of 4×4 and 8×8. The second TU has a second size comprising of 4×4 and 8×8. The second size is different from the first size, the second color component is different from the first color component, or both the second size is different from the first size and the second color component is different from the first color component. The derived context selection table is then used for significance map processing of the second TU.