A method of video coding with reduced implementation cost by reusing transform coefficient buffer palette for palette coding is disclosed. If the current prediction mode is an Intra prediction mode or the Inter prediction mode, information related to transform coefficients for prediction residual of the current block resulted from Intra prediction or Inter prediction is stored in the transform coefficient buffer. If the current prediction mode is the palette coding mode, information related to palette data associated with the current block is stored in the transform coefficient buffer. The current block is then encoded or decoded based on the information related to the transform coefficients if the current block is coded in the Intra prediction mode or the Inter prediction mode, or the information related to the palette data stored in the transform coefficient buffer if the current prediction mode is the palette coding mode.