A method and apparatus of coding using multiple coding modes with multiple color spaces are provided. For the encoder side, a coding mode is selected from a coding mode group. A corresponding color domain is associated with the coding mode and the corresponding color domain is selected from a color-domain group including at least two different color domains. The current coding unit is then encoded in the corresponding color domain using the coding mode. Furthermore, the syntax of the corresponding color domain is signaled in current coding unit syntaxes. The different color domains may include RGB color domain and YCoCg color domain. According to another method, if the midpoint prediction (MPP) mode is selected, a current block is color transformed into another color domain and the MPP coding process is performed in said another color domain.