A method of handling component carrier activation and deactivation for a mobile device capable of receiving and transmitting on a plurality of component carriers in a wireless communication system is disclosed. The method includes starting a deactivation timer for a component carrier of the plurality of component carriers when the component carrier is activated, wherein the deactivation timer provides a period of time for component carrier activation, performing a Hybrid Automatic Repeat Request (HARQ) process on the component carrier, and extending activation time of the component carrier when a retransmission of the HARQ process is not finished before deactivation of the component carrier.