A satellite communication system and a method of cancelling interference in the satellite communication system are disclosed. The method of cancelling interference in the satellite communication system may include receiving a signal by a central station from a terminal via a satellite transponder, estimating, from the signal, a nonlinear characteristic associated with a central station signal transmitted from the central station, and cancelling an interference signal from the signal using the estimated nonlinear characteristic.