A data transmission method in a wireless relay system, and a method of selecting an optimal transmission mode based on a channel capacity of an individual link that is measured by a mobile station. The data transmission method includes: transmitting a first ratio of first partial data of the data from a base station to a mobile station; transmitting a second ratio of second partial data of the data from the base station to a first relay station; and forwarding the second partial data from the first relay station to the mobile station, wherein the first ratio or the second ratio is determined based on any one of a channel capacity of a first link between the base station and the mobile station, and a channel capacity of a second link between the first relay station and the mobile station.