An apparatus and operating method of a Relay Station (RS) and a Base Station (BS) controller for a multi-hop relay communication in a broadband wireless communication system is provided. The method of the RS includes, when receiving at least one traffic Media Access Control (MAC) Protocol Data Unit (PDU) from a Mobile Station (MS) accessing the RS, extracting a Layer-3(L3) packet from the at least one traffic MAC PDU, generating a relay MAC PDU including the L3 packet, and transmitting the MAC PDU to the BS through an RS Service Flow (SF) which supports Quality of Service (QoS) required by an MS SF carrying the L3 packet.