An audio signal decoder for providing a decoded multi-channel audio signal representation on the basis of an encoded multi-channel audio signal representation has a time warp decoder configured to selectively use individual audio channel specific time warp contours or a joint multi-channel time warp contour for a reconstruction of a plurality of audio channels represented by the encoded multi-channel audio signal representation. An audio signal encoder for providing an encoded representation of a multi-channel audio signal has an encoded audio representation provider configured to selectively provide an audio representation having a common time warp contour information, commonly associated with a plurality of audio channels of the multi-channel audio signal, or an encoded audio representation having individual time warp contour information, individually associated with the different audio channels of the plurality of audio channels, in dependence on an information describing a similarity or difference between time warp contours associated with the audio channels of the plurality of audio channels.