An operation method of a first communication node in a communication system includes receiving a signal from a second communication node; demodulating the signal to obtain LLR values; calculating a first codeword based on the LLR values; selecting error patterns from among all error patterns based on Hamming weights; generating second codewords by applying the first codeword to each of the selected error patterns; and determining a codeword having a highest similarity to the first codeword among the second codewords as an optimal codeword.