Provided is an optical transmission system that includes an optical transmitter, and an optical receiver. The optical transmitter includes: a signal coding unit configured to perform nonlinear trellis coding, which corresponds to nonlinear calculation, on a symbol sequence; and a modulator configured to modulate the symbol sequence subjected to the nonlinear trellis coding by the signal coding unit, and transmit the modulated symbol sequence to the optical receiver. The optical receiver includes: a light receiving unit configured to receive an optical signal transmitted from the optical transmitter, and convert the received optical signal into an electrical signal; and a digital signal processing unit configured to perform digital signal processing on the electrical signal to reconstruct the symbol sequence.