Provided herein is a semiconductor memory device and a method of operating the semiconductor memory device. The semiconductor memory device includes: a memory cell array including a plurality of word lines; a peripheral circuit coupled to the memory cell array through the plurality of word lines and configured to apply a program voltage to a selected word line of the plurality of word lines during a program operation and apply a pass voltage to unselected word lines of the plurality of word lines; and control logic configured to control the peripheral circuit to apply a first pass voltage to word lines adjacent to the selected word line among the unselected word lines during a first program operation of the program operation and apply a second pass voltage to the word lines adjacent to the selected word line during a second program operation of the program operation.