A semiconductor memory device includes a substrate, a first stack, a plurality of first columnar portions, a second stack, a plurality of second columnar portions, and a third stack. In the first stack, first conductive layers and first insulating layers are alternately stacked in a thickness direction of the substrate. Each of the plurality of first pillars extends inside the first stack in the thickness direction. In the second stack, second conductive layers and second insulating layers are alternately stacked in the thickness direction. Each of the plurality of second pillars extends inside the second stack in the thickness direction. The third stack is positioned between the first stack and the second stack in the first direction. In the third stack, third insulating layers and fourth insulating layers including a material different from a material of the third insulating layer are alternately stacked in the thickness direction of the substrate.