A pillar-shaped semiconductor memory device includes a silicon pillar, and a tunnel insulating layer, a data charge storage insulating layer, a first interlayer insulating layer, and a first conductor layer, which surround an outer periphery of the silicon pillar in that order, and a second interlayer insulating layer that is in contact with an upper surface or a lower surface of the first conductor layer. A side surface of the second interlayer insulating layer facing a side surface of the first interlayer insulating layer is separated from the side surface of the first interlayer insulating layer with a distance therebetween, the distance being larger than a distance from the side surface of the first interlayer insulating layer to a side surface of the first conductor layer facing the side surface of the first interlayer insulating layer.