A semiconductor memory device includes a conductive layer on a source side; a first electrode layer provided on the conductive layer; a second electrode layer provided between the conductive layer and the first electrode layer; a semiconductor layer extending through the first electrode in a first direction from the conductive layer to the first electrode layer; a first semiconductor body provided between the conductive layer and the semiconductor layer, the first semiconductor body including first impurities; and a second semiconductor body provided between the conductive layer and the first semiconductor body, the second semiconductor body including second impurities with a higher concentration than a concentration of the first impurities in the first semiconductor body. A diffusion coefficient of the second impurities in the second semiconductor body is smaller than a diffusion coefficient of the second impurities in the first semiconductor body.