A nonvolatile semiconductor memory device includes: a semiconductor substrate; a stacked body provided on the semiconductor substrate, the stacked body having electrode films and insulating films being alternately stacked; a first and second semiconductor pillars; and a first and second charge storage layers. The first and second semiconductor pillars are provided inside a through hole penetrating through the stacked body in a stacking direction of the stacked body. The through hole has a cross section of an oblate circle, when cutting in a direction perpendicular to the stacking direction. The first and second semiconductor pillars face each other in a major axis direction of the first oblate circle. The first and second semiconductor pillars extend in the stacking direction. The first and second charge storage layers are provided between the electrode film and the first and second semiconductor pillars, respectively.