A nonvolatile semiconductor memory device includes a first stack unit with a first selection transistor and a second selection transistor formed on a semiconductor substrate and a second stack unit with first insulating layers and first conductive layers stacked alternately on the upper surface of the first stack unit. The second stack unit includes a second insulating layer formed in contact with side walls of the first insulating layer and the first conductive layer, a charge storage layer formed in contact with the second insulating layer for storing electrical charges, a third insulating layer formed in contact with the charge storage layer, and a first semiconductor layer formed in contact with the third insulating layer so as to extend in a stacking direction, with one end connected to one diffusion layer of the first selection transistor and the other end connected to a diffusion layer of the second selection transistor.