A memory device comprises a stack of conductive strips separated by insulating layers on a substrate, and a vertical channel structure disposed in a hole through the stack of conductive strips to the substrate. A vertical channel structure is disposed in a hole through the stack of conductive strips to the substrate. Charge storage structures are disposed at cross points of the conductive strips and the vertical channel structure, the charge storage structures including multiple layers of materials. The insulating layers have sidewalls recessed from the vertical channel structure. A charge storage layer of the multiple layers of materials of the charge storage structures lines sidewalls of the insulating layers. Dielectric material is disposed between the vertical channel structure and the charge storage layer on sidewalls of the insulating layers.