An alternating stack of insulating layers and sacrificial material layers is formed over a substrate. A dielectric collar structure can be formed prior to formation of an epitaxial channel portion, and can be employed to protect the epitaxial channel portion during replacement of the sacrificial material layers with electrically conductive layers. Exposure of the epitaxial channel portion to an etchant during removal of the sacrificial material layers is avoided through use of the dielectric collar structure. Additionally or alternatively, facets on the top surface of the epitaxial channel portion can be reduced or eliminated by forming the epitaxial channel portion to a height that exceeds a target height, and by recessing a top portion of the epitaxial channel portion. The recess etch can remove protruding portions of the epitaxial channel portion at a greater removal rate than a non-protruding portion.