An alternating stack of insulating layers and sacrificial material layers are formed over a substrate. Memory stack structures are formed through the alternating stack. A backside trench is formed and the sacrificial material layers are replaced with electrically conductive layers. After formation of an insulating spacer in the trench, an epitaxial pedestal structure is grown from a semiconductor portion underlying the backside trench. A source region is formed by introducing dopants into the epitaxial pedestal structure and an underlying semiconductor portion during and/or after epitaxial growth. Alternatively, the backside trench can be formed concurrently with formation of memory openings. An epitaxial pedestal structure can be formed concurrently with formation of epitaxial channel portions at the bottom of each memory opening. After formation and subsequent removal of a dummy trench fill structure in the backside trench, a source region is formed by introducing dopants into the epitaxial pedestal structure.