A three-dimensional memory device includes an alternating stack of word-line-isolation insulating layers and word-line-level electrically conductive layers located over a substrate, a plurality of drain-select-level electrodes that are laterally spaced apart from each other overlying the alternating stack, memory stack structures containing a respective vertical semiconductor channel laterally surrounded by a respective memory film and vertically extending through the alternating stack and the plurality of drain-select-level electrodes, inter-select-gate electrodes located between a respective neighboring pair of the drain-select-level electrodes, and inter-select-gate dielectrics located between each of the inter-select-gate electrodes and a neighboring one of the drain-select-level electrodes. The inter-select-gate electrodes are not electrically connected to the drain-select-level electrodes.