Embodiments of 3D memory devices having an inter-deck plug and methods for forming the same are disclosed. In an example, a 3D memory device includes a substrate, a first memory deck including interleaved conductor and dielectric layers above the substrate, a second memory deck including interleaved conductor and dielectric layers above the first memory deck, and a first and a second channel structure each extending vertically through the first or second memory deck. The first channel structure includes a first memory film and semiconductor channel along a sidewall of the first channel structure, and an inter-deck plug in an upper portion of the first channel structure and in contact with the first semiconductor channel. A lateral surface of the inter-deck plug is smooth. The second channel structure includes a second memory film and semiconductor channel along a sidewall of the second channel structure. The second semiconductor channel is in contact with the inter-deck plug.