A method includes forming a semiconductor fin protruding higher than top surfaces of isolation regions. The isolation regions extend into a semiconductor substrate. A portion of the semiconductor fin is etched to form a trench, which extends lower than bottom surfaces of the isolation regions, and extends into the semiconductor substrate. The method further includes filling the trench with a first dielectric material to form a first fin isolation region, recessing the first fin isolation region to form a first recess, and filling the first recess with a second dielectric material. The first dielectric material and the second dielectric material in combination form a second fin isolation region.