A method includes forming a gate stack on a middle portion of s semiconductor fin, and forming a first gate spacer on a sidewall of the gate stack. After the first gate spacer is formed, a template dielectric region is formed to cover the semiconductor fin. The method further includes recessing the template dielectric region. After the recessing, a second gate spacer is formed on the sidewall of the gate stack. The end portion of the semiconductor fin is etched to form a recess in the template dielectric region. A source/drain region is epitaxially grown in the recess.