A method includes forming a first semiconductor fin and a second semiconductor fin in an n-type Fin Field-Effect (FinFET) region and a p-type FinFET region, respectively, forming a first dielectric fin and a second dielectric fin in the n-type FinFET region and the p-type FinFET region, respectively, forming a first epitaxy mask to cover the second semiconductor fin and the second dielectric fin, performing a first epitaxy process to form an n-type epitaxy region based on the first semiconductor fin, removing the first epitaxy mask, forming a second epitaxy mask to cover the n-type epitaxy region and the first dielectric fin, performing a second epitaxy process to form a p-type epitaxy region based on the second semiconductor fin, and removing the second epitaxy mask. After the second epitaxy mask is removed, a portion of the second epitaxy mask is left on the first dielectric fin.