A device includes a semiconductor substrate, and isolation regions extending into the semiconductor substrate. A semiconductor fin is between opposite portions of the isolation regions, wherein the semiconductor fin is over top surfaces of the isolation regions. A gate stack overlaps the semiconductor fin. A source/drain region is on a side of the gate stack and connected to the semiconductor fin. The source/drain region includes an inner portion thinner than the semiconductor fin, and an outer portion outside the inner portion. The semiconductor fin and the inner portion of the source/drain region have a same composition of group IV semiconductors.