A method includes forming a dummy gate stack, forming a dielectric layer, with the dummy gate stack located in the dielectric layer, removing the dummy gate stack to form a opening in the dielectric layer, forming a metal layer extending into the opening, and etching back the metal layer. The remaining portions of the metal layer in the opening have edges lower than a top surface of the dielectric layer. A conductive layer is selectively deposited in the opening. The conductive layer is over the metal layer, and the metal layer and the conductive layer in combination form a replacement gate.