A method includes removing a dummy gate to leave a trench between gate spacers, forming a gate dielectric extending into the trench, depositing a metal layer over the gate dielectric, with the metal layer including a portion extending into the trench, depositing a filling region into the trench, with the metal layer have a first and a second vertical portion on opposite sides of the filling region, etching back the metal layer, with the filling region at least recessed less than the metal layer, and remaining parts of the portion of the metal layer forming a gate electrode, depositing a dielectric material into the trench, and performing a planarization to remove excess portions of the dielectric material. A portion of the dielectric material in the trench forms at least a portion of a dielectric hard mask over the gate electrode.