A disposable gate structure and a gate spacer are formed on a semiconductor substrate. A disposable gate material portion is removed and a high dielectric constant (high-k) gate dielectric layer and a metal nitride layer are formed in a gate cavity and over a planarization dielectric layer. The exposed surface portion of the metal nitride layer is converted into a metal oxynitride by a surface oxidation process that employs exposure to ozonated water or an oxidant-including solution. A conductive gate fill material is deposited in the gate cavity and planarized to provide a metal gate structure. Oxygen in the metal oxynitride diffuses, during a subsequent anneal process, into a high-k gate dielectric underneath to lower and stabilize the work function of the metal gate without significant change in the effective oxide thickness (EOT) of the high-k gate dielectric.