A method of forming a gate structure of a semiconductor device including depositing a high-k dielectric layer over a substrate is provided. A dummy metal layer is formed over the high-k dielectric layer. The dummy metal layer includes fluorine. A high temperature process is performed to drive the fluorine from the dummy metal layer into the high-k dielectric layer thereby forming a passivated high-k dielectric layer. Thereafter, the dummy metal layer is removed. At least one work function layer over the passivated high-k dielectric layer is formed. A fill metal layer is formed over the at least one work function layer.