Embodiments of mechanisms for forming a semiconductor device are provided. The semiconductor device includes a semiconductor substrate and a metal gate stack formed over the semiconductor substrate. The semiconductor device also includes an insulating layer formed over the semiconductor substrate and surrounding the metal gate stack, wherein the metal gate stack includes a metal gate electrode. The semiconductor device further includes a metal oxide structure formed over the insulating layer and in direct contact with the insulating layer. The metal oxide structure includes an oxidized material of the metal gate electrode.