A method is provided for fabricating a semiconductor device that includes providing a semiconductor substrate, forming a transistor in the substrate, the transistor having a gate structure that includes a dummy gate structure, forming an inter-layer dielectric (ILD), performing a first chemical mechanical polishing (CMP) to expose a top surface of the dummy gate structure, removing a portion of the ILD such that a top surface of the ILD is at a distance below the top surface of the dummy gate structure, forming a material layer over the ILD and dummy gate structure, performing a second CMP on the material layer to expose the top surface of the dummy gate structure, removing the dummy gate structure thereby forming a trench, forming a metal layer to fill in the trench, and performing a third CMP that substantially stops at the top surface of the ILD.