A method of fabricating a semiconductor device includes following steps. A substrate is provided, wherein a first dielectric layer having a trench therein is formed on the substrate, a source/drain region is formed in the substrate at two sides of the trench, and a second dielectric layer is formed on the substrate in the trench. A first physical vapor deposition process is performed to form a Ti-containing metal layer in the trench. A second physical vapor deposition process is performed to form an Al layer on the Ti-containing metal layer in the trench. A thermal process is performed to anneal the Ti-containing metal layer and the Al layer so as to form a work function metal layer. A metal layer is formed to fill the trench.