A semiconductor device includes an N-channel transistor having an N-type gate electrode and a P-channel transistor having a P-type gate electrode which are formed on a semiconductor substrate. The P-type gate electrode includes a first silicon layer formed as the lowest layer, and doped with a P-type impurity; a second silicon layer formed on the first silicon layer; and a metal containing layer formed on the second silicon layer. The N-type gate electrode includes a third silicon layer formed as the lowest layer and doped with an N-type impurity; a fourth silicon layer formed on the third silicon layer; and a metal containing layer formed on the fourth silicon layer. At least one of the second silicon layer and the fourth silicon layer is doped with no impurity or an impurity of a conductive type opposite to that of the impurity in a corresponding one of the first silicon layer and third silicon layer.