A method of manufacturing a semiconductor device includes forming a conductive layer over a semiconductor substrate, selectively removing the conductive layer for forming a resistance element and a gate electrode, forming sidewall spacers over sidewalls of the remaining conductive layer, forming a first insulating film containing a nitrogen over the semiconductor substrate having the sidewall spacers, implanting ions in the semiconductor substrate through the first insulating film, forming a second insulating film containing a nitrogen over the first insulating film after implanting ions in the semiconductor substrate through the first insulating film, and selectively removing the first and the second insulating film such that at least a part of the first and the second insulating films is remained over the semiconductor substrate and over the conductive layer.