A method of manufacturing the semiconductor device includes: (a) providing a substrate having a semiconductor layer; (b) forming a first insulating film over an insulating layer so as to cover the semiconductor layer; (c) forming an opening extending through the first insulating film and reaching the semiconductor layer; (d) forming, over the semiconductor layer exposed at a bottom surface of the opening, a semiconductor portion having a thickness smaller than that of the first insulating film over the semiconductor layer by a selective epitaxial growth method; (e) forming a second insulating film over the first insulating film and the semiconductor portion; (f) removing the second insulating film from over the first insulating film, while leaving the second insulating film in the opening; (g) removing a semiconductor particle formed over the first insulating film in the (d); and (h) forming a third insulating film over the first insulating film.