The method of manufacturing the semiconductor device includes amorphizing a first region and a second region of a semiconductor substrate by an ion implantation, implanting a first impurity and a second impurity respectively in the first region and the second region, activating the implanted impurities to form a first impurity layer and a second impurity layer, epitaxially growing a semiconductor layer above the semiconductor substrate with the impurity layers formed on, growing a gate insulating film above the first region and the second region, and forming a first gate electrode above the gate insulating film in the first region and the second gate electrode above the gate insulating film in the second region.