A plug electrode is subject to etch back to remain in a contact hole and expose a barrier metal on a top surface of an interlayer insulating film. The barrier metal is subject to etch back, exposing the top surface of the interlayer insulating film. Remaining element structures are formed. After lifetime is controlled by irradiation of helium or an electron beam, hydrogen annealing is performed. During the hydrogen annealing, the barrier metal is not present on the interlayer insulating film covering a gate electrode, enabling hydrogen atoms to reach a mesa part, whereby lattice defects generated in the mesa part by the irradiation of helium or an electron beam are recovered, recovering the gate threshold voltage. Thus, predetermined characteristics of a semiconductor device having a structure where a plug electrode is provided in a contact hole, via barrier metal are easily and stably obtained when lifetime control is performed.