In accordance with an aspect of the present disclosure, in a method of manufacturing a semiconductor device, a fin structure in which first semiconductor layers and second semiconductor layers are alternately stacked is formed. A sacrificial gate structure is formed over the fin structure. A first cover layer is formed over the sacrificial gate structure, and a second cover layer is formed over the first cover layer. A source/drain epitaxial layer is formed. After the source/drain epitaxial layer is formed, the second cover layer is removed, thereby forming a gap between the source/drain epitaxial layer and the first cover layer, from which a part of the fin structure is exposed. Part of the first semiconductor layers is removed in the gap, thereby forming spaces between the second semiconductor layers. The spaces are filled with a first insulating material.