A semiconductor device with a high-quality epitaxial layer and a method of manufacturing the same. The semiconductor device may include: a substrate; a fin-shaped first semiconductor layer spaced apart from the substrate; a second semiconductor layer at least partially surrounding a periphery of the first semiconductor layer; an isolation layer formed on the substrate, exposing at least a part of the second semiconductor layer, wherein the exposed part of the second semiconductor layer extends in a fin shape; and a gate stack formed on the isolation layer and intersecting the second semiconductor layer.