A method of forming isolation layers of a semiconductor memory device. In accordance with an embodiment of the invention, a semiconductor substrate in which trenches are formed is provided. A first dielectric layer is formed over the semiconductor substrate including the trenches. An opening width of the trench is widened by performing a first etch process to remove a part of the first dielectric layer, followed by an annealing process. Fluorine-containing impurities formed in the first dielectric layer as a result of the etching and annealing processes are removed by performing a second etch process. A second dielectric layer is formed over the semiconductor substrate including the first dielectric layer.