According to one embodiment, a semiconductor memory device includes a magnetic tunnel junction (MTJ) element includes a first magnetic layer, a second magnetic layer and a non-magnetic layer between the first and second magnetic layers, a contact layer formed underneath the MTJ element, the contact layer being formed of a first material, and a first layer formed around the contact layer, wherein the first layer in contact with a side surface of the contact layer, has a first width extending parallel to a stacking direction of the MTJ element, and a second width extending perpendicularly to the direction of extension of the first width, the second width being less than the first width.