A method of improving occluded edge quality in an augmented reality system includes a data input step of using a camera set to capture a scene and an object set in the scene to obtain an original image; an occluded region extracting step of obtaining an extracted image by performing a difference operation according to a first image and a second image, which are obtained according to depth buffer calculations according to first and second virtual planes set in the scene; and an occluded image generating step of inserting a virtual object into the original image according to the extracted image, the original image and a third virtual plane, which is set between the first and second virtual planes, to obtain an occluded image with a smoothened occluded edge.