A method, a system and a recording medium for building environment map are provided. In the method, a plurality of captured images are captured by using a head-mounted display (HMD). Then, a viewing direction and a translation direction of the HMD are calculated, and an importance map corresponding to the captured images is determined according to the viewing direction and the translation direction of the HMD. Finally, the captured images are stitched to generate a panoramic image according to the importance map.