A method for long-term reference prediction video compression, includes: dividing a viewing area into a plurality of viewing sections; measuring an elevation and an azimuth of a field of view (fov) in one or more of the viewing sections; determining a storage location of a video frame associated with the field of view based on the measured elevation and azimuth. A long-term reference prediction video compression encoder, includes: a buffer for storing long-term video frames; a detector to measure an elevation and an azimuth of a field of view associated with a video frame; a buffer manager to determine, based on the measured elevation and azimuth, to store data associated with the video frame in the long-term frame buffer.