Systems and methods for adaptive Simultaneous Localization and Mapping (SLAM) using world-facing cameras in virtual, augmented, and mixed reality (xR) applications. An Information Handling System (IHS) may include a processor and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution, cause the IHS to: select at least one region-of-interest (ROI) within an image captured by a camera mounted on a Head-Mounted Display (HMD) worn by a user in a physical environment during execution of a virtual, augmented, or mixed reality (xR) application; and manage utilization of an IHS resource, at least in part, by creating a map of the physical environment and tracking a location of the HMD using the at least one ROI to the exclusion of another portion of the image.