Features of the present disclosure solve the above-identified problem by implementing remote localization techniques that enable coordination between multi-user display devices. Specifically, the remote localization techniques identify user location (or “position”) in the virtual reality (VR)/augmented reality (AR) scene using a key-frame that is subset of available information. Thus, when a first display device is uncertain regarding its position within a VR/AR scene, the first display may generate, for example, a single key-frame that is shared between one or more second display devices such that the receiving display device(s) may locate the key-frame with respect to a spatial anchor within the client's map, identify the user's location within the VR/AR scene, and transmit the location back to the first HMD device that created the key-frame. Based on the location information, the first HMD device may synchronize with the second HMD device within a shared scene or map.