A virtual reality (VR) system comprising a head-mounted display (HMD) and handheld controller set is enhanced to provide a more realistic end user VR experience, e.g., for boxing or other interactive training. In this approach, and in lieu of simply establishing a boundary area for the VR experience, the user also maps a position of a real-world object into a reference frame of the VR environment. This mapping is facilitated using the handheld controller itself, e.g., as positioned in a backwards-facing manner on the user's forearm. The real-world object is then simulated in the 3D VR environment as rendered by the VR HMD, and the user interacts with the real-world object (or its simulation) to provide a more enjoyable and useful interactive experience.