An apparatus and method for controlling a virtual monitor in an augmented reality display system are provided. The method for controlling a virtual monitor in an augmented reality display system, including a camera for photographing an image for determining a location of the virtual monitor and for photographing an image of a user's finger, and a graphic device including window images, includes parallel-processing the image for determining a location of the virtual monitor and the window images and mapping the window images to a set location to create the virtual monitor, extracting a finger tip from the image of the user's finger, stencil-buffering an image of the finger tip not to render the image of the finger in the virtual monitor, determining whether the finger tip collides with the stencil-buffered virtual monitor, and controlling an operation of the virtual monitor according to the determination of the finger tip collision.