An apparatus and a method of recognizing a gesture uses an infrared ray. The apparatus for recognizing a gesture, includes a sensing unit which detects a gesture using an infrared sensor to obtain a sensing value from the sensing result; a control unit which performs gesture recognition to which an intention of a user is reflected in accordance with a predetermined recognizing mode based on the obtained sensing value; and a storing unit which stores the predetermined recognizing mode when the gesture recognition set in advance by the user is performed. The predetermined recognizing mode includes a first recognizing mode in which the gesture is directly recognized and a second recognizing mode in which the gesture is recognized after recognizing a hold motion for determining start of the gesture recognition.