Embodiments of the present invention relate to object tracking in video. In an embodiment, a computer-implemented method tracks an object in a frame of a video. An adaptive term value is determined based on an adaptive model and at least a portion of the frame. A pose constraint value is determined based on a pose model and at least a portion the frame. An alignment confidence score is determined based on an alignment model and at least a portion the frame. Based on the adaptive term value, the pose constraint value, and the alignment confidence score, an energy value is determined. Based on the energy value, a resultant tracking state is determined. The resultant tracking state defines a likely position of the object in the frame given the object's likely position in a set of previous frames in the video.