The invention provides a method and a device for controlling video playback. The method includes: extracting playback time of a video clip taken for a predetermined scene from a video to be played; displaying a predetermined control corresponding to the playback time of the video clip taken for the predetermined scene on a video player playing the video; and controlling the video player to adjust a playback progress of the video to the playback time of the video clip taken for the predetermined scene to be played, when a trigger operation for the predetermined control is detected.