A system for video scrolling may allow a user to control speed and/or direction of video playback through user inputs. Such user inputs can include swiping one or more fingers on a touch-sensitive surface, clicking and dragging a computer mouse, or other input actions comprising lateral movements. The rate of playback can be configured based on the speed of the input action. In one embodiment, the user may play the video backward by reversing direction of the input action.