Methods and apparatus provide for a Scene Detector to optimize the location of scene breaks in a set of video frames. Specifically, the Scene Detector receives a set of video frames and a corresponding content model for each video frame. As the Scene Detector identifies a scene in the set of video frames, the Scene Detector updates statistical predictors with respect to characteristics of that scene's characteristics. The Scene Detector thereby utilizes the updated statistical predictors to identify a video frame that may be the next scene break. The Scene Detector analyzes video frames with respect to the possible next scene break in order to identify the actual second scene break that occurs after the previously identified scene break.