Systems and methods are provided herein for performing an action based on a feature in a media asset. In many media assets, specific features appear at portions of a program that users find interesting. These features can be compared with a database of stored objects that commonly appear in media assets to determine the object corresponding to the feature. The context of the object in the media asset then may be determined so that an appropriate action is selected for the system to take.