This disclosure relates generally to computer vision, and more particularly to method and system for tracking objects within a video. In one embodiment, a method for tracking objects within a video is disclosed. The method includes receiving one or more regions of interest (ROIs) corresponding to one or more objects in an initial frame of the video, extracting a set of scale and rotation invariant interest data points in each of the ROIs, clustering the set of scale and rotation invariant interest data points in a ROI into a set of clusters based on corresponding locations in the ROI, determining an optimal set of interest data points from each of the set of clusters based on corresponding feature response values and spread values, and initiating tracking of the optimal set of interest data points in subsequent frames of the video to track the one or more objects in the video.