Image feature trackability ranking systems and methods are disclosed. A method of establishing a trackability ranking order from tracked image features within a training video sequence at a tracking analysis device includes establishing a tracking region within the training video sequence using a feature detection algorithm. Trajectories of tracked image features within the tracking region are compiled using a feature tracking algorithm. Saliency metrics are assigned to each one of the trajectories of tracked image features based on one or more feature property measurements within the tracking region, and a trackability ranking algorithm that is a function of the saliency metrics and a defined feature trajectory ranking associated with the training video sequence is determined, the trackability ranking algorithm being usable for ranking, based on trackability, tracked image features within another video sequence.