Approaches presented herein enable automatically removing a potential tag of a subject depicted in a multimedia item based on a context of the multimedia item and one or more preferences of the subject being tagged. Specifically, a request to tag a subject in the multimedia item is received, and the multimedia item is analyzed to determine a context. The context is compared to the preferences related to the subject to determine a variance. If a variance exists, a removal action related to the tag is performed, such as disallowing or removing the potential tag.