A system and method are disclosed for identifying a user based on the classification of user movement data. An identity verification system receives a sequence of motion data from a mobile device operated by a target user. From the sequences of motion data, the identity verification system identifies a plurality of identity blocks representing different movements performed by the target user and encodes a set of signature sequences from each identity block into a feature vector. Each feature vector is input to a confidence model to output an identity confidence value for an identity block. An identity confidence value describes a confidence that the movement in the identity block was performed by the target user. The identity confidence value is compared to an operational security threshold and if identity confidence value is above the threshold, the target user is granted access to an operational context.