Aspects of the disclosure include an escalated authentication system based on user behavior patterns. A user's behavior pattern on a device is collected and/or learned. The collected or learned pattern can be compared to subsequent behavior patterns to determine whether the current user is genuine or suspicious. Users deemed suspicious are subject to increased authentication requirements, often on-the-fly.