Embodiments are directed towards online content classification that includes training a machine learning system. A batch of data items may be randomly selected from unlabeled test data. The batch of data items may be communicated to a client computer enabling a user to label each data item based on the contents of each data item. These labeled data items may be employed to train the machine learning system. While a classification result score is less than a threshold value, iteration may be performed to train the machine learning system. For each iteration another batch of data items may be selected from the unlabeled test data. This batch of data items may be classified using the machine learning system. The batch of classified data items may be communicated back to the client computer to be labeled by the user.