The current subject matter describes a method and system of detecting frauds or anomalous behavior. The procedures include extracting characteristics from a dataset to generate words and documents, executing a topic model to obtain the respective probabilities of appearance of a document in each latent archetype, dividing the dataset into a plurality of subsets based upon the archetypes. The formed subsets are further utilized to estimate the quantiles and calculate scores using a self-calibrating outlier model. The score of each new transaction is determined based on a single archetype or based on the sum of weighted scores determined from all the archetypes and associated statistics. Such methods are superior to a simple self-calibration outlier model without an LDA archetype. The detection system with the LDA archetypes and self-calibrating outlier model is implemented with the sliding window technique incorporating new transactions into the topic model and it is capable of operating in real-time for the purpose of identifying frauds and outliers.