Embodiments of the present invention are directed to evaluating the performance characteristics of annotator configurations against text pattern constructs in unstructured text. In a non-limiting embodiment of the invention, unstructured text is received by a processor. A text pattern construct is identified in the unstructured text and a first performance characteristic of an annotator is determined based on the text pattern construct. The text pattern construct is converted to a natural language text and a second performance characteristic of the annotator is determined based on the natural language text. A delta is determined between the first performance characteristic and the second performance characteristic. An alternative annotator configuration is identified for a portion of the unstructured text comprising the text pattern construct.