An approach is provided to correct natural language processing (NLP) annotators. The approach operates by receiving a set of supporting text noted by a user in response to the user identifying an error to a user question in a question answering (QA) system. The set of supporting text includes one or more text passages from which a correct answer should have been generated by the QA system. The QA system generates one or more scored candidate corrections with each of the scored candidate corrections is based on the identified error and the set of supporting text. The user can then select one or more of the scored candidate corrections as a confirmed correction to the error. The confirmed corrections are then applied to a corpus that is utilized by the QA system when answering questions.