The present disclosure relates to an image processing device, an image processing method, a program, and an endoscope system capable of arranging an endoscope so that the endoscope is not fixed to a retractor and has a degree of freedom, and capable of correcting an image deterioration such as a blur that might occur in this case. The image processing device according to an aspect of the present disclosure is the image processing device configured to correct, frame by frame, a motion image having a predetermined frame rate, and the image processing device includes: a marker detecting unit configured to detect a marker provided on an operation tool and photographed in a current frame; a calculation unit configured to calculate a blur correction amount on the basis of a position of the detected marker; and a blur correction unit configured to perform a blur correction on the current frame in accordance with the blur correction amount. The present disclosure can be applied to the endoscope system.