A caster with a brake structure comprises a pivot plate. The pivot plate is allowed to pivot so as to be inserted into a corresponding positioning recess in an inner side of a wheel to stop the rotation of the wheel. Furthermore, while pivoting, the pivot plate will press the lifting block to move downwards, and then the lifting block will push the slide block to move horizontally, and finally the slide block will push the anti-slip block to press against the inner side of the wheel, thus improving the braking effect.