A video anti-shaking method and a video anti-shaking device are provided. The method includes: obtaining a first GRMV (global relative motion vector) of a current video frame relative to its previous video frame, and a sequence of second GRMVs of reference video frames relative to their respective previous video frames; calculating a sequence of third GRMVs of the reference video frames relative to the current video frame, respectively, based on the first GRMV and the sequence of second GRMVs; and filtering the sequence of third GRMVs to obtain a motion vector for compensating shaking of the current video frame. Camera shaking and intended motion can be distinguished.