A video processing method includes dividing a coding tree unit into one or more coding units using a quadtree plus binary tree division method, dividing a coding unit into one or more sub-blocks, determining a specific neighboring image block of the coding unit having a reference image same as a co-located reference image of the coding unit, determining a related reference block of the coding unit in the co-located reference image according to a motion vector of the specific neighboring image block, dividing the related reference block into several related reference sub-blocks, determining a scaling factor of a motion vector of the related reference sub-block in different manners according to whether that motion vector points to a short-term reference image or a long-term reference image, scaling that motion vector using the scaling factor, and performing prediction for the coding unit according to the scaled motion vector.