A coding mode selection method for selecting a coding mode for motion vector estimation is disclosed. The method includes the following steps. First, a first coding mode is selected to perform motion vector estimation for a macroblock having M sub-blocks. M motion vectors corresponding to the M sub-blocks are separately calculated. It is determined whether correlations for the M motion vectors match to a predetermined condition. If not, the first coding mode is selected and the M motion vectors are utilized to perform a motion vector estimation; and if so, a second coding mode is selected and the N motion vectors corresponding to the second coding mode are utilized to perform the motion vector estimation, wherein the N motion vectors are generated by using the M motion vectors and 1≦N