A reference direction determination unit 44 detects, using an input image, a scene change, a flash scene, or a monotonous luminance variation section. In a case where an encoding target image is the scene change or flash scene, the reference direction determination unit 44 determines its mode as a fixed reference direction mode in which a reference direction in a screen is fixed uniformly. Further, in a case where the encoding target image is the monotonous luminance variation section, that is, a section where luminance of an entire screen monotonously increases or monotonously decreases, the reference direction determination unit 44 determines as the fixed reference direction mode in which the reference direction in the screen is fixed uniformly. A motion prediction/compensation unit 42 performs inter prediction processing according to a determination result of the reference direction determination unit 44, thereby enabling efficient encoding while suppressing deterioration of image quality.