There is provided an operation processing device including an input direction acquisition unit configured to acquire information indicating an input direction according to a direction generated in response to an operation indicating the direction which is performed on an input unit to which the operation is input, an output direction indication unit configured to indicate, to a display unit configured to display an image, an output direction that is a direction in which the image of the display unit is changed in response to the operation on the input unit, and a determination unit configured to determine an orientation of the output direction with respect to the input direction according to a relative position relationship between the input unit and the display unit.