A system for controlling a motor of a hybrid vehicle that secures robustness and stability of control by selecting an optimal approximation model according to a driving condition of the motor and determines current order for controlling the motor by using the selected optimal approximation model is disclosed. In particular, a current order generator utilizes a reference current determination module to determine reference currents of the first and second axes, a compensation value determination module to determine compensation values of the reference currents of the first and second axes, and a current determination module to determine the currents of the first and second axes from the reference currents of the first and second axes and the compensation values of the reference currents of the first and second axes in order to control the motor more efficiently.