An USB matrix switch system provided for a plurality of USB devices shared with a plurality of hosts is disclosed. The system comprises: (1) a plurality of DP PHYs, respectively, corresponding to the USB devices so as to transfer those series signal received from the USB devices to parallel signals; (2) a plurality of UP PHYs, respectively, corresponding to the hosts for transferring signal received therefrom to parallel signals or doing the reversal operation; (3) a plurality of hubs, respectively, corresponding to the UP PHYs, each hub has a digital repeater, a TT, an up router, a down router, and a hub controllers; (4) a signal fetching monitor for fetching a control signal provided by a user; (5) a plurality of signal control switch modules having a down stream port controller, a disconnection emulator, provided for simulating a disconnection status while the down stream port controller is empty, and a select MUX provided for switching a selected USB device to a selected hub in accordance with content of the hot key.