A torque adjustment device of a torque wrench includes a handle having a passage defined axially therethrough, and a driving head is connected to one end of the handle. A torque adjustment unit is located in the passage of the handle and includes a stationary member, a threaded shaft threadedly extending through the stationary member, a rotary member and a spring. The shaft includes a threaded rod that extends beyond the rotary member. A sleeve is mounted to the threaded shaft. A rotary unit includes a tube which has a mounting section and a threaded section on two ends thereof. By rotating the tube, the rotary member and the threaded shaft are rotated relative to the stationary member so as to compress the spring to set the output torque value. A knob is then threadedly connected to the threaded rod to secure the output torque value.