A height adjustment structure for an armrest contains: an accommodation sleeve for accommodating a fixer and a slider, a cap connected with a support arm, a pedestal, and a pull member. The pull member includes an extension extending out of a first window of the accommodation sleeve so as to pull an engagement protrusion to horizontally remove from or engage into one of multiple recesses of the fixer. The fixer includes a groove and two closed slots. The slider includes an extending lid, a second window, two tabs, a cutout, an elongated orifice, and an abutting protrusion. The pull member includes a receiving orifice, a spring, and a tilted orifice. The engagement protrusion abuts against the cutout and inserts into the one recess, and the engagement protrusion includes a column extending from a side thereof and inserted into the elongated orifice via the tilted orifice.