A paper sheet processing apparatus capable of preventing authentication accuracy from degrading even when a state change occurs due to attachment of water or the like. The paper sheet processing apparatus includes: a light receiving unit which receives transmitted and reflected lights from the paper sheet radiated by light emitting parts; a converter which converts the transmitted and reflected lights received by the light receiving part per pixel into data including color information having brightness; a RAM which stores transmitted and reflected light images received by the light receiving unit; an authenticity judgment processing part which determines the authenticity of the paper sheet based on the images stored in the RAM; and a determining part which excludes, based on a comparison result between brightness of the transmitted and reflected images in the predetermined area.