A system and a method are disclosed for fast pen tracking a low latency display updates on an electronic paper display. Pen input information is received on an electronic paper display that updates at a predetermined display update rate. A line drawing module of the electronic paper display driver determines at least one pixel to activate based on the received pen input information. The at least one pixel is updated independent of the display update rate of the electronic paper display. Active pixel state information is maintained separately for each pixel in real time until the pixel update is complete and the pixel is deactivated. In some embodiments, a future pixel to activate is determined based on the received pen input information. The future pixel is deactivated if pen input information is not received on the activated pixel for a predetermined amount of time.