An apparatus including a local shared memory and a processor. The local shared memory and the processor may be connected as a circuit. The circuit may be configured to communicate with an external memory device. The local shared memory may comprise a plurality of cache blocks. Each of the cache blocks generally corresponds to one pane of a current horizontal strip of a warped image region to be generated from unwarped image data stored in the external memory. A size of the cache blocks may be set to provide a sufficient amount of data to correct distortion for a corresponding section of the current horizontal strip. The circuit may be further configured such that each pixel of the unwarped image data stored in the external memory is fetched only once.