A stream based memory access system for a video processor for executing video processing operations. The video processor includes a scalar execution unit configured to execute scalar video processing operations and a vector execution unit configured to execute vector video processing operations. A frame buffer memory is included for storing data for the scalar execution unit and the vector execution unit. A memory interface is included for establishing communication between the scalar execution unit and the vector execution unit and the frame buffer memory. The frame buffer memory comprises a plurality of tiles. The memory interface implements a first sequential access of tiles and implements a second stream comprising a second sequential access of tiles for the vector execution unit or the scalar execution unit.