Scalable techniques for dynamic data encoding and decoding are directed toward a system including a plurality of frame processing units. A main frame processing unit manages frame processing unit resource, dispatches frames to appropriate frame processing units. One or more auxiliary frame processing units encode or decode the non-reference frames dispatched by the main frame processing unit. The main frame processing unit encodes or decodes the reference frames and encodes or decodes non-reference frames if none of the auxiliary frame processing units are available.