A system for creating composite encoded video from two or more encoded video sources in the encoded domain. In response to user input, a markup language-based graphical layout is retrieved. The graphical layout includes frame locations within a composite frame for at least a first encoded source and a second encoded source. The system either retrieves or receives the first and second encoded sources. The sources include block-based transform encoded data. The system also includes a stitcher module for stitching together the first encoded source and the second encoded source according to the frame locations of the graphical layout to form an encoded frame. The system outputs an encoded video stream that is transmitted to a client device associated with the user. In response to further user input, the system updates the state of an object model and replaces all or a portion of one or more frames of the encoded video stream. The system may be used with MPEG encoded video.