A modular video server includes a Video Blade module and a separate physical layer module. The Video Blade performs video processing functions, such as video encoding and decoding, resizing, slow motion, and color space conversion, and the physical layer module implements the physical layer for input and output of video from devices external to the video server. The modular design permits the type of Video Blade and the type of physical layer module to be specified independently of each other, with each type of Video Blade being compatible with each type of physical layer module. This reduces the effort and cost of providing new video servers that support evolving video processing requirements and new video transmission standards.