Systems and methods are provided for adding or extracting metadata for video streams. The system receives a video stream compressed according to a video encoding standard, and generates a packet for transmission across a packet-switched network. The system inserts at least one compressed segment of the video stream into a payload of the packet. The system also receives metadata that describes content within the video stream and is not compressed according to the standard, inserts the metadata into the packet without compressing the metadata according to the standard, and transmits the packet across the packet-switched network to a target device.