A system and method to interconnect heterogeneous compute resources using graph based execution flow for image and video processing is disclosed. The system includes a mix of compute engines (SIMDs, fixed-function pipelines and hardware kernels) which work on pipelined streams of data based on programming instructions and/or configuration data. The nodes act as proxies for compute engines in graph and message based routing system to establish graph based flow control mechanism between these nodes. A message based routing mechanism to assign tasks to heterogeneous compute engines and receive their status in time-multiplexed manner thus allowing compute engines to be reused. A configuration method that allows hardware compute engines are uniquely configured based on graph node thus allows reusability and programming or selecting heterogeneous compute system for physical implementation using simple standard definition file generation method.