Dynamic resource allocation in MapReduce
Abstract:
Methods, systems, and articles of manufacture for dynamic resource allocation in MapReduce are provided herein. A method includes partitioning input data into one or more sized items of input data associated with a MapReduce job; determining a total number of mapper components, and a total number of reducer components for the MapReduce job based on said partitioning; dynamically determining an allocation of resources to each of the total number of mapper components and reducer components during run-time of the MapReduce job, wherein said dynamically determining the allocation of resources comprises monitoring one or more utilization parameters for each of the total number of mapper components and total number of reducer components during run-time of the MapReduce job; and dynamically determining a number of concurrently executing mapper components and reducer components from the total number of mapper components and the total number of reducer components for the MapReduce job.
Public/Granted literature
Information query
Patent Agency Ranking
0/0