Embodiments of the present invention provide a system for scheduling memory accesses for one or more memory devices. This system includes a set of queues configured to store memory access requests, wherein each queue is associated with at least one memory bank or memory device in the one or more memory devices. The system also includes a set of hierarchical levels configured to select memory access requests from the set of queues to send to the one or more memory devices, wherein each level in the set of hierarchical levels is configured to perform a different selection operation.