An apparatus in one embodiment comprises a plurality of container host devices of at least one processing platform. The container host devices implement a plurality of containers for executing applications on behalf of one or more tenants of cloud infrastructure. The containers have associated layer structures each characterizing container images of respective different ones of the containers. Movement of container data between different storage devices by at least one of the container host devices for at least one of the containers is controlled based at least in part on one or more characteristics of a corresponding one of the layer structures. For example, controlling movement of container data between the different storage devices may comprise assigning at least one of different prefetching priority weights and different cache-swapping priority weights to different layers of the given layer structure.