A method, system and computer program product for efficiently utilizing connections in connection pools. A period of time an application running on a virtual machine needs a greater number of connections to an external resource than allocated in its pool of connections is identified. The connection pool for this application as well as the connection pools for the other applications containing connections to the same external resource are merged to form a logical pool of connections to be shared by those applications during the identified period of time. Alternatively, in an application server cluster environment, the connection pools utilized by the application servers to access the external resource may be reconfigured based on the weight assigned to each member (or application server) of the cluster which is based on the member's load size. In these manners, the resource connections in these pools of connections will be more efficiently utilized.