Embodiments of the present invention address deficiencies of the art in respect to audio stream management in an audio conferencing server and provide a method, system and computer program product for balancing audio server loadability and audio server scalability based upon monitored resource consumption. In an embodiment of the invention, a method for balancing audio server loadability and audio server scalability based upon monitored resource consumption can be provided. The method can include monitoring computing conditions in an audio conferencing server receiving encoded audio packets from coupled Internet Protocol (IP) audio clients and routing the encoded audio packets as audio streams to the IP audio clients, and responsive to detecting deteriorating computing conditions in the audio conferencing server, reducing a number of the audio streams forwarded to the IP audio clients by the audio conferencing server.