A thermally conductive silicone rubber sponge composition which is composed of 100 parts by weight of heat-curable organopolysiloxane composition, 0.1 to 50 parts by weight of organic resin hollow filler having an average particle diameter of up to 200 μm and a true specific gravity of up to 0.3, and 20 to 300 parts by weight of thermally conductive filler having an average particle diameter of 1 to 30 μm and a thermal conductivity of at least 15 W/m·K, with the highly thermally conductive silicone rubber sponge having voids accounting for 10 to 70% of the entire volume thereof. The silicone rubber sponge composition of the present invention gives a silicone rubber sponge having high thermal conductivity, small heat capacity, low hardness, light weight, and low permanent compression set.