A composite membrane that is suitable for use in an electrochemical cell, an electrochemical cell including the composite membrane, and a method of making the composite membrane. In one embodiment, the composite membrane includes a porous support and a solid electrolyte. The porous support is a unitary structure made of a polymer that is non-conductive to ions. The porous support is shaped to include a plurality of straight-through pores. The solid electrolyte has alkali ion conductivity and preferably completely fills at least some of the pores of the porous support. A variety of techniques may be used to load the solid electrolyte into the pores. According to one technique, the solid electrolyte is melted and then poured into the pores of the porous support. Upon cooling, the electrolyte re-solidifies, forming a monolithic structure within the pores of the porous support.