A contact via cavity can be filled with a lower structure and an upper structure. The lower structure can be a conductive structure that is formed by depositing a conformal conductive material, and subsequently removing an upper portion of the conformal conductive material. A disposable material portion can be formed at a bottom of the cavity to protect the bottom portion of the conformal conductive layer during removal of the upper portion. After removal of the disposable material, at least one conductive material can fill the remainder of the cavity to form the upper structure. The upper structure and the lower structure collectively constitute a contact via structure. Alternatively, the lower structure can be a dielectric spacer with an opening therethrough. The upper structure can be a conductive structure that extends through the dielectric spacer, and provides an electrically conductive vertical connection.