A semiconductor device is made by forming a plurality of conductive pillars vertically over a temporary carrier. A conformal insulating layer is formed over the conductive pillars. A conformal conductive layer is formed over the conformal insulating layer. A first conductive pillar, conformal insulating layer, and conformal conductive layer constitute a vertically oriented integrated capacitor. A semiconductor die or component is mounted over the carrier. An encapsulant is deposited over the semiconductor die or component and around the conformal conductive layer. A first interconnect structure is formed over a first side of the encapsulant. The first interconnect structure includes an integrated passive device. The first interconnect structure is electrically connected to the semiconductor die or component and vertically oriented integrated capacitor. The carrier is removed. A second interconnect structure is formed over a second side of the encapsulant opposite the first side of the encapsulant.