A method and apparatus of scaling list data signaling for inter-layer or inter-view sharing of the scaling list data from a reference layer or a reference view in a scalable or three-dimensional video coding system are disclosed. A first flag may be incorporated in the current bitstream to indicate the scaling list data sharing from a reference layer or view. When the first flag exists and the first flag has a first value, the scaling list data for the current layer or the current view is determined from a reference bitstream for a reference layer or a reference view. When the first flag exists and the first flag has a second value, the scaling list data for the current layer or the current view is determined from the current bitstream.