Systems and methods for inter-layer reference picture set derivation based on sub-layer reference prediction dependency are described herein. One aspect of the subject matter described in the disclosure provides a video encoder comprising a memory configured to store one or more direct reference layer pictures of one or more current pictures in a sequence, wherein the one or more current pictures are associated with a current layer, the current layer being associated with the one or more direct reference layers. The video encoder further comprises a processor in communication with the memory unit. The memory unit is configured to set an indication associated with a current picture to indicate whether all of the one or more direct reference layer pictures of the current picture that are not restricted for use in inter-layer prediction are included in an inter-layer reference picture set associated with the current picture.