Devices, systems and methods for the simplification of sub-block motion candidate lists for video coding are described. In a representative aspect, a method for video processing includes determining, for a conversion between a current block of a video and a bitstream representation of the video, one or more temporal motion vector prediction candidates for the current block and performing the conversion based on the one or more temporal motion vector prediction candidates. The one or more temporal motion vector prediction candidates are determined by identifying a first temporal adjacent block of the current block based on an initial motion vector, wherein the first temporal adjacent block includes invalid motion information, and examining additional temporal adjacent blocks of the current block to obtain the one or more temporal motion vector prediction candidates.