A method and apparatus for video coding using template-based Intra prediction are disclosed. According to one method, where determining whether to apply the template-based Intra prediction or one or more parameters associated with the template-based Intra prediction depends on the current block size. According to yet another method, the quad-tree plus binary tree (QTBT) structure is used to partition an image or one or more areas of the current image into blocks. If the template-based Intra prediction is used for a current block and the current block is non-square, the width and height of the L-shaped reference pixel line are determined according to width and height of the current block. The L-shaped reference pixel line comprises a top reference pixel segment above the top template and a left reference pixel segment adjacent to a left side of the left template.