A method includes forming a dummy gate electrode layer over a semiconductor region, forming a mask strip over the dummy gate electrode layer, and performing a first etching process using the mask strip as a first etching mask to pattern an upper portion of the dummy gate electrode layer. A remaining portion of the upper portion of the dummy gate electrode layer forms an upper part of a dummy gate electrode. The method further includes forming a protection layer on sidewalls of the upper part of the dummy gate electrode, and performing a second etching process on a lower portion of the dummy gate electrode layer to form a lower part of the dummy gate electrode, with the protection layer and the mask strip in combination used as a second etching mask. The dummy gate electrode and an underlying dummy gate dielectric are replaced with a replacement gate stack.