A method of forming a semiconductor device is provided. A first substrate is provided with a conductive feature therein, a metal bump over the conductive feature and a passivation stack aside the metal bump. A first insulating layer is formed over the metal bump and the passivation stack. First and second patterning processes are performed to form first and second opening patterns in the first insulating layer. The metal bump is exposed by the second patterning process. A second substrate is provided with a second insulating layer thereon. The second substrate is bonded to the first substrate with the second insulating layer and the first insulating layer facing each other, so that the second insulating layer fills in the first and second opening patterns of the first insulating layer. The first insulating layer and a portion of the passivation stack are removed.