A parking assist device includes an electronic control unit. The electronic control unit is configured to: perform parking assist of a vehicle to automatically park the vehicle at a target position while restricting a vehicle speed to an upper limit or lower in accordance with an instruction of a user who is outside the vehicle; determine whether or not the vehicle moves on a road surface having a downhill grade along a travelling direction of the vehicle by the parking assist; and set the upper limit in the case where the vehicle is determined to move on the road surface having a downhill grade to be smaller than the upper limit in the case where the vehicle is not determined to move on the road surface having a downhill grade.