According to one embodiment, an elevator passenger detection system comprises an image capturing unit, a detection area setting unit, and a detection processing unit. The image capturing unit captures an image in a predetermined range including a vicinity of a doorway at which a door is opened or closed, from an inside of a car. The detection area setting unit sets a detection area on a front return panel provided on at least one of sides of a doorway of the car, on the captured image obtained by the image capturing unit. The detection processing unit detects presence of a passenger or an object, based on the image in the detection area set by the detection area setting unit.