Provided are an image processing device, an image processing method, and a program which are capable of high density restoration and which are also strong to image processing. An image processing device mainly consists of a projector serving as a projection means, a camera as a photographing means, and an image processing means consisting of, for example, a personal computer. The image processing means acquires the intersection point between patterns from a photographed image and calculates a first solution including degree of freedom by using the constraint condition of a first tentative plane and a second tentative plane including the intersection point and the constraint condition obtained from the positional relationship between the projector and the camera. The degree of freedom is cancelled by primary search, thereby restoring a three-dimensional shape.