In image segmentation it is usually assumed that all the surfaces of a scene are reflecting light in a perfectly diffuse or Lambertian manner. However, most existing surfaces are non-Lambertian surfaces. Examples of non-Lambertian surfaces are specular, glossy or mirror-like surfaces whose appearance may change depending on the point of view of the scene. The method according to an embodiment of the invention does not make the assumption that surfaces of objects in the view are perfect Lambertian reflectors by providing a surface color segmentation which takes into account the change in appearance due to the nature of the surfaces.