A method and system for fully automatic segmentation the prostate in multi-spectral 3D magnetic resonance (MR) image data having one or more scalar intensity values per voxel is disclosed. After intensity standardization of multi-spectral 3D MR image data, a prostate boundary is detected in the multi-spectral 3D MR image data using marginal space learning (MSL). The detected prostate boundary is refined using one or more trained boundary detectors. The detected prostate boundary can be split into patches corresponding to anatomical regions of the prostate and the detected prostate boundary can be refined using trained boundary detectors corresponding to the patches.