This invention presents an approach to first estimate a depth/disparity map using spectrally coded plenoptic camera, and then based on the disparity map the parallax between different spectral channels is rectified. Based on our new technique, we can reconstruct not only multispectral images but also a depth/disparity map. Moreover, the quality of reconstructed spectral images is significantly improved using our parallax rectification.