A method and apparatus for aligning and unwarping distorted images in which an initial unwarping function is applied to the coordinates of feature points of a set of input component images to generate substantially rectilinear feature points. The substantially rectilinear feature points are then used to estimate focal lengths, centers, and relative rotations for pairs of the input images. A global nonlinear optimization is applied to the initial unwarping function(s) and the relative rotations to generate optimized unwarping functions and rotations for the component images. The optimized unwarping functions and rotations may then be used to render a panoramic image, generally in the form of a spherical projection, from the input component images.