A negative pattern is formed by applying a resist composition onto a substrate, prebaking, exposing to high-energy radiation, baking (PEB), and developing the exposed resist film in an organic solvent developer to dissolve the unexposed region of resist film. The resist composition comprising a polymer comprising recurring units having an acid labile group-substituted carboxyl group and/or hydroxy group and recurring units having an oxirane or oxetane ring, an acid generator, and an organic solvent displays a high dissolution contrast in organic solvent development and controlled acid diffusion. A fine hole pattern featuring good size control can be formed.