Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network that is configured to receive a network input and to generate a network output for the network input. The neural network comprises a plurality of layers arranged in a sequence, including a plurality of capsule layers. Each particular capsule in a particular capsule layer is configured to receive respective inputs including: (i) outputs generated by capsules of a previous capsule layer that is before the particular capsule layer in the sequence, and (ii) final routing factors between capsules of the previous capsule layer and the particular capsule, wherein the final routing factors are generated by a routing subsystem. Each particular capsule in the particular capsule layer is configured to determine a particular capsule output based on the received inputs, wherein the particular capsule output is of dimension greater than one.