A neural network unit has at least one RAM, an output buffer and an array of neural processing units that: read first time step context layer node values from the output buffer; read second time step input layer node values from the RAM; generate second time step hidden layer node values based on the read input and context layer node values; output the hidden layer node values to the output buffer rather than to the RAM; read the hidden layer node values from the output buffer; generate second time step context layer node values based on the read hidden layer node values; output the context layer node values to the output buffer rather than to the RAM; generate output layer node values using the hidden layer node values; write the output layer node values to the RAM; and repeat for a sequence of time steps.