A semiconductor device with an arithmetic processing function is provided. In the semiconductor device, an imaging portion and an arithmetic portion are electrically connected to each other through an analog processing circuit 24. The imaging portion includes a pixel array 21 in which pixels 20 used for imaging and reference pixels 22 used for image processing are arranged in a matrix, and a row decoder 25. The arithmetic portion includes a memory element array 31 in which memory elements 30 and reference memory elements 32 are arranged in a matrix, an analog processing circuit 34, a row decoder 35, and a column decoder 36.