[Object] To provide an optical connector and an optical connector set which are excellent in environment resistance and suitable for the use in medical instruments, an image pickup unit and an image pickup system that use the optical connector, and an optical transmission module. [Solving Means] An optical connector according to the present technology includes a lens support, a fiber ferrule, a lens, and a lens retainer. The lens support includes a through-hole. An optical fiber is connected to the fiber ferrule. The fiber ferrule is press-fitted in the through-hole. The lens is inserted into the through-hole. The lens retainer is press-fitted in the lens support and sandwiches the lens between the lens retainer and the fiber ferrule.