A method, computer program, and computer system is provided for estimating three-dimensional hand poses in images. Data corresponding to two hand images is receive, and an optical flow value corresponding to a change in a hand gesture in the received hand image data is calculate. A heat map is generated based on the calculated optical flow, and a hand mesh map is estimated based on the generated heat map. A hand pose present within the hand images is determined based on the estimated hand mesh map.