Described is a system (and method) for locating a center point of a lug nut for an automated vehicle wheel removal system. To improve the accuracy of the center point, the system may perform machine learning inferences using two-dimensional (2D) and three-dimensional (3D) image data. The system may process a 2D image to infer an initial center point, and potentially improve the accuracy by leveraging a 3D image. More particularly, the system may process a 3D image to infer a location of one or more edges (or edge points) around the perimeter of the lug nut and measure a set of distances between the initial center point and the located set of edges. The system may then refine (or adjust) the center point based on such measurements.