A metrology device with automated compensation and/or alert for orientation errors. The device may include a processor, a probe portion and at least one orientation sensor. The probe provides an output representative of a raw measurement of a characteristic of a device under test and the orientation sensor provides a sensor output representative of an orientation of the metrology device to the device under test. The processor applies a correction factor to the raw measurement in response to the sensor output to establish a compensated measurement to compensate for misalignment of the metrology device to the device under test. In addition, or alternatively, the processor provides an alert indicating the existence and/or extent of the misalignment.