Apparatuses, methods, and non-transitory computer readable medium that evaluate a source code scanner are described. In one implementation, the method comprises obtaining source code. One or more good code snippets and one or more bad code snippets are inserted into the source code to obtain a modified source code. An issue list generated by the source code scanner upon scanning the modified source code is obtained. The issue list comprises code segments having security defects identified by the source code scanner, reasons for the security defects, and locations of the security defects in the modified source code. The code segments present in the issue list are compared with the one or more good code snippets and the one or more bad code snippets. A plurality of metrics, indicating quality of the source code scanner, are generated based on the comparison.