A system and method for maintaining a grey list for controlling incoming calls from numbers on a telephone network. Information associated with previous calls received by the telephone is stored in the form of a grey list. The stored information includes an indication of whether each of the previously received calls passed a previous challenge. Based on the stored information, when an incoming call is received, a determination is made whether the incoming call corresponds to a call that passed a previous challenge. The incoming call connected to the telephone if the stored information indicates that the incoming call previously passed a challenge. An action is taken if the stored information does not indicate that the incoming call previously passed the challenge. The action may include blocking the call, allowing the call or providing a current challenge to the call.