A method and system for distributed machine learning and model training are disclosed. In particular, a finite asynchronous parallel training scheme is described. The finite asynchronous parallel training takes advantage of the benefits of both asynchronous parallel training and synchronous parallel training. The computation delays in various distributed computation nodes are further considered when training parameter are updated during each round of iterative training. The disclosed method and system facilities increase of model training speed and efficiency.