A method and a system are presented for controlling a performance of a fusion model. The method includes obtaining a first set and a second set of candidate models for a first and second neural networks, respectively. Each of the first and second set of candidate models is pre-trained with a first source and a second source, respectively. For each possible pairing of one candidate model from the first neural network and one candidate model from the second neural network, a model distance Dm is determined. A subset of possible pairings of one first candidate model and one second candidate model is selected based on the model distance Dm between them. Using the subset of possible parings, the first neural network and the second neural network are combined to generate two branches for a fusion model neural network.