A ranking system is described for adjusting quality distributions of items responsive to pairwise comparisons. The pairwise comparisons are selected for a user providing a preference between the items in the pairwise comparisons. The selection is determined based on the cost of an error in the rankings of items in combination with the likelihood there is an error in the item rankings. On receiving the comparison result from the user, the quality distributions are updated and can include a confidence modifier to determine the confidence the user's preference is reflective of the actual preference between the items.