A relationship graph system disclosed herein provides a system and method for inferring relationships between various entities based on analysis of a plurality of observations including elements related to two different entities. The method comprises representing each of the plurality of observations using a relationship edge between two nodes, each of the two nodes representing an element related to one of the two different entities, assigning a score to each relationship edge based on metadata related to the observation represented by the relationship edge, combining a plurality of relationship edges between the elements related to the two different entities to generate an aggregate relationship edge between the two different entities, and inferring a relationship between the two different entities based on length of the aggregate relationship edge between the two different entities.