mid-14c., patrimoine, "property of the Church," also "spiritual legacy of Christ," from Old French patremoine "heritage, patrimony" (12c.) and directly from Latin patrimonium "a paternal estate, inheritance from a father," also figurative, from pater (genitive patris) "father" (see father (n.)) + -monium, suffix signifying action, state, condition.
In English law, the meaning "right or property inherited from a father or ancestors" is attested from late 14c. Figurative sense of "immaterial things handed down from the past, heritage" is from 1580s. A curious sense contrast to matrimony.