1620s, an official position, "person who examines patent applications" (see refer), a sense now obsolete. By 1660s as "one to whom any matter in question is referred for decision, an umpire." Also in legal use, "person selected under authority of law to try a case in place of the court" (1680s). Sporting use is recorded by 1820 (specifically of baseball from 1856).
"preside over as a referee or umpire," 1883, originally colloquial, from referee (n.). Related: Refereed; refereeing.