Middle English roste, "a chicken's perch," from late Old English hrost "wooden framework of a roof; pole or perch upon which domestic fowl perch or rest for the night," from Proto-Germanic *hro(d)-st- (source also of Old Saxon hrost "framework of a roof, attic," Middle Dutch, Flemish, Dutch roest "roost," Old Norse hrot, Gothic hrot "roof"), a word of unknown origin. Extended sense "hen-house" is from 1580s; that of "fowls which occupy the roost collectively" is by 1827.
To rule the roost is recorded from 1769, according to OED apparently an alteration of earlier rule the roast "be the master, have authority " (c. 1500), which, OED reports, was "In very common use from c 1530 onwards." However, Fowler (1926) has doubts: "most unliterary persons say roost & not roast ; I have just inquired of three such, & been informed that they never heard of rule the roast, & that the reference is to a cock keeping his hens in order. Against this tempting piece of popular etymology the OED offers us nothing more succulent than "None of the early examples throw any light on the precise origin of the expression'." The spelling in the earliest example is reule the roste.
1520s, "occupy a roost, perch as a bird," from roost (n.). Related: Roosted; roosting. Chickens come home to roost in reference to eventual consequences of bad actions attested from 1824; the original proverb seems to have been curses, like chickens, come home to roost.