Middle English roste, "a chicken's perch," from late Old English hrost "wooden framework of a roof; pole or perch upon which domestic fowl perch or rest for the night," from Proto-Germanic *hro(d)-st- (source also of Old Saxon hrost "framework of a roof, attic," Middle Dutch, Flemish, Dutch roest "roost," Old Norse hrot, Gothic hrot "roof"), a word of unknown origin. Extended sense "hen-house" is from 1580s; that of "fowls which occupy the roost collectively" is by 1827.
To rule the roost is recorded from 1769, apparently an alteration of earlier rule the roast "be the master, have authority " (c. 1500), which, OED reports, was "In very common use from c 1530 onwards, but none of the early examples throw any light on the precise origin of the expression."
1520s, "occupy a roost, perch as a bird," from roost (n.). Related: Roosted; roosting. Chickens come home to roost in reference to eventual consequences of bad actions attested from 1824; the original proverb seems to have been curses, like chickens, come home to roost.