c. 1600, from Latin toga "cloak or mantle," from PIE *tog-a- "covering," suffixed form of root *(s)teg- "to cover." The outer garment of a Roman citizen in time of peace.
The toga as the Roman national dress was allowed to be worn by free citizens only. A stranger not in full possession of the rights of a Roman citizen could not venture to appear in it. Even banished Romans were in imperial times precluded from wearing it. The appearance in public in a foreign dress was considered as contempt of the majesty of the Roman people. Even boys appeared in the toga, called, owing to the purple edge attached to it (a custom adopted from the Etruscans) toga praetexta. On completing his sixteenth, afterward his fifteenth, year (tirocinium fori), the boy exchanged the toga praetexta for the toga virilis, pura, or libera—a white cloak without the purple edge. Roman ladies (for these also wore the toga) abandoned the purple edge on being married. [Guhl & Koner, "The Life of the Greeks and Romans," transl. Francis Hueffer, 1876]
Breeches, like the word for them (Latin bracae) were alien to the Romans, being the dress of Persians, Germans, and Gauls, so that bracatus "wearing breeches" was a term in Roman geography meaning "north of the Alps." College fraternity toga party was re-popularized by movie "Animal House" (1978), but this is set in 1962 and the custom seems to date from at least the mid-1950s.
Down on Prospect Street, Campus Club held a toga party, at which everyone wore togas. Charter held a come-as-you-are party, at which everyone wore what they happened to have on, and Cloister held a party called "A Night in Tahiti," at which we'd hate to guess what everyone wore. The borough police reported that only one false alarm was turned in. [Princeton Alumni Weekly, March 19, 1954]