also riff-raff, late 15c., "persons of disreputable character or low degree," from earlier rif and raf (Anglo-French rif et raf) "one and all, everybody; every scrap, everything," also "sweepings, refuse, things of small value" (mid-14c.), from Old French rif et raf, from rifler "to spoil, strip" (see rifle (v.)). Second element from raffler "carry off," related to rafle "plundering," or from raffer "to snatch, to sweep together" (see raffle (n.)); the word presumably made more for suggestive half-rhyming alliteration than for sense.
The meaning "refuse, scum, or rabble of a community" is by 1540s. In 15c. collections of terms of association, a group of young men or boys was a raffle of knaves.