1530s, "a boisterous, brutal fellow, one ready to commit any crime," from French rufian "a pimp" (15c.), from Italian ruffiano "a pander, pimp," of uncertain origin, perhaps from a Germanic source related to rough (adj.), but Dutch roffiaan, German Ruffian are said to be from French. English meaning might have been influenced by similarity of sound to rough. Related: Ruffianly.
The Romanic words (such as Medieval Latin ruffianus, Provençal rufian, Catalan rufia, Spanish rufian) preserve the sense of "protector or owner of whores." For sense evolution in English, compare bully (n.).