"loud, disorderly, confusing noise," 1560s, probably imitative. Klein and Century Dictionary compare Gaelic racaid "noise, disturbance," but OED says this "is no doubt from Eng."
Meaning "dishonest activity" (1785) is perhaps an extended sense, from the notion of "something going on" or "noise or disturbance made to distract a pick-pocket's victim." Or it might be from racquet, via the notion of "a game," or from or reinforced by rack-rent "extortionate rent." There also was a verb racket "carry on eager or energetic action" (1753), and the gangster sense might be via the notion of "exciting and unusual." Weakened sense of "way of life, one's line of business" is by 1891.