"loafer, idle person," 1855, possibly an extension of the British word for "backside" (similar development took place in Scotland by 1540), but more probably from German slang bummler "loafer," agent noun from bummeln "go slowly, waste time." The earliest uses are in representations of German immigrant dialect in the U.S. In the American Civil War it was common in the sense "camp-follower, plundering straggler."
According to Kluge, the German word is from 17c., and its earliest sense is "oscillate back and forth." It is perhaps connected to words in German for "dangle" (baumeln), via "back-and-forth motion" of a bell clapper, transferred to "going back and forth," hence "doing nothing." Meaning "bad experience" is 1968 slang.