"sodomite," 1550s, earlier "heretic" (mid-14c.), from Medieval Latin Bulgarus "a Bulgarian" (see Bulgaria), so called from bigoted notions of the sex lives of Eastern Orthodox Christians or of the sect of heretics that was prominent there 11c. Compare Old French bougre "Bulgarian," also "heretic; sodomite."
Softened secondary sense of "fellow, chap," is in British English "low language" [OED] from mid-19c. Meaning "something unpleasant, a nuisance" is from 1936. Related: Buggerly.