bugger (n.)

"sodomite," 1550s, earlier "heretic" (mid-14c.), from Medieval Latin Bulgarus "a Bulgarian" (see Bulgaria), so called from bigoted notions of the sex lives of Eastern Orthodox Christians or of the sect of heretics that was prominent there 11c. Compare Old French bougre "Bulgarian," also "heretic; sodomite."

Softened secondary sense of "fellow, chap," is in British English "low language" [OED] from mid-19c. Meaning "something unpleasant, a nuisance" is from 1936. Related: Buggerly.

bugger (v.)

"to commit buggery with," 1590s, from bugger (n.). Meaning "ruin, spoil" is from 1923. Related: Buggered; buggering. Bugger off "go away" is from 1922, but the connection is obscure.