World War II aviator slang for "unidentified aircraft, presumably hostile," probably ultimately from bogge, a variant of Middle English bugge "a frightening specter" (see bug (n.)).
Thus it shares ancestry with many dialect words for "ghost, specter," such as bog/bogge (attested 16c.-17c.), bogeyman (16c.), boggart "specter that haunts a gloomy spot" (c. 1570, in Westmoreland, Lancashire, Cheshire, and Yorkshire). The earliest modern form appears to be Scottish bogle "ghost," attested from c. 1500 and popularized c. 1800 in English literature by Scott, Burns, etc.
in golf, c. 1891, originally "number of strokes a good player is supposed to need for a given hole or course;" later, "score one over par" (1946); from the same source as bogey (n.1), on the notion of a "phantom" opponent, represented by the "ground score." The word was in vogue at the time in Britain through the popularity of a music-hall tune "Hush, Hush, Hush, Here Comes the Bogey Man."
One popular song at least has left its permanent effect on the game of golf. That song is 'The Bogey Man.' In 1890 Dr. Thos. Browne, R.N., the hon. secretary of the Great Yarmouth Club, was playing against a Major Wellman, the match being against the 'ground score,' which was the name given to the scratch value of each hole. The system of playing against the 'ground score' was new to Major Wellman, and he exclaimed, thinking of the song of the moment, that his mysterious and well-nigh invincible opponent was a regular 'bogey-man.' The name 'caught on' at Great Yarmouth, and to-day 'Bogey' is one of the most feared opponents on all the courses that acknowledge him. [1908, cited in OED]
Other early golfing sources give it an American origin. As a verb, attested by 1948.