1895, "crazy, insane," from bug (n.) + house (n.); probably originally tramps' jargon. As a noun, from 1891 as "insanity," 1898 as "insane asylum."
IT is often the case in the Penitentiary, as well as in the out side world, that men get "wheels in their head," and "talk through their hats." When a "boy" gets in this "offish" state the prisoners call him "buggy" he becomes a bug-a-boo, and to keep him safe so that he can hurt no one, nor destroy himself, he is duly examined, and when adjudged "bugy" is placed for safe keeping in the "Bug House." [Dan J. Morgan, "Historical Lights and Shadows of the Ohio Penitentiary," 1898]