"crazy, not right in the head," 1846, from earlier colloquial or slang be nuts on "be very fond of" (1785), which is possibly from nuts (plural noun) "any source of pleasure or delight" (1610s), from nut (q.v.). Nuts as a special treat or favorite foodstuff led to other figurative phrases, now obsolete. The "crazy" sense probably has been influenced by metaphoric application of nut to "head" (1846, as in to be off one's nut "be insane," 1860). Also compare nutty. Nuts as a derisive retort is attested from 1931.
Connection with the slang "testicles" sense has tended to nudge the word toward taboo territory. "On the N.B.C. network, it is forbidden to call any character a nut; you have to call him a screwball." [New Yorker, Dec. 23, 1950] "Please eliminate the expression 'nuts to you' from Egbert's speech." [Request from the Hays Office regarding the script of "The Bank Dick," 1940] This desire for avoidance probably accounts for the euphemism nerts (c. 1925).