kind of thick, soft leather, 1570s, buffe leather "leather made of buffalo hide," from Middle French buffle "buffalo" (15c., via Italian, from Latin bufalus; see buffalo (n.)).
The color term "light brownish-yellow" (by 1788) comes from the hue of buff leather. Association of "hide" and "skin" led c. 1600 to the sense in in the buff "naked." Buff-colored uniforms of New York City volunteer firefighters since 1820s led to the meaning "enthusiast" (1903).
The Buffs are men and boys whose love of fires, fire-fighting and firemen is a predominant characteristic. [N.Y. "Sun," Feb. 4, 1903, quoted in OED]