"dark sandstone," 1849, from brown (adj.) + stone (n.). It was quarried extensively from Triassic deposits in the U.S. Northeast and much-used there as a building stone. As "house or building fronted with brownstone" from 1932.
ONLY a few years ago to live in a brownstone front was a badge of distinction in Manhattan. Novelists always had their rich housed in brownstone fronts. There was magic in the name a quarter of a century ago. The brownstone front was the home of the merchant prince. The material had to be mined on the western plains of New Jersey and teamed and lightered to New York at a great cost in those days. O.O. Mclntyre writes there are blocks and blocks of them above Forty-second street, but of late they have fallen into decay. The advent of the luxurious apartment house put them in the shade. Now they are being torn down with ruthless abandon and the last shred of dignity has vanished. ["The American Architect," Dec. 8, 1920]