1540s, "miserable person, wretch," from Latin miser (adj.) "unhappy, wretched, pitiable, in distress," a word for which "no acceptable PIE pedigree has been found" [de Vaan]. Original sense now obsolete; main modern meaning of "money-hoarding person" recorded 1560s, from presumed unhappiness of such people.
Besides general wretchedness, the Latin word connoted also "intense erotic love" (compare slang got it bad "deeply infatuated") and hence was a favorite word of Catullus. In Greek a miser was kyminopristes, literally "a cumin seed splitter." In Modern Greek, he might be called hekentabelones, literally "one who has sixty needles." The German word, filz, literally "felt," preserves the image of the felt slippers which the miser often wore in caricatures. Lettish mantrausis "miser" is literally "money-raker."