"inept person; stupid, dull old man," 1842, especially "bad golfer" (by 1875), perhaps from Scottish duffar "dull or stupid person," from dowf "stupid," literally "deaf," from Old Norse daufr, with pejorative suffix -art. Or perhaps from 18c. thieves' slang duff (v.) "to dress or manipulate an old thing and make it look new," hence duffer "one who sells spurious goods at high prices" (1766).
Since 16c. mainly a sailors' word for those inept or inexperienced at sea (as in landlubber), but earliest attested use is of lazy monks (abbey-lubber). Compare also provincial English lubberwort, name of the mythical herb that produces laziness (1540s), Lubberland "imaginary land of plenty without work" (1590s).