1570s, "soft, wet, boggy land; a marsh," from obsolete quag "bog, marsh" + mire (n.). Early spellings or related forms include quamyre (1550s), quabmire (1590s), quadmire (c. 1600), quavemire (1520s), qualmire.
The extended sense of "difficult situation, inescapable bad position" is recorded by 1766; but this seems to have been not in common use in much of 19c. (absent in "Century Dictionary," 1897, which does, however, have a verb, marked "rare," meaning "to entangle or sink in or as in a quagmire"). It revived in a narrower sense in American English in reference to stalled military actions, 1965, with reference to the U.S. war in Vietnam (popularized in the book title "The Making of a Quagmire" by David Halberstam).