Old English hem "a border" of cloth or a garment, from Proto-Germanic *hamjam (source also of Old Norse hemja "to bridle, curb," Swedish hämma "to stop, restrain," Old Frisian hemma "to hinder," Middle Dutch, German hemmen "to hem in, stop, hinder"), from PIE *kem- "to compress" (source also of Armenian kamel "to press, squeeze," Lithuanian kamuoti "press together, stop," Russian kom "mass, clot, clod").
Apparently the same root yielded Old English hamm, common in place names (where it means "enclosure, land hemmed in by water or high ground, land in a river bend"). In Middle English, hem also was a symbol of pride or ostentation.
If þei wer þe first þat schuld puplysch þese grete myracles of her mayster, men myth sey of hem, as Crist ded of þe Pharisees, þat þei magnified her owne hemmys. [John Capgrave, "Life of Saint Gilbert of Sempringham," 1451]
1520s, probably imitative of the sound of clearing the throat; late 15c. as a verb meaning "to make the sound 'hem.'" Hem and haw (v.) first recorded 1786, with haw "hesitation" (1630s; see haw (v.)); hem and hawk attested from 1570s.
late 14c., "to provide (something) with a border or fringe" (surname Hemmer attested from c. 1300), from hem (n.). Meaning "to enclose, circumscribe" is from 1530s. Related: Hemmed; hemming. The phrase hem in "shut in, confine," first recorded 1530s.