also man-handle, mid-15c., "wield a tool," also, late 15c., "to attack (an enemy)," from man (n.) + handle (v.). Nautical meaning "to move by force of men" (without levers or tackle) is attested from 1834, and is the source of the slang meaning "to handle roughly" (1865). Related: Manhandled; manhandling.
[T]he two Canalers rushed into the uproar, and sought to drag their man out of it toward the forecastle. Others of the sailors joined with them in this attempt, and a twisted turmoil ensued; while standing out of harm's way, the valiant captain danced up and down with a whale-pike, calling upon his officers to manhandle that atrocious scoundrel, and smoke him along to the quarter-deck. [Melville, "The Town-Ho's Story," Harper's magazine, October 1851]