1650s, from hand in cap, a game whereby two bettors would engage a neutral umpire to determine the odds in an unequal contest. The bettors would put their hands holding forfeit money into a hat or cap. The umpire would announce the odds and the bettors would withdraw their hands -- hands full meaning that they accepted the odds and the bet was on, hands empty meaning they did not accept the bet and were willing to forfeit the money. If one forfeited, then the money went to the other. If both agreed either on forfeiting or going ahead with the wager, then the umpire kept the money as payment. The custom, though not the name, is attested from 14c. ("Piers Plowman").
Reference to horse racing is 1754 (Handy-Cap Match), where the umpire decrees the superior horse should carry extra weight as a "handicap;" this led to sense of "encumbrance, disability" first recorded 1890. The main modern sense, "a mental or physical disability," is the last to develop, early 20c.