also hobby-horse, 1550s, "mock horse used in the morris dance;" 1580s, "child's toy riding horse," from hobby (n.) + horse (n.). Transferred sense of "favorite pastime or avocation" first recorded 1670s (shortened to hobby by 1816). The connecting notion being "activity that doesn't go anywhere."
The hobbyhorse originally was a "Tourney Horse," a wooden or basketwork frame worn around the waist and held on with shoulder straps, with a fake tail and horse head attached, so the wearer appears to be riding a horse. These were part of church and civic celebrations at Midsummer and New Year's throughout England.