"extremely small," mid-15c., from earlier noun use in sense of "quantity, amount" (such as a littel wei "a little thing or amount," c. 1300), from Old English wæge "weight, unit of weight," from Proto-Germanic *wego, from PIE root *wegh- "to go, move, transport in a vehicle." The original sense was of motion, which led to that of lifting, then to that of "measure the weight of" (compare weigh, from the same source).
Adjectival use wee bit apparently developed as parallel to such forms as a bit thing "a little thing." Wee hours "hours after midnight" is attested by 1891, from Scottish phrase wee sma' hours (1819); so called for their low numbers. Wee folk "faeries" is recorded from 1819. Weeny "tiny, small" is from 1790.