The spelling with wh- developed early 15c. The sense in whole number is from early 14c. Whole milk is from 1782. On the whole "considering all facts or circumstances" is from 1690s. For phrase whole hog, see hog (n.).
"woven fabric, pliable stuff made of intertexture of threads or fibers," Old English claþ "a cloth, sail, cloth covering, woven or felted material to wrap around one," hence, also, "garment," from Proto-Germanic *kalithaz (source also of Old Frisian klath "cloth," Middle Dutch cleet, Dutch kleed "garment, dress," Middle High German kleit, German Kleid "garment"), which is of obscure origin, perhaps a substratum word.
As an adjective, "made or consisting of cloth," from 1590s. Meaning "distinctive clothing worn by some group" (servants of one house, men of some profession or trade) is from 1590s, hence The cloth "the clerical profession" (1701).