Old English smæl "thin, slender, narrow; fine," from Proto-Germanic *smal- "small animal; small" (source also of Old Saxon, Danish, Swedish, Middle Dutch, Dutch, Old High German smal, Old Frisian smel, German schmal "narrow, slender," Gothic smalista "smallest," Old Norse smali "small cattle, sheep"), perhaps from a PIE root *(s)melo- "smaller animal" (source also of Greek melon, Old Irish mil "a small animal;" Old Church Slavonic malu "bad"). Original sense of "narrow" now almost obsolete, except in reference to waistline and intestines.
My sister ... is as white as a lilly, and as small as a wand. [Shakespeare, "Two Gentlemen of Verona," 1591]
Sense of "not large, of little size" developed in Old English. Of children, "young," from mid-13c. Meaning "inferior in degree or amount" is from late 13c. Meaning "trivial, unimportant" is from mid-14c. Sense of "having little property or trade" is from 1746. That of "characterized by littleness of mind or spirit, base, low, mean" is from 1824. As an adverb by late 14c.
Small fry, first recorded 1690s of little fish, 1885 of insignificant people. Small potatoes "no great matter, something petty or insignificant" is attested by 1924; small change "something of little value" is from 1902; small talk "chit-chat, trifling conversation" (1751) first recorded in Chesterfield's "Letters." Small world as a comment upon an unexpected meeting of acquaintances is recorded from 1895. Small-arms, indicating those capable of being carried in the hand (contrasted to ordnance) is recorded from 1710.