mid-13c., "garment, cloak, mantle; a part of a garment;" later "side of a building, section of a wall," from Old French pan "section, piece, panel" (11c.) and directly from Latin pannum (nominative pannus) "piece of cloth, garment," possibly from PIE root *pan- "fabric" (source also of Gothic fana "piece of cloth," Greek pēnos "web," Old English fanna "flag"). De Vaan writes, "If the Gr. and Gm. words listed are related, they probably represent loanwords from an unknown source."
From late 14c. as "section of a wall," also "ornamental hanging, coverlet," and c. 1400 as "a bedspread." The general notion in the word is "distinct part or piece of a surface." Sense of "piece of glass inserted in a window" is attested by mid-15c.