1864, "spot within one's range of vision but where one cannot see," from blind (adj.) + spot (n.). Of the point on the retina insensitive to light (where the optic nerve enters the eye), from 1872. The figurative use of the older sense (in reference to moral, intellectual, etc. sight) is by 1907.

symbol of artistic or intellectual aloofness, by 1889, from French tour d'ivoire, used in 1837 by critic Charles-Augustin Sainte-Beuve (1804-1869) with reference to the poet Alfred de Vigny (1797-1863), whom he accused of excessive aloofness.

Et Vigny, plus secret, comme en sa tour d'ivoire, avant midi rentrait. [Sainte-Beuve, "Pensées d'Août, à M. Villemain," 1837]

Used earlier as a type of a wonder or a symbol of "the ideal." The literal image is perhaps from Song of Solomon [vii:4]:

Thy neck is as a tower of ivory; thine eyes like the fishpools in Heshbon, by the gate of Bathrabbim: thy nose is as the tower of Lebanon which looketh toward Damascus. [KJV]

late 14c., translating Latin artes liberales; the name for the seven attainments directed to intellectual enlargement, rather than immediate practical purpose, and thus deemed worthy of a free man (liberal in this sense is opposed to servile or mechanical). They were divided into the trivium — grammar, logic, rhetoric (see trivial) — and the quadrivium — arithmetic, geometry, music, astronomy. Explained by Fowler (1926) as "the education designed for a gentleman (Latin liber a free man) & ... opposed on the one hand to technical or professional or any special training, & on the other to education that stops short before manhood is reached."

The study of [the classics] is fitly called a liberal education, because it emancipates the mind from every narrow provincialism, whether of egoism or tradition, and is the apprenticeship that every one must serve before becoming a free brother of the guild which passes the torch of life from age to age. [James Russell Lowell, "Among my Books"]