Et Vigny, plus secret, comme en sa tour d'ivoire, avant midi rentrait. [Sainte-Beuve, "Pensées d'Août, à M. Villemain," 1837]Used earlier as a type of a wonder or a symbol of "the ideal." The literal image is perhaps from Song of Solomon [vii:4]:
Thy neck is as a tower of ivory; thine eyes like the fishpools in Heshbon, by the gate of Bathrabbim: thy nose is as the tower of Lebanon which looketh toward Damascus. [KJV]