late Old English castel "village" (this sense from a biblical usage in Vulgar Latin); later "large building or series of connected buildings fortified for defense, fortress, stronghold" (late Old English), in this sense from Old North French castel (Old French chastel, 12c.; Modern French château), from Latin castellum "a castle, fort, citadel, stronghold; fortified village," diminutive of castrum "fort," from Proto-Italic *kastro- "part, share;" cognate with Old Irish cather, Welsh caer "town" (probably related to castrare via notion of "cut off," from PIE root *kes- "to cut"). In early bibles, castle was used to translate Greek kome "village."
Latin castrum in its plural castra was used for "military encampment, military post" and thus it came into Old English as ceaster and formed the -caster and -chester in place names. Spanish alcazar "castle" is from Arabic al-qasr, from Latin castrum. Castles in Spain "visionary project, vague imagination of possible wealth" translates 14c. French chastel en Espaigne (the imaginary castles sometimes stood in Brie, Asia, or Albania) and probably reflects the hopes of landless knights to establish themselves abroad. The statement that an (English) man's home is his castle is from 16c.
THAT the house of every man is to him as his castle and fortress, as well for his defence against injuries and violence, as for his repose .... [Edward Coke, "Semaynes Case," 1604]