cardinal number (n.)

early 14c., "book setting forth the order of services in the Church," from Late Latin adjective ordinalis (see ordinal (adj.)), which was used as a noun in Medieval Latin.

"chief, pivotal," early 14c., from Latin cardinalis "principal, chief, essential," a figurative use, literally "pertaining to a hinge," from cardo (genitive cardinis) "that on which something turns or depends; pole of the sky," originally "door hinge," which is of unknown origin. Related: Cardinally.

The cardinal numbers (1590s) are "one, two, three," etc. as opposed to ordinal numbers "first, second, third," etc.; they are so called because they are the principal numbers and the ordinals depend on them.

The cardinal points (1540s) are north, south, east, west. The cardinal sins (c. 1600) are too well known to require rehearsal. The cardinal virtues (c. 1300) were divided into natural (justice, prudence, temperance, fortitude) and theological (faith, hope, charity). The natural ones were the original classical ones, which were amended by Christians. But typically in Middle English only the first four were counted as the cardinal virtues:

Of þe uour uirtues cardinales spekeþ moche þe yealde philosofes. ["Ayenbite of Inwyt," c. 1340]

By analogy of this, and cardinal winds (late 14c.), cardinal signs (four zodiacal signs marking the equinoxes and the solstices, late 14c.), etc., the adjective in Middle English acquired an association with the number four.