"consisting of or pertaining to facts, not fanciful or ideal," 1712, from the noun phrase matter of fact "reality as distinguished from what is fanciful or hypothetical, truth attested by direct observation or authentic testimony," which is originally a legal term (1570s, translating Latin res facti), "that which is fact or alleged fact; that portion of an inquiry concerned with the truth or falsehood of alleged facts, subject of discussion belonging to the realm of fact" (as distinguished from matter of inference, opinion, law, etc.). See matter (n.) + fact.
The meaning "prosaic, unimaginative, adhering to facts" is from 1787. Related: Matter-of-factly; matter-of-factness. German Tatsache is said to be a loan-translation of the English word.
In law, that which is fact or alleged as fact; in contradistinction to matter of law, which consists in the resulting relations, rights, and obligations which the law establishes in view of given facts. Thus, the questions whether a man executed a contract, and whether he was intoxicated at the time, relate to matters of fact; whether, if so, he is bound by the contract, and what the instrument means, are matters of law. [Century Dictionary]