"brief examination of a student on some subject," 1852, perhaps from quiz (v.), or from slang quiz "odd person" (1782, perhaps originally university slang), via the notion of "schoolboy prank or joke played at the expense of a person deemed a quiz" (a noun sense attested frequently 1840s).
A Quiz, in the common acceptation of the word, signifies one who thinks, speaks, or acts differently from the rest of the world in general. But, as manners and opinions are as various as mankind, it will be difficult to say who shall be termed a Quiz, and who shall not: each person indiscriminately applying the name of Quiz to every one who differs from himself .... [The London Magazine, November 1783]
According to OED, the anecdote that credits this word to a bet by the Dublin theater-manager Daly or Daley that he could coin a word is regarded by authorities as "doubtful" and the first record of it appears to be in 1836 (in Smart's "Walker Remodelled"; the story is omitted in the edition of 1840).
The word Quiz is a sort of a kind of a word
That people apply to some being absurd;
One who seems, as t'were oddly your fancy to strike
In a sort of a fashion you somehow don't like
A mixture of odd, and of queer, and all that
Which one hates, just, you know, as some folks hate a cat;
A comical, whimsical, strange, droll — that is,
You know what I mean; 'tis — in short, — 'tis a quiz!
[from "Etymology of Quiz," Charles Dibdin, 1842]
1847, "to question," quies, perhaps from Latin qui es? "who are you?," first question in oral exams in Latin in old-time grammar schools. Spelling quiz first recorded 1886, though it was in use as a noun spelling from 1854, perhaps in this case from apparently unrelated slang word quiz "odd person" (1782, source of quizzical). Compare quisby "queer, not quite right; bankrupt" (slang from 1807). From the era of radio quiz shows comes quizzee (n.), 1940.