Maybe the first thing to remember is which characters use this word–in the entire Austen canon, it is used by Isabella Thorpe, John Thorpe, and Tom Bertram. These are characters who believe themselves to be at the height of “cooldom”–and it’s also clear that Austen believes them to be less than reasonable people. The term was very much slang. When the Thorpes use it, they are often inflating their own discernment or judgment

“Quiz” can be a verb or a noun.

John Thorpe’s, “Where did you get that quiz of a hat?” implies that the hat is an oddity.

A person can be a quiz–as in someone whom you might tease. When you call someone a quiz, you’re saying that you see him or her as being somewhat strange. Isabella uses it to describe men she believes are ridiculous in some way or other.

The act of quizzing is the act of teasing.

John Thorpe uses it in both senses when he says to Catherine, “…let us walk about and quiz people. Come along with me, and I will show you the four greatest quizzers in the room; my two younger sisters and their partners. … at last he walked off to quiz his sisters by himself.” He is teasing people,and he says his sisters and their partners are odd.