What Is Item Response Theory (IRT)? Why IQ Scoring Isn't a Raw Count

Item response theory (IRT) is a scoring framework that estimates a test taker's ability from which questions they answered correctly, not how many. Most major modern ability and intelligence tests use it, and this site scores its test with IRT. This article lays out the mechanism with diagrams.

The weakness of raw scoring: are two scores of 20/20 the same?

Turning the number correct straight into a score has an obvious flaw: someone who got 15 easy items right and someone who got 15 items right including hard ones both end up with 15 points.

For the same reason, raw scores depend heavily on which set of questions you happened to receive. Change the mix and the same person's score changes, which makes it impossible to compare different test takers on one ruler. IRT solves this by treating the item's parameters and the person's ability separately.

Item characteristic curves: modeling how each question behaves

In IRT, every question carries a curve describing the probability that a person of ability θ (theta) answers it correctly — the item characteristic curve (ICC). Two parameters characterize it:

• Difficulty (b): the ability level at which the probability of a correct answer is 50%. The further right the curve, the harder the item
• Discrimination (a): the slope. The steeper it is, the more sharply the item separates ability levels

The three curves in the figure share a discrimination value and differ only in difficulty. At the same ability θ, the probability of getting the easy item right is high and the hard item low. Turn that around and it means a correct answer on a hard item carries more information about ability.

The estimate shifts with which items you got right

IRT scoring works backward from your entire response pattern to the ability value θ that makes that pattern most likely. Two people can both score 15 out of 20 — one missing the hard items while sweeping the easy ones, the other taking hard items and dropping easy ones — and come out with different estimated θ.

θ comes out on a mean-0, SD-1 scale, so converting to IQ is a linear transformation: IQ = 100 + 15 × θ. By that formula θ = +2 corresponds to IQ 130 and θ = −1 to IQ 85. Why IQ is a relative indicator with a mean of 100 and an SD of 15 is covered in what is the average IQ.

Why adaptive tests can be short

IRT's other advantage is that items close to a test taker's ability yield the most information. Exploit that and you can switch the next question by response — slightly harder after a correct answer, slightly easier after a wrong one. That is computerized adaptive testing (CAT).

Because adaptive testing skips items far from your ability (obviously trivial or obviously impossible), it reaches comparable precision with far fewer questions. That information design is why a test of around 20 items can still produce a meaningful estimate.

Error still does not go to zero

IRT is a framework for improving precision, not a way to eliminate error. Every estimate carries a standard error, and it grows at the tails of the distribution (very high or very low ability) where items run short. The principle that no single result should be treated as a fixed value is unchanged (see how accurate are free IQ tests and does practicing for IQ tests help?).

Take an IRT-scored test yourself

BrainRank's free IQ test scores 20 questions across spatial reasoning, pattern recognition, logical reasoning, and classification (about 10 minutes) with item response theory, producing an estimated IQ, deviation score, and percentile on the mean-100, SD-15 scale. The question types in each domain are covered in our problem category guides, and the scoring detail in methodology.