Cognitively Lumpy

What IQ tests really do and don’t measure.

I saw a particularly painful and teeth pulling twitter video tonight by an eminent Mr. Rogers scholar. In it, I witnessed a wonderfully smart person with a variety of diverse interests and experiences, with notes in hand, being slowed to a crawl trying to explain intelligence and IQ. It got me thinking …

This is the kind of pressure those words put on us! What is my IQ? Am I intelligent? How do I compare to the very ‘smartest’ people in the world? What does intelligent mean, anyway?

Since the video starred a self proclaimed ‘cognitively lumpy’ person, I was delighted to hear the speaker share some facts and figures mixed in with the theories. It is true that we don’t know exactly how to explain ‘intelligence’ or what factors make a person more or less intelligent.

There are many theories. It is fun to try them on and see how they fit in our heads, not unlike those bars with 172 types of beer that call to your soul as you pass by. It is just interesting what might be at the end of that journey. The most stout of them (the theories and the beer) have a lot of texture, character, fine ingredients, and are brewed with care.

I’m not sure if I can add value to the discussion, but as with the beer, I shall give it the old college try.

The true sign of intelligence is not knowledge but imagination. — Albert Einstein


When we talk about things we prefer to be clear about, we have no problems. It is when we want to create our own personal measuring stick that we run into trouble, particularly if we want to then use it to measure other people.

We don’t measure Olympic sprinters by their marathon times. We don’t measure the average rainfall in an area by the number of windows in the average building. We don’t measure time by counting the number of breaths we take while a snowflake falls. We would be laughed at and lose our membership in the Intellectuals Guild.

If we are trying to assess the value, level, position, or some other category in people, it would make sense to have something in mind that we could measure and that everyone would agree on. For distance, most of the world uses kilometers and it is easy to get GPS directions, even in the lesser known ‘miles’ unit. For prices of goods, each country tends to agree on a currency that means the same thing to everyone, like dollars.

When we want to inflate our egos, fool people, or just have something to brag about, we use our made up measurements. Psychologist W. Joel Schneider calls these types of assessments ‘folk concepts.’ This description does not mean to describe these concepts as irrelevant, outdated, or useless. On the contrary, ‘folk concepts’ such as polite, athletic, greedy, and intelligent have great cultural value. They are part of the glue that keeps our emotions, social responses, and behaviors moving along smoothly in between all of those smart activities.

A much more useful measure to share between people would be simply skill. At what level can a person perform a particular skill. At what skill level does a person play the trumpet? What is the level of quality, as measured by acid etching and x-ray analysis, of the weld joints performed on standard coupons using standard equipment. What is the mathematical ability of a first year electrical engineering graduate student? At what skill level can a person generate working and tested Python code during a 30 minute, 5 step skill challenge?

These are things that could be directly measured and applied to nearly any skillset. Intelligence is a fashion show. It is an intellectual beauty pageant. It is a karaoke contest being judged by the people who have tried all 172 of those beers on tap.

If we wanted to know something real about people, we would create a set of 20 or more tests and then let them pick the one they want to take to show off their strengths.

IQ Tests:

As they are commonly presented in the US, IQ tests are a measure of mathematical skill, simple problem solving/pattern matching, and English vocabulary. Most of the skills that get you a high IQ score can be practiced easily, if you can stand the boredom, by any college educated English speaker.

This shines a bright light on one of the major biases of IQ testing: there are some extremely talented people in countries with different languages and cultures than the US who will never be identified correctly by these pseudoscientific instruments. There are also many fluent English speakers who happen to have very strong skills in an area not tested on any tests.

People who are very good sculptors are not called ‘intelligent.’ Nobody calls a person ‘smart’ because they are fluent in Spanish. I’ve never heard of anyone judging an ‘intelligent’ plumber.

“I do struggle more with typically mathematical concepts … than with sortof verbal concept structures.” -

That is the most brave and creative way I have ever heard anyone say, “Math is hard and I like to talk a lot.” It is honest. It is real. It is the way nearly everyone feels about this stuff. Our species developed language and communication very far back in our history, with estimates ranging drastically from 50000 to over 2 million years ago.

It is well established that mathematical record keeping came much more recently, followed by written languages, contracts, books, and the printing press. After that, technological developments have accelerated the pace and widespread use of all knowledge and information skills and they have become more or less equally accessible to everyone in the developed world.

IQ tests are given to assess a very limited scope of abilities, most of which take years of living in a particular environment amongst a particular group of people. They are a measure of our bias towards the ‘good student’ who can sit still in a desk and listen to a droning lecture before happily working for 30 minutes on a worksheet full of problems that generally have nothing to do with any real situation. The tests are always written in English in the US and use a particular vernacular that will only be familiar to eccentric advocates of the academia.

To summarize: IQ tests are designed to assess the skills that

  • are accessible to the smallest number of people
  • are based primarily on American English in the US, with all of the cultural background experiences and biases that come with it
  • contain words, phrases, succinct vocabulary, and even cultural and religious overtones that lead to large biases
  • require experience with a fair amount of technology
  • require the development of skills, starting at a very young age, that are of nearly no use for a large amount of the population

They are thought to be predictors of future success, but continuously show up in research as lagging indicators, at best. In fact, it is likely that these factors are residual effects of experience, cultural environment, intellectual curiosity, exposure to art and music at a young age, extensive travel, high level family discussions, and any of a number of similar factors.

IQ tests are a measure of how much a person is like the person who wrote the test.

They are a measure of how closely a person matches college professors and psychology researchers. The real challenge is getting kids to understand that the goal isn’t to be like someone who makes tests for a living, it is to find something you are passionate about and look for ways that you can help others solve their problems.

Even if IQ tests did work, most of the important skills and outcomes in our society are social endeavors. It is through having compassion for others and understanding that some suffering is inevitable that we can find our place.

(Edit: I have received several comments regarding the exact nature of IQ tests, what they are used for, and how they are produced. While I have designed many test instruments for many reasons over the last 20 years, but I am not an expert in IQ tests. The comments are most likely correct.

This was intended to be an informal essay regarding the stigma caused by inappropriate use of IQ tests and how the idea of ‘intelligence’ is flawed in general. It was also intended to be a somewhat ‘light’ and ‘accessible’ story about some of the damage being done by overreliance on tests, whether official IQ tests or others. I hope that clarifies any confusion.

Thank you for the comments and DM’s! The discussion is thought provoking!)

If you have a differing opinion or some data to share, please leave a comment or find me on twitter (@skeptycal)

BONUS: Our education system isn’t helping!

(Since I was an award winning chemistry teacher for twenty years, I am qualified to give my opinion on this.)

The critique of our education system near the end of this video should be eye opening to anyone who was unaware. This is what it actually looks like to a very thoughtful and insightful young person. This is how kids see school now. In many ways, they are exactly right.

Many of their classes are worthless to them, with lectures that ‘may as well be pre-recorded.’ The oversimplified tasks and skewed assessments are a joke. This description does not represent all teachers, but there is an alarmingly high number of completely unqualified teachers … and the number is growing …

Mike is a software developer, chemist, motivational speaker, parent, and musician who writes about creativity and human nature.