Yesterday I wrote a post that was critical of Amanda Ripley’s exclusive focus on the PISA test in her new book, The Smartest Kids in the World. There are other international 
tests that American kids do better on, after all, so looking solely at PISA makes our educational outcomes look worse than they probably are. Ripley emailed me the following response:
First of all, I am excited whenever anyone is interested in a debate about TIMSS versus PISA! I geeked out on this debate for many months while working on the book, and it was a pretty lonely existence—so I am happy to have an excuse to talk about it.
I agree it is critical to be skeptical of PISA—or any test or metric. All have their flaws. Which is why I spent a long time learning about all the various international tests—studying sample questions, reading about their strengths and weaknesses and analyzing their results over time.
In fact, I used many different data points to decide which countries to feature in the book, including high school graduation rates, college attainment rates, spending per pupil, rankings of national competitiveness and other economic indicators, as well as test data from TIMSS, PIRLS and NAEP.
As it turns out, international test data is strongly correlated from one test to another. (The correlation between TIMSS 2007 and PISA 2007 was 0.93.) But as you note, there are some differences between PISA and TIMSS findings. In the end, I made a very conscious decision to prioritize PISA findings for two main reasons:
- PISA is a test administered to 15-year-olds, which means it catches kids closer to the end of their compulsory schooling. TIMSS is given to 4th and 8th graders, which is useful, too, but I was most interested in the cumulative effects of countries’ education systems, rather than the midpoint.
- Unlike TIMSS, PISA was designed to test students’ abilities to apply knowledge to solve real-world problems and think for themselves. (TIMSS is a test of school curriculum.) I was most interested in those higher-order thinking skills, since they are increasingly valuable in the modern economy. To see if the hype on PISA was true, I took the test myself, and I found it to be a remarkably sophisticated test.
You are right that American kids do better on TIMSS, especially in reading. And you are right that many people exaggerate our failings relative to other countries. It drives me nuts. Which is why I went to great lengths throughout the book to avoid such hyperbole.
I feel weird quoting myself, but just in case you don’t believe me, this is from p. 4: “The vast majority of countries did not manage to educate all their kids to high levels, not even all of their better-off kids. Compared to most countries, the United States was typical, not much better nor much worse…Our elementary students did fine on international tests, thank you very much, especially in reading. The problems arose in math and science, and they became most obvious when our kids grew into teenagers….”
Anyway, this is a healthy debate to be having, but some things are fairly clear. We now have a lower high-school graduation rate than about 20 other countries. Our young adults perform far below young adults in many other countries, including Finland, especially in numeracy, according to the results of the new PIAAC test (yet another ridiculous acronym!).
There is no need for alarmism, but there is need for concern. Even our wealthiest teenagers, those in the top quartile of the country based on socioeconomic background, perform 18th in the world on the PISA math test compared to wealthy kids worldwide. And those kids have highly educated parents and attend some of the best resourced school in the world.
We could do better. We could do worse. Whatever we do, I am grateful for the chance to debate these complicated matters in an open-minded and civil way.
For now, I’ll let Ripley have the last word on this. But I’ve put The Smartest Kids in the World on my reading list, and I may have more to say after I’ve made my way through it.
