Testing That Je Ne Sais Quoi

By Rebecca Winters

Psychologist Robert Sternberg's first field study in intelligence took place in grade school, when poor scores on IQ tests convinced him he was a "dum-dum." Largely thanks to an exceptional fourth-grade teacher, Sternberg managed to shed his self-doubt, improve his grades and go on to attend Yale University, but he never shook the sense that traditional tests are missing something. "You don't get to the top in life just on your IQ points or your SAT score," says Sternberg, now a professor at Yale and president of the American Psychological Association (APA). "You have to psych out the system. How do you measure that skill?"

For three years, Sternberg has been working on a new test to augment the SAT, one that asks students to write captions for New Yorker cartoons, dictate stories into tape recorders and persuade friends to help haul a bulky mattress up a flight of stairs. These unorthodox tasks are designed to measure the creative and practical skills that Sternberg says are crucial to success in college and in life but are ignored by the typical pencil-and-paper exam. If Sternberg succeeds in quantifying these types of intelligence--and linking them to concrete accomplishments--his efforts may change forever the way colleges pick their students.

About 800 freshmen at 13 colleges took a trial version of the test, called the Rainbow Project, in 2001. Teams of judges scored the creative portions for humor and originality. The raters graded the practical sections on the basis of conformity to social norms. The more closely students' responses matched the average test taker's, the higher their scores.

Those who took the test in its early phase were volunteers rather than a random sample of undergraduates. But the preliminary results, which Sternberg presented in August at an APA conference, were dramatic. The Rainbow Project was nearly twice as successful at predicting students' first-year college GPAs as their SAT scores had been. The College Board, which produces the SATs, is funding Sternberg's research because the ability to predict college performance from a test--any test--hasn't improved much in 50 years, says Wayne Camara, the board's vice president of research.

Another impetus for the board to explore alternative tests is the persistence of gaps in SAT scores between racial and ethnic groups. Here, too, the Rainbow Project shows some promise. On the practical-intelligence portions of the test (the part in which students persuade friends to haul the mattress), there were no differences in scores between groups. On the creative portions, the differences were considerably smaller than they are on the SAT. And in some sections, groups that traditionally fare poorly on standardized tests thrived. Native Americans did especially well on the oral part.

In the next trial phase, Sternberg will expand his study to 5,000 to 10,000 students, who will take the test next spring and fall. These students will be followed for four years and will be scrutinized more closely than the first group was. In addition to GPAs, Sternberg will look at how well the students adapt to college socially and whether or not they graduate.

Widespread adoption of the test--a possibility several years and hundreds of thousands of research dollars away--would create some new dilemmas. For the time being, students know that their answers to quirky questions--like how to handle gossipy co-workers--don't matter. But if in the future the answers affect their college admissions, test subjects may be tempted to bluff--a problem that employer-administered personality tests have. "You can't fake solving a math problem," says Linda Gottfredson, a professor of education at the University of Delaware. "You can fake conscientiousness." For now, however, Sternberg's task is to persuade more colleges to take his offbeat exam seriously. In other words, he's got to prove he can psych out the system. --By Rebecca Winters