In an effort to make its services more timely and relevant, the federally supported has launched a series of 鈥渜uick reviews鈥� to assess the methodological soundness of studies that have made national news.
鈥淭here are all kinds of studies coming out daily that are cutting-edge,鈥� said Mark Dynarski, who directs the clearinghouse, which is based at Mathematica Policy Research Inc. in Princeton, N.J. 鈥淭he idea is whether the clearinghouse can help the public in determining whether studies are well crafted, if they use sound methods and make good inferences.鈥�
Created in 2002 by the U.S. Department of Education鈥檚 Institute of Education Sciences, the online clearinghouse vets the research base for programs, practices, or policies that educators might try out in their own jurisdictions. But the first reviews were slow in coming, and few studies met the clearinghouse鈥檚 strict evidence standards, leading some critics to dub the project the 鈥渘othing works鈥� clearinghouse.
The clearinghouse plans to post the new quick reviews, in contrast, within a month of a study鈥檚 public release. The turnaround time is faster because reviewers are evaluating a single study, rather than a body of research.
The first five reviews, starting late last month, give mixed assessments of recent studies on after-school programs, a student-athlete drug testing program, a cash-incentive program for teachers and students of Advanced Placement courses, a computer-based program for teaching algebra, and on whether 6th graders are better off in middle schools or elementary schools.
Some of the reports are already drawing sharp criticism from study authors.
Research Into Practice: The Push For Usability
This four-part series examines the movement to make education research more 鈥渦sable鈥� and explores some efforts to connect the worlds of research and practice.
鈥淭his is all politics. I鈥檓 telling you from the bottom of my heart,鈥� said Dr. Linn Goldberg, a professor of medicine at Oregon Health and Science University, in Portland. Clearinghouse reviewers determined that the study Dr. Goldberg led on school-based drug testing for student-athletes was 鈥渘ot consistent with WWC evidence standards鈥� 鈥攖he lowest of three possible ratings.
The other rating choices are 鈥渃onsistent with WWC evidence standards鈥� and 鈥渃onsistent with WWC evidence standards with reservations.鈥�
Tough Standards
In the study, published in November in the peer-reviewed Journal of Adolescent Health, Dr. Goldberg and his colleagues evaluated the effects of a drug-testing program in which high school student-athletes were randomly selected for unannounced tests for alcohol and drug use.
After two years, the researchers found, students taking part in the testing were no less likely than their control-group counterparts to be using prohibited substances. Dr. Goldberg said the findings, which were widely reported in the press in February, run counter to the Bush administration鈥檚 stance favoring such programs.
In their review, What Works Clearinghouse analysts said the findings were inconclusive, in part, because of the high attrition rates among the schools and students involved. Seven of the 18 schools that signed up for the study dropped out, according to the clearinghouse鈥檚 report. In addition, 29 percent of students in the drug-testing program and 19 percent of the control-group students failed to return final surveys.
However, Dr. Goldberg said that the schools withdrew before the drug-testing programs began in any school, and that they did so in response to a threatened lawsuit. Final student data are missing, Dr. Goldberg added, because students either graduated or quit their teams the next year.
鈥淚t鈥檚 natural attrition,鈥� he said, 鈥渁nd it鈥檚 much lower than we expected.鈥�
For his part, Mr. Dynarski of the What Works Clearinghouse denied that the study was selected to serve as a political target. He said reviewers chose it because it was published within the time frame the clearinghouse had set for its first round of reviews. Although federal education officials can refer studies to the clearinghouse under the new 鈥渜uick review鈥� guidelines, none of the studies in the first round came to the clearinghouse by that route, he added.
鈥淎ttrition is a legitimate concern,鈥� Mr. Dynarski said. 鈥淐learinghouse standards pay a lot of attention to attrition, and here there is a 10-percentage-point difference between the control and experimental groups.鈥�
The clearinghouse also gives a 鈥渘ot consistent鈥� rating to an eight-state study of 35 high-quality after-school programs that was first reported in 91直播 last November. (鈥淗igh-Quality After-School Programs Tied to Test-Score Gains,鈥� Nov. 28, 2007.) Deborah Lowe Vandell, the study鈥檚 lead author and the chairwoman of the education department at the University of California, Irvine, said the reviewers鈥� determination was not surprising, given the narrow range of study designs that pass the clearinghouse鈥檚 evidence screens.
鈥淎 clinical trial [one of the designs that pass muster] is an important standard, but it represents only one type of research evidence,鈥� she said.
Clearinghouse reviewers gave a 鈥渃onsistent鈥� rating to a randomized study that found positive academic effects for students using SimCalc Mathworlds, a software program for teaching algebra concepts. Two other studies鈥攐ne on a Texas effort to give cash bonuses to students and teachers for high scores on AP tests and the other on the effects of teaching 6th graders in middle schools鈥攚ere rated 鈥渃onsistent with WWC evidence standards with reservations.鈥�
