I received an email today regarding questions about ISI's impact factors. There is an editorial by Mike Rossner, Heather Van Epps, and Emma Hill published in the Journal of Cell Biology and is available at http://www.jcb.org/cgi/content/full/179/6/1091

The editors from the Journal of Cell Biology and the Journal of Experimental Medicine and the Executive Director of the Rockefeller University Press reported their inability to verify published impact factors using data provided provided by ISI itself. They are questioning the validity of ISI's dataset and ISI's published impact factors.

The journal editors and Rockefeller University press bought ISI's data for The Journal of Experimental Medicine, The Journal of Cell Biology, and The Journal of General Physiology and some of their direct competitor journals. According to the editorial their intention was not to question the integrity of ISI's data, it was to to determine which topics were being highly cited and which were not.

Upon examination, they discovered two immediate problems with the data. First, there were numerous incorrect article-type designations. Second, the numbers did not add up. The total number of citations for each journal was substantially fewer than the number published on the Thomson Scientific, Journal Citation Reports website. The editorial states that the difference in citation numbers was as high as 19% for a given journal. The editors found that impact factor rankings of several journals were different and were affected when they used the data purchased from ISI.

When questioned about the discrepancy, Thomson Scientific replied that there were two separate databases (one for their Research Group and one for the published impact factors) and the authors were using the wrong set. After purchasing the right set, the authors discovered data still did not match ISI's published impact factor data.

Yikes!
ISI's impact factor has become the standard by which people determined "quality and "impact." Love it or hate it, it was the standard. However this is first time I think people have discovered that the very data that ISI is using is completely suspect. IF that is true then there is some definite reason for concern. Hidden and incorrect data is a serious offense. As the editors said, "Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific's impact factor, which is based on hidden data."

It should be interesting to see if Thomson Scientific has a response and what it might be.