A Reminder that Science is Hard and Studies are Often Flawed

Scientific work is very challenging, and most projects have a significant chance of failure - meaning they generate no good data, or worse, bad data that looks good enough to use. That's not an issue in the long term, however. The scientific method is a process by which a good rate of progress and reliable data can be produced from many diverse efforts that individually have poor odds of success.

This means that you shouldn't read too much into the results of any one study. The odds of it being flawed in some way are fairly good, especially if it's recent. The scientific consensus on any given topic only comes about after a great many researchers have examined and published, and fought long and hard over discrepancies and points of confusion - and in enormously complex and rapidly changing fields like biology and medicine, consensus is often hard to find.

As a case in point, and a good insight into what actually goes into making the scientific sausage, here is an update on a recent study of longevity genes:

Remember that Science study from last week linking a whole bunch of genes - including unexpectedly powerful ones - to extreme old age in centenarians? Newsweek reported that a number of outside experts thought it sounded too good to be true, perhaps because of an error in the way the genes were identified that could cause false-positive results. Since last Thursday, they’ve been trying to figure out what might be lurking in the data, and now there’s a suspect: a DNA chip called the 610-Quad, which is used to identify and sequence the chemical letters of DNA, and which has an apparent tendency to get some small but critical details wrong. The flaw with the chip and the way it was used could cast serious doubt on the study’s strongest results, suggesting that they stem from a lab mishap rather than a real link to long life.

The great thing about this modern age of low-cost communication is that flawed papers can be pulled apart fairly rapidly when there is interest in the community. All bugs are shallow given enough eyes, and science can work exactly like the open source software development community in this respect. The publication model of the future is far more dynamic than the present model of journals: publish early, publish often, publish as a part of engaging a community in conversations on the research. The more collaboration that occurs along the way from ideation to completed research the better.

Comment Submission

Post a comment; thoughtful, considered opinions are valued. New comments can be edited for a few minutes following submission. Comments incorporating ad hominem attacks, advertising, and other forms of inappropriate behavior are likely to be deleted.

Note that there is a comment feed for those who like to keep up with conversations.