The scientific method is the greatest of inventions: when used to organize and analyze the flawed output of we flawed humans, it leads to truth and discovery. It is how we sift the gems of progress from the rubble of short-sighted human nature, magical thinking, willful ignorance, and other self-sabotaging but entirely natural behaviors.

The scientific community doesn't produce an output of nice, neat tablets of truth, pronouncements come down from the mountain, however. It produces theories that are then backed by varying weights of evidence: a theory with a lot of support stands until deposed by new results. But it's not that neat in practice either. The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape.

So how do you read the output of the research community for a particular topic of interest? Firstly you have to recognize that the output of any single researcher is meaningless when considered in isolation. At least a good half of the results produced by scientists are in some way flawed. That is par for the course when working at the cutting edge of new knowledge, and no slur upon the hard work of those involved - they know the odds. The high failure rate of risky ventures like scientific research is why we need the scientific method to weed out those failed results as time progresses and new results emerge. So when you read up on a topic of current research, you need to read around the subject: dig in to various sources, find the work of multiple scientific groups, and assess the support for different theories and interpretations.

Here is a crude representation of the sorts of things you are likely to see:

• One group says "A is true"

  The only reliable conclusion you can draw from this is that someone is looking into A. If you're interested in A, it's a hopeful sign: perhaps other groups will join in in the years ahead.
  

• One group produces a demonstration to support "A is true"

  A demonstration of A makes it much more likely that other research groups will soon become involved. At this stage, the demonstration is more a sign that there is something interesting going on here than a sign that A is true. For example, if you manage to extend maximum life span in mice in support of A, then you've certainly found something interesting - but it may or may not be A.

• A couple of groups say "A is true", a couple say "A is not true"

  This tells you that A has enough potential if true to attract investigative funding, but it is as yet poorly understood and not well studied. If A were a business plan, a venture capitalist might bet on it. But you can't draw any firm conclusions as to whether A is true from a few contradictory results: you must await further evidence.

• Ten groups say "A is true", and ten say "A is not true"

  This shows that scientific investigation into A is in full swing, and that scientists and their funding sources consider A to be important in its field. When many groups are finding contradictory results, it usually means that there is some unknown factor at work yet to be discovered, and researchers are on the brink of finding it. When that factor is illuminated, it will explain how "A is true" and "A is not true" can both be correct under specific circumstances.

• Ten groups say "A is true", and one says "A is not true"

  Here, that "A is true" is either largely accepted or about to be largely accepted. There will always be dissenting research even for theories supported by overwhelming evidence. The small level of dissent may later be validated by more research results (leading to the "ten groups for and ten groups against" position above), but more likely it is simply incorrect. An outside observer really has to go with the majority position here.

In reality, things are of course more complex than this. It's rarely as simple as "A is true"; more usually you'll see many discrete versions of A competing, and research results lend support to several versions of a given theory. Additionally, you can rarely consider A in isolation; it's usually related to theories B, C, and D, such that it's helpful to look at what is going on in other areas as well when considering how much support there might be. But the concept of weight of support still applies: look to see how many researchers support a given theory.

The only cure for being unable to read the output of the scientific community is to dive in and practice; understanding research is a skill, no different from any other. Start with the press releases aggregated by sites such as EurekAlert! or ScienceDaily, and when you feel you have a handle on that, you can dig in to the scientific papers at PubMed or similar resources. This isn't a skill you can pick up overnight, however, and you certainly can't expect to gain an good read of a given field in less than a few months of casual research. You have to keep tabs, follow along as new information becomes available, dig into the archives, assemble your own view of what is going on, and follow up on your questions as they arise - in general treat it as you would any knowledge hobby.

The end result will not just be that you know more; it will be that you have become better at learning to know more. That's a worthwhile goal.

Posted by Reason

Share |