Wednesday, August 31, 2011

The Difference Between a Scientific Study and a Survey

Recently, I posted some information surrounding some recent studies which have once again shown a strong genetic component to autism.  This included three peer-reviewed scientific studies which have been published in reputable medical journals, and conclusions were drawn from the studies that were in line with what many previous studies have found... that autism is primarily genetic, and that there can be some environmental factors that contribute to a lesser degree.

When I wrote that post, I fully expected that Lowell Hubbs would appear out of the woodwork and attempt to discredit the studies by claiming they were biased or that they were secretly funded by some arm of "big pharma", but what I did not expect was the nonsense that he posted as a comment (shown here):

"You can not possibly be that dumb as to suggest autism is genetic? That is over the top for even the level of stupidity you have already put forth. How, in one generation did the genetic gene pool change from 1 in 10,000 autistic children to now 1 in 100 or at times even less than 100? The vaccines were steadily increased after 1988 and after the vaccine manufacturers were given complete legal immunity. They are now 49 shots of 14 vaccines before the age of 6. And you think there is no grey area, no questionable area anywhere? Who the hell pays you for this shit? Vax UnVax Study Results http://www.ageofautism.com/2011/08/vax-unvax-study-results.html?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+ageofautism+%28AGE+OF+AUTISM%29 Managing Editor's Note: From the Child Health Safety site: A new survey of 7724 participants shows unvaccinated children are healthier and have vastly fewer chronic conditions than the vaccinated. The survey is published here The Health of Unvaccinated Children, Survey Results."  ~Lowell Hubbs
Ok a few things need to be pointed out here.  First of all this comment was added soon after the original post was added to this site, meaning it is doubtful Mr. Hubbs even bothered to review the original studies as he would not have had time to analyze them or find any deficiencies in their methodology.  It isn't really a new phenomenon to have a vaccine conspiracy theorist simply ignore reputable science, but it does just show yet again how Mr. Hubbs is unable to even comprehend or fully understand the science he attempts to ignore. 

Had Mr. Hubbs responded with direct criticism of the studies or questions about the results it would be one thing, but it is obvious that rather than be bothered to actually review the source data, Mr. Hubbs instead just scans the titles and makes up his mind.  In fact, Mr. Hubbs has already made up his mind in regards to vaccinations and autism, so no amount of new data or scientific study can change it.  This is the basic premise of a vaccine conspiracy theorist like Mr. Hubbs... they don't allow the data to lead them to conclusions, but rather they form a conclusion and then spend all of their time finding data which supports it while ignoring data that challenges it.  This is not how science works - which is why Lowell Hubbs and his ilk will always remain nothing more than a distraction to the real science being performed each and every day.

So what else can we learn from Mr. Hubbs' comments?  Well... as sad as it is, Mr. Hubbs has displayed an ignorance to basic scientific concepts which honestly calls into question whether he is even fit to comment on these issues when it is so clear he has no clue what he is talking about.  I'm referring to the fact that Mr. Hubbs provides a link to a survey, and yet he confuses this with a study.  Mr. Hubbs believes the nonsense he has linked to somehow proves the unvaccinated children are healthier, and he honestly believes this is a smoking gun.

Let me explain the major flaws in Mr. Hubbs thinking.  First of all, the survey he linked to is just that... a survey.  This means it is nothing more than anecdotal evidence and is in no way considered scientific.  No research was performed to identify the people who responded to the survey, there is no indication that bias was addressed, and no mandate for those responding to the survey to even bother to answer honestly.  There was no direct contact between any doctor, scientist, or researcher and the survey respondents, and there was no analysis performed on medical records or test results.  The fact that Mr. Hubbs feels a survey is worthy of being considered a "study" only shows how poorly he understands the scientific process.

So for the sake of discussion, lets go ahead and examine this little survey to see what we find.  Let's look at the science behind how they gathered their data, and let's examine the methodology behind the survey itself.  The first glaring red flag here is who actually designed the survey.  It was distributed by the website www.vaccineinjury.info which is nothing more than an anti-vaccination website which seeks to label all vaccines as harmful as they try to sell various antivaxxer books (as any good antivaxxer will tell you, fear is very profitable).

Second, and perhaps more important that who distributed the survey, is HOW they distributed the survey.  According to their website, the responses they received were "[d]ue to social network pages and the help of many people who supported the survey".  So essentially they relied upon social media like facebook pages to distribute the survey, and then they even openly admit it was distributed by people who supported the survey!  Do these people even understand the concept of bias?  This is like asking a five year old to distribute a survey to other five year olds and then reporting that the survey says peanut butter and jelly sandwiches are the most healthy meal on the planet while playing kickball is the world's most popular sport.

These people just don't get it... and I doubt they ever will.

It gets better though.  Out of the 7,799 responses to the survey (none of which were validated in any way which means a single person could respond to the survey 300 times if they wished), only 582 of the survey respondents indicated that conventional medicine is their preferred medical treatment.  Considering that conventional medicine is actually the preferred treatment of well over 95% of the public, this figure shows how skewed this survey is, but there is no mention of this discrepancy by the authors.  Apparently they are more concerned with distributing their bias and they have no interest in pointing out the fact that there are numerous glaring flaws in their methodology. 

So what was the preferred medical treatment?  Well not surprising, 2,826 of the respondents relied upon homeopathy, 2,370 relied upon naturopathic medicine and 1,729 relied upon other medicine which according to the survey's authors is mainly chiropractic and supplemental.  This basically tells us of the 7,799 people who took this silly survey on their facebook pages or from anti-vaxxer websites that 6,925 of them (or 89%) rely upon unproven forms of 'alternative' medicine while only 7% use conventional medicine.  Don't ask me what happened to the other 292 survey respondents (4%) which apparently were not counted... I guess statistical accuracy or explaining major discrepancies in the data just isn't important to antivaxxers.

It is painfully obvious this survey is a joke, and not even a good one at that.  Even the most ardent anti-vaccination quackjob on the planet understands that most people use conventional medicine as their primary source of treatment, thus if you have a survey with drastically different results it doesn't take a brain surgeon (or even a research scientist or statistical analyst) to figure out the survey is heavily, heavily skewed and full of bias.

As entertaining as this is - it actually gets better!  The next piece of data we are shown indicates that over 99% of the survey respondents were happy that they didn't vaccinate their children.  It was actually 99.69% which is a statistical super-mega-amazing-majority! 

But wait... I thought this survey was supposed to show how much healthier unvaccinated kids were when compared to vaccinated kids.  If 99.69% of the kids in the survey are unvaccinated, that would equate to 7,775 of the 7,799 children which would only leave a maximum of 24 kids who are potentially vaccinated.  How do they expect to perform a valid comparison between group A of 7,775 people (99.69%) and group B of 24 people (0.31%)?  The fact that I even have to point this out to someone like Mr. Hubbs who has taken the survey at face value only serves as more evidence on how disconnected the antivaxxers are from reality.  Critical thinking skills, common sense, and any level of scientific understanding apparently don't exist within the antivaxxer community - and this little survey acts as a prime exhibit of those facts.

In fact, if you actually read on into the survey results, you will soon discover that they did no such comparison between vaccinated and unvaccinated children who were counted in the survey.  None whatsoever.  Instead, they are comparing their survey of unvaccinated children to other scientific studies which determine the rate of various medical conditions such as allergies or asthma.  Once again there is a severe failure to understand the scientific process and the fact that the distributors of this survey fail to understand the glaring flaws in their methodology is comical at best.

Where things really get interesting is when you drill down into the survey results a bit further and you come across their graphic which shows their survey results displaying autism cases in unvaccinated children.  The graphic they use is shown below:

Click to View in Full Size

Unfortunately there is a lot left up to the imagination surrounding this image, and no explanation is given for the numbers shown.  It is understandable that there would be very few if any cases of autism reported in children under the age of three since autism is often undiagnosed until the child is a toddler, but beyond that the numbers shown in this table seem rather high.

We are continually told that the rates of autism are roughly 1 in 110 American children and that this number has bloomed in the past decade, yet this graphic (if you average all children from 0-12 years of age) shows the rate to be 1.46%.  If you factor in all children through age 18 the rate averages to 1.25%.  So if the rate of autism in the general population is 1 in 110, that equates to 0.9%.  However this survey is suggesting that the rate of autism in unvaccinated children is anywhere from 1.25% to 1.46%... which is an increase over the general population of 38% to 62%!

Now let me be clear that I give no credibility to this survey nor do I feel it is in any way scientific.  However if the antivaxxers wish to consider this a valid survey, then by all means they need to be willing to accept the data, and that means an unvaccinated child is 38% to 62% more likely to be autistic than a vaccinated child.  Hey - I can only go off of the chart provided, and math doesn't lie, so I guess that is what they want us to believe.

Then again, the website does include a disclaimer which readily admits that they "cannot guarantee that the information provided is complete, accurate and current", so I guess that allows them to wipe their hands from all of the fear-inducing headlines that don't seem to mirror the actual results of their survey.  Funny how these antivaxxers always seem to have some type of a disclaimer, so that made me wonder... do legitimate peer-reviewed medical studies have disclaimers on the bottom that state the study's authors can't guarantee that the results are complete or accurate?  The answer is no - because real science needs no such disclaimer, and a legitimate study would never be published until the study's authors were confident the data was not only complete, but that it was accurate and current as well.

This it is more than obvious even to a casual observer that this survey is worthless.  I really was looking forward to a legitimate scientific study that includes a large enough control group to make it meaningful, but once again the antivaxxers have let me down.  Obviously I could go on and on about the flaws in this survey, the overwhelming unscientific bias that has blended into everything from the survey creation, to survey distribution, collection, analysis, and even production... but I think what I have shown in this one post alone is enough to prove that not only is the survey entirely non-scientific, but even using their own skewed and heavily biased data still shows a result which contradicts their own predetermined viewpoints.

This also goes to show that how the Age of Autism website authors have no understanding of what true science is.  The fact that their article states "[t]his is excellent work from an independent source" just goes to show they not only misunderstand the term "excellent" but they have no clue what it means to be an independent source.  Unfortunately most of the comments added to their article thus far appear to have bought into the bait hook, line, and sinker and most likely those individuals haven't bothered to objectively analyze the data.

Is there really any wonder why anti-vaccination conspiracy theorists like Lowell Hubbs are never taken seriously when they can't even be bothered to identify the fatal flaws in the very data they use as evidence?  The only thing this survey has proven is that the ignorant will remain ignorant, and the scientifically inept antivaxxers will remain scientifically inept.

Real science wins.  Again. 

8 comments:

  1. I think rather than continuing the he said / she said, trains passing in the night vitriol, there needs to be some acknowledgement and understanding of the available information from all sides.

    From the scientific side, it's not reasonable to say that just because a scientific justification cannot be found, it can't be so. That implies a completeness of scientific knowledge which any scientist worth his or her salt would deny. From the side of parents and concerned parties, there needs to be an understanding that drawing conclusions simply from available symptomatic evidence is going to be error-prone.

    The scientific community fielding the "real science" that unequivocally denies any ties between vaccinations and subsequent injuries (autism or otherwise) does itself and the community a disservice by portraying a facade that does not engender trust from the world community at large.

    The best way to move forward is for science to focus not on CYA, but rather to find concrete, unequivocal explanations for the observations and experiences presented by parents and concerned parties worldwide. These findings need to be complete, inclusive, and compelling. Like gravity. Like HIV causes AIDS. Like malaria is spread by a mosquito. Just some examples of incontrovertible science, as I see it.

    Posting articles like this tearing into the mechanisms of a survey doesn't help anything or anyone move forward. It just serves (again) to attempt to invalidate the concerns of parents and other involved parties that are trying to understand what's going on. When I read this and other articles, they always read as "case closed, here's what science says and you're wrong." It doesn't take a PhD to raise questions on the scientific studies published to date ... what's the impact influence of epigenetics? Specificall nutriepigenomics? From Wikipedia's own summary: If metabolic disturbances occur during critical time windows of development, the resulting epigenetic alterations can lead to permanent changes in tissue and organ structure or function and predispose individuals to disease. So - where are the studies that look at metabolic disturbances introduced by vaccinations? What about adverse effects introduced by drug interactions between vaccinations and a child's diet?

    Many of these things could be meta-studies, sure. But in the end, there needs to be a clear picture developed. What we have now is anything but.

    ReplyDelete
  2. Brice I agree with much of what you have written and will concede there is much more work to be done, however I'll disagree on only a few small points.

    First, I don't beleive mainstream peer-reviewed science has ever stated that they know there is no way vaccines (or anything else for that matter) is not involved. What they have said, and what they continue to show is that when studies are performed, there is no link that is able to be drawn between vaccines and conditions such as autism. Many have tried to show such a link, but the data just doesn't support it. On the flip side numerous studies have examined groups of vaccinated chidren versus those who remain unvaccinated and found the rates of autism are statistically neutral. This may not "prove" the absense of a link, but it more certainly creates doubt of one.

    Since you can't prove a negative we need to accept that currently there simply is no evidence to suggest vaccines lead to autism. Maybe one day a study will be able to show such a link even if it is an extreme small amount... but we aren't there yet, and the evidence thus far strongly suggests other root causes other than vaccines are to blame.

    Therein lies the rub. Vaccine conspiracy theorists such as Mr. Hubbs here or Jenny McCarthy seem to believe they are somehow more in-tune with reality than the scientists performing the research. They make bold claims and state that they 'know' vaccines cause autism, but when pressed for the data they are unable to produce one single peer-reviewed study that can support that claim. Their actions are nothing other than intellectually dishonest... and they know it.

    The second point I would disagree with you on is when you state that analysis such as this don't help anything. Actually I think when people dig in it helps a great deal, because it helps peel back the layers of dishonesty and misleading junk science that is presented as evidence.

    The bottom line is even if their conclusions were accurate, we would have no way of knowing because their methodology is flawed. I don't feel junk science does anyone any favors, and for those without the knowledge or intellect to be able to look beyond a flashy headline, this type of fear mongering and disinformation can be dangerous. It sends the wrong message to allow it to remain unchallenged, and it confuses the casual reader.

    The debate has no room for junk science regardless of which side it attempts to present. I firmly believe that reasonable people acknowledge a lot more reseach is needed and the case is far from closed, but at the same time I believe reasonable people also understand that there are those out there who manipulate data and create garbage to push their own agenda... facts and science be damned.

    ReplyDelete
  3. Nice try Hubbs - but I'm still not posting your comments.

    ReplyDelete
  4. Great post, thanks for sharing. The problem with science vs. surveys comes down to motivation, in my opinion. You can use survey templates to gauge the likelihood of some product being well received, or you can ask a group of Fox news viewers whether or not they dislike anything, any democrat is doing or saying, and then reporting that as the general American consensus. Whereas a scientific study might look a common genetic trait among said Fox viewers for no other reason than an academic one.

    ReplyDelete
  5. To put this debate to bed someone should do a long term study on those children who have not been vaccinated and see over the course of 50 years if they develop as many illnesses or conditions compared to those who have been vaccinated. Lets see if this sheep clown will publish my comment.

    ReplyDelete
    Replies
    1. Well when you call me a "sheep clown" how can I possibly resist?

      As to your idea for a long-term study, you may find this interesting:

      The "Vax vs. Unvax" Study Myth

      Delete
  6. I am studying health promotion post grad and I have a degree in health science. I had an underlying secondary thyroid condition while pregnant in 2006 but I didn't get diagnosed until this year the late diagnosis also lead to adrenal fatigue.

    I had to persist with indignant doctors to get tests done but pubic health wouldn't diagnose me they refused to because my so called TSH results were normal and they were extremeley rude treating me as though I was a hypocondriac. I had to fight them to get certain tests done under bulk billing then take the results to a private reputable doctor and pay for a dignosis who recongised my symptoms even though the TSH said that I was in range.

    Fast forward to present day and my beautiful child has autism and was diagnosed last year a painful process. My child was fully immunised I honestly believe vaccines alone is not the cause but before you shut me down I believe it is a contributor to a child developing autism. I think the culprit is low functioning thyroid and repeated vaccinations together.

    Why do I think this well in my case and with other women who are pregnant we naturally produce alot of estrogen which for some reason the body holds on to copper which is found naturally in alot foods such as chocolate, coffee, avacado and seafood a good diet of protein such as veal and beef can counter act this.

    But my copper build up interfered with my thyroid's T3 T4 conversion so it slowed my endocrine system down which all organs need to function. Which then slowed my liver function down which is essential to bind and dispose of heavy metals such as the copper. So I was in a catch 22 situation the more my liver slowed down the more copper accumulated inside my body because it could not get rid of it of which copper was the problem in the first place that slowed my thyroid.


    My health was compromised I was aneamic my hair fell out because hypothyroid was directing all nutrients in my body and was going into survival mode for a long time before I got pregnant. If this could happen for my hair, nails and skin just think what this would do to an unborn childs development. This is what I believe also compromised my childs health who's endocrine system was not fully developed or healthy at the time to take on the brunt off vaccinations.


    This has lead me to believe that its the mothers maternal health which is the telling genetic factor not the fathers alone. She should be in complete health so that it is not passed on to her child. When my child was born my child had jaundice and colic thanks to my liver dysfunction.

    Now that I am a parent of an autistic child and I want answers too the scientific community needs to get their research out there on what the causes of autism are because people don't know what the causes are that is why they are taking it upon themselves not to vaccinate.

    In my area I would love to do research because for some reason autism is prevelant just 50 minutes south from where I live. I would do my research for free no-one would have to pay me. I would approach all parents who have autistic children to take part in this research.

    Do you remember Lorenzo's oil I want to aspire to be that not waiting for the scientific community to find answers or the cure we need answers now. All parents of autistic children should never stop looking for answers I agree that it occurs naturally but nobody knows how it occurs.

    In time after I complete my postgraduate studies I hope to do my own research while using a preventative health lens that people with compromised health such as myself or thyroid should get interventions as soon as possible. If I am successful in proving those connections then we can go about putting particular interventions in place asap so that once women know they are pregnant test their thyroid health and if it cannot be corrected through interventions they should not have to get their children vaccinated.

    ReplyDelete
  7. So you admit you had numerous health conditions prior to being pregnant and/or during pregnancy such as thyroid conditions, adrenal fatigue, endocrine system issues, liver problems, anemia, and who knows what else yet you still somehow seem to bring it back around and place blame (at least in part) upon vaccines.

    And the scientific evidence you have to support this hypothesis is what exactly? Anecdotes don't quite work here - and you having a hunch of a believe is worthless unless you are able to prove it scientifically (which thus far you, or anyone else on the planet has been unable to do).

    It is fine to have a theory, but the issue is when people take unproven theories and try to persuade public behavior. We have seen the ramifications of what happens when people have knee jerk reactions to vaccines based upon someone's unproven theory - and it doesn't end well.

    What amazes me more than anything however is you claim to hold a degree in health science and are pursuing postgraduate work yet you state "the scientific community needs to get their research out there on what the causes of autism are".

    Are you kidding me? Are you simply not paying attention, or are you just ignoring the data that has been provided? There are dozens of studies that have centered around autism and although no smoking gun has been found, it also means there is zero scientific evidence to support a link between vaccines and autism. To suggest the scientific community is either not looking at the issue and/or withholding their research is absurd - and I cannot fathom how anyone who has studied science or healthcare could believe such a thing.

    Let me guess.... "health science" is really code for "complimentary and alternative medicine" or some other form of quackery. Maybe you've studied chiropractic or acupuncture or reiki or whatever else and decided the best way to sound as if you are educated in the field of science and medicine is to water down your specialty to simply be "health science".

    Either way one thing is certain - with views like yours you clearly did not attend a reputable medical school or established university with a reputation for teaching legitimate science.

    ReplyDelete

All comments are moderated and comments from obvious sockpuppet accounts as well as spam accounts that do not add anything of value to the discussion will not be published.