"Take the case of the medical controversy surrounding the safety of calcium-channel blockers, a class of drugs for heart disease. One study discovered that 100 percent of the scientists who found and published results supportive of the drugs had received prior support (free trips, research funding, or employment) from the pharmaceutical companies; but only 37 percent of those critical of the drugs had received any such prior support. "His statement was based on a scientific paper published in The New England Journal of Medicine in 1998. Details of the research can be found here:
Conflict of Interest in the Debate over Calcium-Channel Antagonists
Wow, these results are staggering. 37% of doctors were critical of a particular form of drug. But when some form of support is involved all doctors became in favor of such drug. This and other related research cited at the end of this article scientifically proves (at least from a psychology and medicine perspective) that industry-supported evaluation or testing of security related products such as antiviruses, IPS's, etc. have an influence on the quality and outcomes of their results.
Examples of such type of research studies in the security industry are:
1. Symantec funded an antivirus testing by PassMark: Consumer Antivirus Performance Benchmarks
2. Symantec sponsored another antivirus evaluation by Dennis Technology Labs: PC Anti-Virus Protection 2011
3.
4. Microsoft sponsored two NSS Labs tests for comparing the security of IE8 with other browsers:
http://arstechnica.com/microsoft/news/2009/08/microsoft-sponsors-two-nss-reports-ie8-is-the-most-secure.ars
5. Trend-Micro commissioned West Coast Labs Anti-Spam comparison tests: http://it.trendmicro.com/imperia/md/content/uk/whitepaper/wp06_wclantispamrpt_090317us.pdf
The results of these studies are not surprising. Symantec was ranked first by Dennis Technology Labs and PassMark.
The reason I am blogging this, is because I have come across a lot of CIO's and security experts who still believe and take into granted the results published by such kind of studies. Its even a pity to see security gurus from notable organizations such as SANS fall into this and cite these results.
For more information please see:
- Study: Industry-Sponsored Research Yields Favorable Results a Majority of the Time: http://www.doctorpundit.com/index.php/2010/08/03/study-industry-sponsored-research-yields-favorable-results-a-majority-of-the-time
- The uncertainty principle and industry-sponsored research: http://www.ncbi.nlm.nih.gov/pubmed/10968436
- Pharmaceutical industry sponsorship and research outcome and quality: systematic review http://www.bmj.com/content/326/7400/1167.full
- Source of funding and outcome of clinical trials - Journal of General Internal Medicine http://www.springerlink.com/content/r654521305u8547k/
Posts
You are incorrect about #3 and #4 (NSS Labs). Neither Trend Micro nor Microsoft has ever sponsored any test by NSS Labs.
ReplyDeleteThe link you provide states that Trend Micro "endorses" the methodology - meaning that they agree with it. And the link you provide re: Microsoft is to an old article by Ars Technica. Both NSS Labs and Microsoft have since made it clear that Microsoft has never sponsored a test by NSS Labs.
In fact, NSS Labs CTO has repeatedly said that if you want to know what is wrong with testing "follow the money trail". And that the problem with vendor sponsored testing is that no VP of Marketing who intends to keep his job will ever pay for a test where his product does not come out on top. This is why the bar is so low in AV testing - the business model every single testing organization (with the exception of NSS Labs) is based upon vendors paying for testing.
There is a possibility that you could be right about #3, other than that, everything else you mentioned in your comment is wrong.
ReplyDeleteFirst, this is a blog discussing about events that has occurred and not a news media reporting current events. By using the fact that the Ars Technica article is "old" as an argument does not invalidate #4, falsify the information cited from that article and disproves Microsoft never sponsored NSS Labs.
Second, if you have actually read the article, it clearly states at the end:
"Rick Moy, president of NSS Labs, sent us a follow-up e-mail to tell us that it was Microsoft's online security engineering team (not marketing) that hired NSS Labs to do recurring benchmark testing so they could improve their services."
Third, NSS Labs themselves have also acknowledged that they have received Microsoft funds for the browser tests as stated on their blog here:
http://nsslabs.blogspot.com/2009/03/web-browser-security-study-socially.html
Fourth, prior to September 2009, NSSLabs business model for several year was mainly driven by vendor funding. Please see:
http://www.networkworld.com/news/2009/091009-nss-labs-independent-testing.html
http://www.thetechherald.com/article.php/200938/4426/NSS-Labs-to-focus-on-non-funded-testing
As according to the article, NSS claims that after Sep 2009, they "now shifting its focus to do many more independent self-funded projects". There are several conclusions based on this statement:
1. NSSLabs receives vendors paying for testing which is contrary to what you have said
2. Rick Moy saying that they "will" be doing non-sponsored tests, does not mean that NSSLabs will not rule out any further vendor-based testing post Sep,2009.
3. The antivirus testing that Trend micro endorsed were performed in/before July 2009, thus before NSS labs publicized that they will focus into non-funded testing. However, I agree that this does not sufficiently prove that trend micro supported the tests.