In the antivirus industry, false positives run amok. No matter which vendor you choose to buy from, you’re going to have the occasional hiccup. Those small snafus are usually easily sorted out, with minimal downtime or expense. Once in a while, though, someone falls asleep at the wheel and all hell breaks loose. Such was the case with McAfee on Wednesday.
The company rolled out an update that took down Windows XP computers around the world. The company then issued a statement claiming that “less than .005% of McAfee users were hit by the update,” which misidentified a legitimate SP function as a virus and killed it. The results were computers locked in a reboot loop. Unfortunately, it appears that there are many thousands of computers affected by this disaster… adding up to a far higher percentage than McAfee is apparently willing to admit.
Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly determine that the poisonous update from McAfee threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that “remediation time is estimated to be 30 minutes per user, ” says Solera CEO Steve Shillingford. “Estimating $100 per hour, this organization’s lost time alone can be conservatively estimated to cost more than $2.5 million,” says Shillingford. “And that does not factor in lost productivity while users are down.” The fix issued by McAfee is a long and arduous one, likely not to be attempted by computer novices.
Others affected by the so-called “false positive situation” include hospitals, police departments, major universities and retail stores. Hospitals in Rhode Island had to refuse treatment for all but life-threatening situations. State police officers in Kentucky were without computers in their patrol cars while the IT department scrambled to fix machines. Australian supermarket behemoth Coles was hit so hard that 10 percent of its point-of-sales terminals were taken down. The company was forced to shut down stores in both western and southern parts of the country.
McAfee apparently sent an email to their larger enterprise customers to explain the situation. According to documents sent to Ed Bott, thorough testing was not even done prior to the update being released. The email admits that “Some specific steps of the existing Quality Assurance processes were not followed: Standard Peer Review of the driver was not done, and the Risk Assessment of the driver in question was inadequate” and that “there was inadequate coverage of Product and Operating System combinations in the test systems used. Specifically, XP SP3 with VSE 8.7 was not included in the test configuration at the time of release.”
This blows my mind. Windows XP SP3 is the most widely-used configuration in the enterprise desktop environment. I fail to understand how such a key testing phase could have just been “overlooked” or bypassed.
The most troubling aspect of the entire situation is McAfee’s seemingly cavalier attitude towards the event. The company apologized in a blog post on Thursday, but little has been said about the entire subject. Meanwhile, customers are complaining loudly all over the McAfee community forums, and they want answers. One commenter called for McAfee to “man up and own up to what happened, instead of trying to sugar-coat it and make it seem as though this is no big deal.”
It will be interesting to watch how this will play out as more information comes to light. I have a feeling we have only just begun to hear about the full effect the McAfee mess had on customers all over the world.