Note: This is the third in a multipart series on the history of the antivirus and security industry by a long time insider (Part One)(Part Two). We will explore how antivirus and antimalware technology works, and why a 1980's solution is no longer applicable to the current threat landscape. The series will conclude with solutions and recommendations on where we might all go next.
The late 1980's and the 1990's were a good time for the antivirus industry as more and more file infector and boot sector viruses blossomed to the point where infected systems were commonplace and everybody was scooping up antiviruses to protect their systems.
The industry made money hand over fist. But there was a new threat in town gathering steam in 1997, following the release of Windows 95 and the distribution of a number of inexpensive and even free compilers, which easily created native programs for it in the form of "trojans."
The first widely distributed "trojan" came from France named "Sockets de Troie" which was designed for Windows95 and resulted in quite a bit of damage, forcing the original author to apologize and offer a manual cure for this trojan, reminiscent of the "(c)Brain" virus an entire decade earlier. And the now ten year old antivirus industry failed to detect it for nearly a year, some as long as two years after the fact.
1998 was the "year of the trojan" bringing forth such nasties as "Netbus", "Master's Paradise", and of course, many others including the granddaddy of them all: "Back Orifice" which finally got the media's attention.
From the antivirus vendors though, the old PR machine wound up in earnest and they claimed to handle them just fine. At the time, I was involved with Privacy Software Corp and our primary products were the NSClean, IEClean and ShareClean internet privacy cleanup tools.
Our business then was making browser history and cookies vanish. But we had numerous government and military customers and so Hughes came to us on behalf of some three-letter agencies that were getting clobbered by these things and begged us to help them clean up the mess resulting from the proliferation of trojans since the antivirus industry informed then that these "trojans" were none of their concern because they didn't infect files and thus were not viruses worthy of their attention.
They were merely "potentially unwanted programs" (or PUP). No need to "disinfect" them even though they were memory resident and couldn't be deleted.
And so, we were recruited (Shanghai'd?) into adding a new product to our stable called "BOClean" as a result of the "Back Orifice" panic, and looking for a name that would fit with our other products, used "BO" for "Back Orifice" in its name, assured that the AV's would surely pick up the slack soon and the need for this more complex product would be incredibly brief and useful only for cleaning up the three or four trojans in widespread distribution at the time. We would have certainly selected a catchier, and less confusing name had we only known what the next ten years were going to be like.
In early 1999, we had to release a "SockLock" product because of a file infector called "HAPPY99/SKA" and several other winsock file infectors that the AV's didn't cover for months after their appearances, likely because they couldn't figure out how to do so at the time because files that were in use by the Windows kernel could not be cleaned or deleted. It took months for the AV's to clean a simple file infector. Imagine that?
This was a business that we never wanted to be in to begin with because of the tremendous resources required for a small company like ours and dealing with these affected our ability to focus on our primary cleaning products. But "mom and pop" were doing a better job at it than the antivirus industry. And throughout the remainder of PSC's history and BOClean, it remained only two people doing battle as compared to a well-financed industry!
Two other small operations also saw the need in the earliest days and responded since the antivirus industry didn't, Wayne Langlois, Gavin Coe and Jason Annice of "Diamond CS" who made a product called "TDS: Trojan Defence Suite" and Daniel Otis-Vigil and his brother of "Moosoft" (another play on Cult of the Dead Cow) who made a product called "The Cleaner."
We all worked with one another collectively though we were "competitors" because the mission was too important and there were just so many trojans to handle. All of us "original players" fell by the wayside as fast-buck artists came along thinking that they were going to be the next McAfee. When the AV's finally started to pay attention, we were all sidelined.
SKA and other trojans however finally got the antivirus industry's attention and was a major turning point. Suddenly as trojans replaced and almost caused file infectors to vanish as malware authors seized all the source code and made vanity variants of their own, the antivirus industry merely applied the same old technology of taking samples, creating a signature and scanning files in hope of detecting trojans.
Success for the AV's however was elusive because there were now "crypters", "packers" (note the date that they noticed!) and "trojan toolkits" which allowed any "script kiddie" to make each output unique and different from other copies of the same trojan.
And so, the antivirus industry focused on detecting and deleting the trojan toolkits (yep, more PUP) and had little success detecting the trojan servers themselves. Mind you, all of this stuff was around since Back Orifice was first released! Packers and crypters were nothing new, and by the simple act of using just a packer without even encrypting the file, a virus sample no longer matched the signature, and thus was no longer detected even if known to the AV.
No worries, said the AV's when Back Orifice 2000 was released the year after the original Back Orifice. They believed they had this one knocked like all the others. Alas, BO2K came with crypters and packers and some neat little memory offset tricks built right in. Not to mention all of the external tools available that can further obfuscate the server it builds. Remarkably systems to this very day still get infected with BO2K with little warning other than from their router logs after 12 years of industry fail.
By 2002, even Microsoft (which was gleefully dismissing any problem with this all along) was trying to get the attention of the antivirus industry to no avail and had to publicly shame them. There were earlier pleas, but this is one of the few surviving "live" pages. It's almost hypocritical of Microsoft to have published this given that the reason why all this malware works at all is the result of Microsoft's incredibly poor design of an operating system, but I'll get into that in another part of this series dedicated to how and why.
By 2005, even the media began to notice the "trojan problem" as it became known as "spyware" prompting me to write an even more long-winded tome than this one on our own site. And just last year, the fail ramped up even further in tests by NSS. Same story the previous year. So why has antivirus been in a constant state of failure going back for almost all of its nearly 30 year run?
To explain that, over the past 30 years, the antivirus industry was used to doing things only one way. And the creative people who came up with unique and sometimes eccentric ways of thinking outside anyone's box in dealing with new problems all got shunted aside in favor of people who did things "by the book."
In the early days, a "malware analyst" took the sample, and ran it on a test machine with a few simple tools. A disk sector editor and perhaps a memory viewer if the virus became "memory resident." Early AV's only wanted a file signature for their engine and a database to hold the definitions.
So the behavior would be observed by a human, and since there were not all that many viruses about, they'd write up the signature, come up with a name and stuff it into the database. Perhaps they'd even write up a report for the PR department about it.
If the virus was a memory resident type, then they would have to set a flag for the engine to find out what process belonged to that executable file and kill that too. If they could. If not, they'd force a reboot to delete the file and hopefully it wouldn't return.
Then they'd take their new database and set it loose within the company to see if any false positives resulted on numerous computers. If not, then the database was released as a signature update every week or so.
This simple methodology worked OK for a few years and the number of analysts required weren't all that many. Very economical for the suits and the analyst had time to play with more interesting ones if they were complex and written about in the press. Of course, more report writing for the PR department as well.
If something was a variant of an earlier one, no problem. Do a simple file signature for that one, add a new name to the database and the number of "we detect" ramped up and all was happy. No need at all to look back to previous viruses and establish some sort of pattern to deal with future variants, that would be counterproductive to the mission and the body count for the marketing department.
Fast forward many years, and that's STILL how it works. Each variant gets its own signature and the body count goes up again. Many AV's really cheat and use an automation blender to simply grab an MD5 or an SHA1 from a submitted sample and they're not even looked at by an analyst. Any variation then by packing or crypting is yet another new signature. WHEN they get around to it. Perhaps.
By 2005, "file infectors" had all but disappeared from the scene. It was all worms, trojans, backdoors and things that the AV industry was never prepared for, particularly in their engine designs. Sure they came up with "unpackers" for some of the more popular packers in order to convert the garbage of a packed file into something parsable by their engines, but they were still operating in "read file" mode.
Our BOClean product was designed to read malware solely in memory once it had been invoked and so packers, crypters and all the modern foolishness never stood in our way. We could see the actual executable inside CPU space before it got going.
The only problem for BOClean however was that since the vast majority of malware was nothing more than repacked variants of the same old stuff, our "unique count" remained low and so we were constantly criticized for "not detecting much" based on the antivirus PR "body count" method.
Meanwhile, the AV's struggled with "heuristics" which resulted in massive false positives on legitimate system files, deleting operating systems as a result and generating extreme havoc when updates were applied to desktops. Some marketed their "heuristics" as "behavioral analysis" but it was the same old cow.
The concept of "heuristics" is its own bad idea. You emulate the execution of a program and watch to see if it wants to connect to the internet, maybe drop a file or two, maybe delete something and make backup copies of itself. Must be malware. Nope ... it WAS an Adobe update, a critical system file or something else legit and you just deleted it and set off the fire alarm after trashing the machine. This experience with the industry's "whack-a-mole" ended up making more people feel like the gopher than the rube on the midway clutching the hammer.
Heuristics may work if something's unique enough to make a signature from, but success is rare since many legitimate programs reuse code that originated in malware that got published for kids to steal and use when they're writing code for their employer.
BOClean's method was to examine the code about to execute and zero in on specific bits of malware that would positively identify the perpetrator since almost every malcoder (even the criminals) usually sign their work in some manner. That was the key to BOClean's success against crimeware like the ZLOB fake codecs and the Fake AV's.
There was ALWAYS a reliable signature by their author in there to be found! Thus, we did very well with "zero day" because we had seen it before. But of course, that flew in the face of "malware analysis" at the big companies. Too much work, too much time spent.
By 2005, a serious rampup of nasties occurred. Instead of updating once a week or perhaps once a day, malware started coming fast and furious requiring frequent updates. More analysts were required even though it remained myself and Nancy at BOClean.
The AV's responded to the rampup by doing what they did to their founders - malware analysts were put on the street and they offshored to India, Ukraine, Philippines and China. China? That's where most of the malware was COMING from! (I'll get back to this) Sounded like a good idea to the suits though - if that was where the malware was coming from, then they had "local talent" and they could pay the same to ten or twenty analysts as they used to pay for just one.
By 2007, Nancy and I were simply unable to keep up, we were losing customers and were headed towards continuing being impossible. We were not in any position to do what the AV's did next, despite their cheap labor. "Automation" was the next big thing. Take in samples from all those machines out there and from public "test this file" sites and just SHA1 or MD5 hash them and dump those into a huge database that the "analysts" were supposed to rummage through whenever they had time (which they didn't).
With only a hash as any indication of what the submitted files were, they stabbed at them the best they could. Meanwhile, they used only the hashes as signatures for the AV engine, including "indeterminates" which meant that if it was submitted, it was malware as long as automation tagged it as "suspicious!" In my very best Mr. Rogers voice, "can we say FP boys and girls?"
I was unaware that this was how it was done until we were acquired by COMODO and I got tossed into that bucket there and got to work with third world veterans of the antivirus business who had worked for various other major names before coming to COMODO. First engineering question I was asked was "how do we detect a packer in a file?" Ummm ... entropy? Apparently the AV industry had just started working on that. I won't disclose whose competitor's AV engine they stole, but it's out there for googling.
Not a chance of malware submissions getting any detailed analysis unless someone knew what the MD5 for the file was (yeah, copy and paste THAT) or it was properly "cursed" by enough other AV's who gave it one of dozens of different names that rang a bell. I kid you not! And you had to search manually through the definitions database knowing what you were looking for to have a look assuming they bothered to even keep a copy of the sample in the first place! So analysis was rarely done unless it was reported back as a false positive problem and the customer remembered what it was detected as.
And there's yet another fail mode. Someone reports a file as an FP. How is it handled? Why it's REMOVED from the signatures and tossed back into the "undetecteds" in the database with hope that someone who is bored might have a look at it. What are the chances? And so the criminals learned quickly that the best way of keeping their malware undetected if they couldn't just go to Jotti or VirusTotal and keep modifying it until nobody detected it or even better, was to report it as an FP to everyone. Problem solved for criminals!
With over 70,000 samples a day coming in the door, analysts just can't be bothered. And with the public completely used to the idea of false positives being commonplace, if only a handful of AV's detected their submission to Jotti, then it must be their AV doing an FP. And there's your antivirus. At work.
I won't even bother to go into what gets hired over there in those countries. At COMODO, there were plenty of people in the AV labs who were dodgy and a few very good people who just walked away. I was told many tales of analysts who "lost things" as far as detected malware went, and it's entirely possible that with these overseas analysts that some are in cahoots with the criminals.
I guess it's simply a matter of whether or not you "trust" these offshore "labs" that everyone's using to control their costs. And my paranoia about this was not served well at COMODO when the BOClean code that I gave them was sent off to Chinese coders who would not EVER let me audit my own code after they'd changed it.
And in the end, not one single innovation in BOClean made it into COMODO's product other than the most rudimentary part of reading memory, and perhaps our ability to unhook some things from the kernel. BOClean wasn't what it was because it could parse system memory properly instead of a file, it was what it was because of the method of creating the signatures for it. Getting into the mind of the perpetrators and seeking out what was unique about the author and using THAT for the signature.
We knew the bad guys would write more code and by focusing on HOW they coded, we were able to successfully detect future zero days. COMODO just went and treated it like any other AV file engine and never grasped the method behind our code. BOClean's prior reputation was utterly destroyed in the process because it never was built on the traditional AV methodology in the first place and our former customers spotted the difference immediately. In addition to stopping the malware, BOClean also cleaned up ALL of the mess it created in the registry, file system and removed all entrails. FULL cleanup of a mess is something that just doesn't happen anymore.
But if nothing else, learning how to do AV "properly" at COMODO was a rude awakening indeed. About the only advice that COMODO took from me when I came on board was the concept of whitelisting as about the only viable way of knocking down malware since blacklisting has always been a complete failure. But they didn't get that right either, because graylisting (we don't know what this is and we won't let you run it until we do) was never part of the process and whitelisting properly was too big a challenge. The other AV's are now following that same course the same way COMODO began to do.
"Reputation-based" protection doesn't work either in the real world since it depends on people voting intelligently and the ballot box not being trolled by ne'er-do-wells. When 4chan sent Justin Bieber to North Korea and talent got whacked on "American Idol", there's a perfect example of how "public voting" works in real life situations. I wouldn't trust the denizens of Tumblr or the Lulzboat either to decide what's "safe". And at COMODO, I learned quickly how "trusted vendors" lists aren't.
"Crowd sourcing" technical decisions isn't a bright idea when you're handing the decision keys to idiots. And THAT is the proclaimed future for the direction of antivirus tomorrow. While reputation has made some AV's appear to be better because they performed so poorly in the past, it also hasn't been gamed yet and there's the reason why it may sound good now, but then all of the other methods have also been worked around quickly.
To add to the fun of "reputation-based" and "trusted vendors" getting an automatic pass in white listing, Adobe (Flash, PDF, etc) and Oracle (Java) are most certainly "trusted vendors" right? The majority of malware today exploits these "trusted vendors" and their "reputation" remains impeccable. Modern exploits don't even need to drop a file for the AV's to sniff anymore. They do their thing in memory INSIDE the "reputable" products without any clue at all. Just more reasons why I threw in the towel on Windows years ago and went yet another way and am doing what I'm doing now.
In the next article, I will explain how excruciatingly bad choices and designs in the operating systems themselves set the stage for the security nightmares we've suffered all these years (and will continue to do so) and then in the article following that, I'll bring all of the pieces together to show how adding even more "scattershot" approaches over time to the problem are making things even worse. Once we all see the magnitude of the situation, we'll discuss what can be done about all this.
(to be continued)
About the author: Kevin McAleavey is the architect of the KNOS secure operating system ( http://www.knosproject.com ) and has been in antimalware research and security product development since 1996.