Almost everybody runs AV, or at least they think they do. On Windows, over 90 percent of all people are running AV, and the number of people who think they are is even higher.

It's far more pervasive than any other end user technology, and is far more common in people's lives than the only other security technology with fairly widespread ubiquity - the firewall.

It amazes a lot of people that AV technology is so ubiquitous, because it is so widely reviled. Technical people will often claim that AV doesn't work, and that it causes stability problems. And almost everyone will claim that it slows your machine down.

First, let's look at what AV is and how the typical technology works, then we'll look at the huge pit of suck and why those problems are there.

What is Antivirus?

You might expect me to first define the word virus as a key to understanding AV.

But AV technologies typically go beyond viruses, also trying to detect worms, botnet software, trojans, and even spyware, adware, and attack tools - even though it can be a touchy subject as to whether most things in the last three categories are bad.

For instance, McAfee (and others) have always detected the program nmap as bad, since it can be used as an attack tool, even though many, many good guys use the tool (it simply helps map out which services are visible on a network - the name comes from "network map").

The logic is basically that the average AV user shouldn't have it on his machine, and having the AV software complain about it isn't going to stop the legitimate practitioner from using the tool.

There's merit to both sides here, and in many cases things get marked bad where the decision is quite clearly a grey area.

Anyway, all these terms are irrelevant for the moment. Suffice it to say, there is a ton of bad software that you probably wouldn't want on your machine.

The industry often refers to generic malicious software as malware, and we'll use that term. Spyware and adware are sometimes in a grey area where they aren't intentionally malicious and may not be called malware, but you should get the basic idea.

AV software is software that tries to identify malware and either prevent you from installing or running it in the first place, or remove it if it's already installed.

Antivirus software reviews

NEXT: How does Antivirus work?

How does Antivirus work?

There are two ways in which AV software tends to run: on-access scanning and on-demand scanning.

On-access scanning means that some program is about to run, or some other file is about to get used, and the AV checks it first to see if might be bad.

On-demand scanning means that files are being checked even if they're not being used. This generally occurs when you're doing a full system scan, which many AV products do when you boot the machine.

Typically, when a file is scanned and found to be bad, the user is notified and some appropriate action is taken, such as removing the file or (particularly in some enterprises) putting it in a quarantine area where it will not run but somebody can go look at it later.

The AV product running on the desktop usually doesn't intrinsically know too much about what's malware and what isn't.

That's the job of what the industry calls DATs (data files) or signature files. The AV product contains an engine that knows how to take a file that you want to scan, then query one or more of these signature files to figure out whether something might be bad. The signature files often also encode information about how to reverse infections, if necessary.

Typically, the AV product goes and grabs new signature files once a day (if you are continually online). Some products check twice a day or even hourly (and McAfee now has real-time updates).

AV engines are typically incredibly generic beasts. They're tuned to do pattern matching on arbitrary file types. They need to understand any file type that might potentially be problematic, which is a huge challenge to do well, especially if you're going to try to detect arbitrary data files, such as pictures that could attack your machine if loaded into the wrong photo viewer.

As an example of how generic AV engines tend to be, the McAfee AV engine basically implements multiple programming languages.

The signature files contain lots of little programs that the AV engine runs every time it looks at a file to determine whether or not that file is bad.

One programming language McAfee uses is tuned for quickly identifying patterns in binaries, and the other is tuned for complicated issues that the other language is too simple to deal with, such as repairs.

The first language is explicitly designed so that people writing an individual program in it won't accidentally cause your machine to hang. The other should be used sparingly and tested thoroughly before it is deployed to users.

There's usually an extensive operation behind any AV technology. The vendor needs to know enough to be able to say, "Hey, this file is malware," so it either has to have some secret sauce that allows it to determine malware using an algorithm, or it needs to look at individual programs and make a determination.

What typically happens is that vendors look at malware and try to spot patterns, then write signatures that are generic enough to catch as much malware as makes sense, without flagging something as bad that is clearly good.

Vendor employees then analyze files using some automation, but typically also involving manual effort. There has to be a workflow for tracking submissions and communicating with people who submit malware.

Once the vendor analyzes the files, it writes a signature, if appropriate. A signature might be generic enough to detect and repair a whole class of bad stuff, or it might just detect a single piece of malware, perhaps without repairing the actual infection.

Once it writes the signatures, the vendor typically needs to test them extensively to ensure that they're not going to cause problems when deployed.

Antivirus software reviews

NEXT: What is a False Positive?

What is a False Positive?

The biggest worry is that the signature will declare something as malware that isn't, in which case the signature is said to have given a false positive, or to have falsed.

The vendor doesn't like false positives, particularly because this stops people from running software they might want to run, potentially even deleting the software.

There have been several prominent false positives in the media, but probably the worst incident occurred in March 2006, when McAfee released a signature update that detected Microsoft Excel (among other things) as a virus and deleted it from machines.

Every major vendor has similar tales, and most vendors have more recent tales. In the case of McAfee, that incident really helped speed up the company's efforts for a dramatic improvement in its technology.

AV companies spend a lot of resources trying to prevent false positives. They tend to do extensive testing for signatures, including running them over massive databases of known good programs to make sure that none of those programs get flagged.

And in most companies, multiple people review each signature to make sure it won't have a negative impact. Yet false positives still happen, and pretty frequently (though usually on applications that aren't commonly used).

After testing, the AV company can publish the signatures. The publishing process can be complicated, but it's often the case that signatures are published at approximately the same time every day.

The desktop AV client tries to download those signatures when it thinks they might be published, and keeps trying fairly frequently if something is wrong (for instance, the computer might not be online or the signatures might be late in being published).

The AV industry has been working more or less this way for the past 20 years or so. Technologies haven't really improved very much, and they aren't as effective as they should be.

Antivirus software reviews

NEXT: What's the matter with Antivirus?

What's the matter with Antivirus?

The most obvious problem is scalability. Thousands of new pieces of malware come out every day. These days, most of them are showing up on a fairly small number of computers (say, dozens) before they are automatically "mutated" into slightly different programs that do the same thing. AV companies tend to have up to 100 people working full time on the problem, but each one of those people isn't likely to be able to handle more than a few pieces of malware a day.

Getting lots of people with the right skills to understand and detect malware is extremely challenging, particularly because of the vast technical expertise necessary to figure out what things the clever bad guys have done to thwart the security industry from doing its job.

Not having enough people to handle the flood of malware is a primary reason why detection rates for AV technology are so low (some people say as low as 30 percent in practice).

The industry tries to deal with this by writing signatures for individual pieces of malware, then trying to write signatures that are generic enough to detect as much malware as possible. But the bad guys have got pretty good at making this more difficult.

In practice, the better detection tends to come in waves, as the good guys at an AV company work hard to analyze trends over a large number of pieces of malware and write code to generically detect as much as it can.

Unfortunately, it doesn't come fast enough, and tends to leave people unprotected from individual threats for long periods of time.

There are a lot of other reasons why there is a long delay in detection (a big window of vulnerability). One reason is that AV vendors typically don't get to see enough bad stuff. They use a few methods to get their malware:

  • Many vendors swap malware with one another on a daily basis.
  • Many vendors have their own systems to crawl around the dark corners of the Internet looking for malware, and leave vulnerable systems around hoping people will break in and leave malware.
  • For the largest vendors, the biggest source of malware comes from the customer base. Customers send the vendor malware that the product didn't detect. These are often large corporate customers, not individuals, and in fact, you can bet the large companies get more attention to their problems than the little guys do.

This may all sound well and good, but while this strategy worked well a decade ago (when a single piece of malware tended to infect thousands of users), it doesn't work well now that there are tons more pieces of malware, infecting only a few dozen users at a time.

Antivirus software reviews

NEXT: Why Antivirus updates take too long

Another reason for there being a big window of vulnerability is that AV vendors don't want to mess up by having their technology give false positives, like the Microsoft Excel problem we discussed earlier.

But, as I've said, false positives are easy to create. Since AV signature files are code, and it's easy to add errors to code, it stands to reason that it is easy to end up with false positives. To combat that problem, AV companies have to do testing, which takes time.

When you combine that with delivering signatures on a daily basis, it's reasonable to expect a 24- to 48-hour lag time between when a piece of malware starts to spread and when an AV product detects it.

In reality, though, it is more like one to three weeks, on average. For example, in 2007, the Yankee Group published a report that discussed a virus known as the Hearse rootkit.

A company called Prevx found this rootkit and offered protection more or less immediately. It took McAfee 10 more days to get out a signature, and it took Symantec 13 days.

A lot of people think that a problem with AV technology is that it is a simple pattern-matching utility dressed up to look like something more powerful.

That might have been true in some cases at some points, but that's certainly not the case today. Since AV engines tend to have real programming languages in them, they can do arbitrary things.

Just because signature writers can do almost anything doesn't mean they do. Usually, brand-new approaches that could make a major difference aren't easy to build using the technology that already exists in AV products, and new technologies could easily have a significant impact to end users because there would inevitably be lots of bugs and performance issues.

So, as a result, AV companies tend to do four things in their signature files (and often some of this gets pushed into the "engine"):

  • Simple pattern matching for individual applications. This often amounts to, "If the file is identical to this piece of malware we saw before, then it is bad." There are some technical tricks to this, but it is effectively looking for an exact match without having to include the entire file for each piece of malware.
  • Simple pattern matching for a group of similar applications. This is known as "generic" detection, the hope being that the pattern uniquely identifies a class of malware, and not any legitimate software. With luck, it will help prevent AV companies from having to write lots of individual signatures for other pieces of malware in the family. In some cases, this detection type may hit only one file, but that's rarely the intent these days, since simple signatures do the same thing and don't risk accidentally flagging a good file as bad.
  • Look at outside factors (such as possibly suspect behaviours when the software runs) to guess whether it is malware. This is called heuristic detection, meaning that the code is taking a best guess, even though the AV company probably hasn't seen that particular program before. This is an area where the biggest AV companies tread very carefully, because if they mess up, they will end up upsetting customers with false positives.
  • Attempt to repair the system from an infection.

None of these types of signature content go far enough in addressing the underlying scale problem, though. Addressing the scaling issues would require a significantly different approach. Certainly, AV companies have been trying new things to do better, but it's a slow, experimental process.

Another fundamental problem with AV accuracy is that the bad guys can run AV products, too.

Let's say Evil Bill writes a bad piece of software. AV products might detect it out of the gate, but Bill will learn that right away just by running them. He can keep tweaking his malware until the programs stop complaining, then unleash it on the world and be guaranteed some time before anybody stops it (as we've seen, it can easily be weeks).

Making it worse, if Evil Bill does manage to get his malware on your machine, it is hard to recover from it. Bill's software will almost certainly disable your AV software, rendering it inoperable.

These problems seem insurmountable, but some technologies actually hold some promise for addressing most of these issues cost-effectively. The question is, why aren't we already using better technologies?

Most existing AV technologies are about 20 years old. They worked well enough for most of that time to achieve and maintain almost 100 percent market penetration.

So, in a sense, as long as the money keeps flowing in, there isn't a huge economic incentive for big companies that already invested a lot in building their technology to invest a lot more to reinvent it.

Instead, money for new development tends to go to new product lines that can potentially earn more money. For the big guys, it can easily make more economic sense to let someone else (say, a startup) go off and build better technology, and then to acquire it when it becomes necessary.

In the long run, there are technologies that show a lot of promise (such as collective intelligence). These technologies could make the AV company's job a lot easier, but it takes a big investment of time and money to get there. I see the industry starting to move in that direction, but it's still going to be a while before we get there.

See also: Why Antivirus software is slow

The Myths of Security by John Viega

Extracted from 'The Myths of Security' by John Viega

ISBN 978-0-596-52302-2

Copyright © 2009 John Viega. All rights reserved. Used with permission.

Published by O'Reilly Media Inc.

1005 Gravenstein Highway North, Sebastopol, CA 95472

PC Advisor security reviews:

Security software reviews

Antivirus software reviews

Internet security suite reviews