2012 Benchmark: A Step-by-Step Guide for Choosing The Best Scanner
A simple and comprehensive how-to guide for choosing the best scanner for *you*, based on the conclusions of the 2012 web application vulnerability scanner benchmark (A comparison of 10 crucial aspects of 60 web application vulnerability scanners).
There's plenty of guides out there for evaluating web application vulnerability scanners, including the Web Application Scanner Evaluation Criteria (WASC), dozens of posts in blogs and newsletters, and even a couple of methodologies published by scanner vendors... Not to mention the occasional advice in forums and newsgroups.
So why would anyone bother to write another one? Fair question indeed.
Although many of the methodologies out there contain great ideas, there hasn't been any independent methodology that I'm aware of in a while, and as far as methodologies published by web application scanner vendors, well, even if the content within is accurate, it's irrational for any vendor, or even an author affiliated with a vendor, to include aspects in the evaluation methodology that might present the vendor in a bad light. After all, how can a participant in a contest behave like a fair judge?
So the world isn't perfect... no point in stating the obvious... so why should this post make a difference?
Well, I have been conducting various massive-scale researches in the field of automated security vulnerability scanners for the past 4 years, have been using them for over 8 years, have worked for an automated security product vendor in the past, and even designed scanning algorithm and attack trees, in both the past and the present.
I'm not affiliated or funded by any vendor, and invested countless hours of my spare time to find a solution to this problem.
Furthermore, now that I've just published my new research - I just happen to have exactly the information you need to make most of the decisions, saving you countless hours of product evaluations, research and frustration, allowing you to narrow the list of options to the relevant products & vendors in minutes, instead of months, and enabling to invest your time or money where you will benefit the most:
- ("Top 10 Aspects: The Web Application Vulnerability Scanners Benchmark, 2012 - An Accuracy, Coverage, Versatility, Adaptability, Feature and Price Comparison of 60 Commercial & Open Source Black Box Web Application Vulnerability Scanners", published on http://sectooladdict.blogspot.co.il/, and http://sectoolmarket.com/)
Since In my experience, the common consumer almost never finds the time to perform a full scale evaluation to a couple of products, not to mention periodic evaluation, I believe this subject to be of importance.
It directly affects the amount and type of exposures detected by organizations that use an a vulnerability assessment solution internally, and can make a great difference for anyone that performs security assessments.
And with that being said, I hope the following post will make it easier for you to decide which product is best for you, for a specific task, or as a whole.
Note: The various tables and charts linked from the article were updated for the vast majority of tools at July 2012.
Phase 1 - Verify the Scanner's Support for the Target Technology
In order for the scanner to produce any results, it must be able to scan the protocol, application entry points and the inputs of the tested application, in addition to being able to handle various "scan barriers" that might interfere with the scanning process, such as authentication requirements, challenge-response mechanisms and session-specific token requirements.
In order to simplify this description, let's assume, just for a second, that you have a dedicated employee that is familiar with every configuration entry in the scanner, knows exactly how to optimize it, and has an endless amount of time to configure and "train" the scanner to scan each and every one of your applications.
Even with those favorable conditions, the scanner will not provide you the results you expect, and in many cases, won't provide results at all, unless it meets the following criteria:
(i) Support the Application's Method of Receiving / Delivering Input (Input Vector):
In general, in order to scan an application, a generic application scanner uses a variety of techniques, but the vast majority of these techniques relies on injecting payloads into key sections in the communication protocol, which are the sections that are used to deliver input to the server, and are the basis for the whole client-server dynamic interaction.
In order to scan these "input delivery vectors", the scanner must support the type of protocol and the type of input delivery method used by the application.
Since applications might use a variety of input delivery methods (ranging from simple querystring parameters to JSON arrays and binary serialized objects), the scanner must support the vast majority of the protocols/input delivery methods used by the application, and ideally, support them all, otherwise, most of the scanner's attack plugins will not be executed in front of the application (at least not properly).
For example, if the application uses a simple GUI based on HTML links and forms, and has no specific scan barriers, it can be tested using most of the scanners in the market, since nearly every scanner supports HTTP querystring (GET) and HTTP body (POST) parameters.
However, using a scanner that supports HTTP querystring, HTTP body and JSON to scan an application that sends and receives objects using the AMF format is utmost useless.
It might not be cost effective to buy the most expensive scanner just to support every possible technology, but in the same breath, there's no point buying a scanner that can't scan most of the applications you're testing. The solution you're using / purchasing should be able to scan most of the input delivery methods used in the applications you're testing (or the suite of products you are about to test).
In order to verify whether or not the scanner you are evaluating supports the input vectors you need, either read the product manual, or use the following cross-product comparison table (notice the glossary for each input vector acronym is found at the bottom of the page):
- http://sectoolmarket.com/input-vector-support-unified-list.html (Updated for the vast majority of tools at July 2012)
Pay attention to the fact that the versatility is not as important as simply meeting the perquisite - being able to scan the application specific input vectors.
(ii) Support the Application's Scan Barriers - Challenge-Response Mechanisms and Session Specific Token Requirements:
Due to the ever increasing amount of attacks that originate from 3rd party web sites (Clickjacking, CSRF, etc) and the constant threat of internet bots, some applications implement mechanisms meant to prevent or mitigate these risks. Although these mechanisms are a significant security enhancement, as a side effect, they also dramatically reduce the efficiency of scanners that were not built to handle them, even those used for internal security assessments.
Common scan barriers include HTTP header requirements, Anti-CSRF mechanisms (usually enforced on the entire list of post-login entry points, or at the very least, on action performing entry points) and CAPTCHA (usually enforced on specific entry points, and not on large groups of URLs).
As a result, using a scanner against an application with "scan barriers" that the scanner cannot handle, will prevent the scanner from detecting most of the vulnerabilities in entry points that are affected by the scan barrier.
Thus, when selecting a scanner for a specific task or a long term usage, verify that it can handle scan barriers that might prevent it from scanning large groups of application entry points, or use following cross-product comparison table (notice the glossary for each scan barrier/coverage feature is found at the bottom of the page):
(iii) Direct / Indirect Support for the Application's Authentication Method:
In case the application has a section that requires authentication, the scanner must support the authentication method required by the application, or at the very least - support a feature that enables it to rely on the services of an external entity that does.
Authentication requirements vary across different implementations, architectures and technologies, and it's very rare for a single tool to support them all; however, any tool with an upstream proxy feature (or in other words - any tool that can forward it's communication through a controlled intermediate layer) can rely on it to perform most of the authentication types instead.
For example, by defining burpsuite or fiddler as an upstream proxy, scanners without the relevant authentication method can in-fact, rely on these products to perform the authentication process instead (assuming the authentication method is supported in the intermediate layer).
To verify whether or not the scanner you are evaluating supports the authentication method, either read the product's manual, or use following cross-product comparison table:
*Phase 1 Summary* - unlike the rest of the sections in this manual, the scan barrier and input vector support are a perquisites for the scanner - an initial basis on which you should start filtering scanners that can't scan large sections of the content you should test. The authentication requirements, however, might be a little less restricting, as long as the scanner supports forwarding the communication to external entities.
Phase 2 - Assess the Scanner Audit Features, Accuracy and Coverage
There's an ancient debate as for which scanner aspects are more important (ok, maybe not ancient, but it still exists):
Better application coverage features, more vulnerability detection features or higher accuracy?
While coverage features enhance to the scanner's ability to increase the size of the attack surface by detecting additional entry-points and parameters to scan, and generally making the scanner better in point-and-shoot scans, vulnerability detection features enable the scanner to detect additional types of common & exotic security exposures, and do it with a greater chance for success and with a lesser chance for false detection.
This debate also exists internally within the feature groups - which is more important? support for detecting additional types of vulnerabilities (audit features) or enhanced accuracy in detecting specific types of vulnerabilities? A better crawling mechanism or a better default-value submission feature?
Although there isn't exactly an accurate formula that can settle this debate, there are certain guidelines you can use to decide what's more important for you.
(i) Coverage Features - Crawling, Enumeration, Value Submission and Impersonation
Coverage can mean a lot of things, and is composed of various features, such as detecting alternative presentation layers by mimicking different user-agents, performing an effective automated crawling process on the various application entry points, detecting different methods to activate each entry point and even using the appropriate value when submitting a form.
These features (and others) enable the scanner to detect entry points that it can scan, while automatically crawling the application, and thus, increase the attack surface of the scanner - how much application interfaces, entry points, parameters, scenarios and behaviors will the scanner identify, emulate and scan.
There is however, another method to (somewhat) increase the scanner's coverage: manual training - a human that "trains" the scanner to recognize the application structure and valid values by manually crawling the tested application using the appropriate scanner module.
Manually training the scanner provides an alternative method to increasing the scanner coverage, by manually "training" it to be able to scan specific entry points, scenarios and behaviors, and thus, a human can in most cases, enhance the coverage of a scanner (up to a certain level), even if was somehow limited.
In general, you can use the following guidelines to decide what's more important:
· If there is no features that support manual crawling/training... there no point reading the next few sections... the scanner must have good coverage features.
· The less experienced the user, the more significant it is to use a product with better automated coverage features.
· The larger the scanned application, the more crucial the automated coverage features become.
· The shorter the amount of time allocated to the test, the more important it is to use a product that can cover larger sections of the application on its own.
And vice versa:
· The more experienced the user, the more you will benefit from manually training scanner to recognize the application structure.
· The smaller the application is (and not just in term of URLs), the less you need coverage features, and the more you can benefit from additional tests and higher accuracy.
· The longer you have, the more time you can afford to waste on long scanner training sessions, extensive customized configuration and long execution times (due to a large amount of tests).
Although there isn't an overall test for coverage, there are a couple of tests for various elements that enhance the process, such as input vector extraction in automated crawling (WIVET):
(ii) Vulnerability Detection Features - Audit Plugins and Detection Accuracy
Each audit plugin (or group of audit plugins), is in fact, a set of tests that attempts to detect a specific vulnerability. Although there are generic (and even non-generic) plugins that can detect vulnerabilities without a specific adaptation, the detection of many vulnerabilities requires designated audit plugins.
However, the algorithms used to identify vulnerabilities may differ, and some algorithms may be superior to other in specific cases, and sometimes even in general, and thus, lead to the detection of additional instances of the given vulnerability.
As for what is more important - accuracy or a large array of audit features, well, there isn't a straight answer for that either, but in general, when scanning specific technologies, having better accuracy with the relevant features is more important than a wide array of irrelevant technologies, but that's not always the case when testing a variety of technologies.
For example, using LDAP injection and XPath Injection plugins on an application that only interacts with an oracle database is less important than using a really good SQL Injection plugin, while most SSJS injection plugins will do better than all three when scanning an application that interacts with MongoDB. That of course, is not necessarily the case when testing an application with a variety of unknown technologies, in which the variety of tests can increase the chances of detecting an exposure.
As a simple rule - verify that the product has plugins that can detect the exposures relevant for the technologies you use, and then verify that these plugins are as accurate as possible.
Remember - these are guidelines, not exact science - if you're not sure, consult with someone who can help, or test the products on your own systems.
In order to verify whether or not the scanner you are evaluating supports the audit features that are relevant for your tested technologies, either read the product manual, or use following cross-product comparison tables (notice the glossary for each audit feature is found at the bottom of the page):
The following links contain information about the accuracy of various scanner audit plugins (tested againt WAVSEP and ZAP-WAVE):
*Phase II - Summary* - A simple formula that should handle most cases: verify that the scanner has the relevant audit plugins for the technology used in the tested applications, make sure the scanner's coverage is as high as possible, and then verify that the audit plugins are as accurate as possible. If you have experienced employees, smaller applications and more time, the accuracy vs. coverage priority can change.
Phase III - Budget and ROI - Compare the Scanner Pricing and Features
This section, unlike the rest, is not relevant for free & open source products.
By this point, you should have already filtered some of the irrelevant scanners, and remained with a bunch of products that you should at the very least test... but not so fast... can you afford these products, does their price justify what you receive, and do you really need all the features that they have to offer?
The solution for two of the questions above is relatively simple - the price vs. value issue can be sorted using a simple price and feature comparison, which you can manually perform by accessing the various vendor pricing sheets, by contacting a sales person, or alternatively, by use the following cross-product comparison table:
The price should be compared to that of products with a similar list of features / accuracy / coverage, which met the perquisites (included in the link above).
As for what to do with the price and budget limit - that's a tough question, but for those of who still want to keep their options open, here's some tricks for making the best of your budget:
1) Many products are actually packages that include multiple solutions, which you might use to justify the price, or expand the budget at the expense of another product which is already implemented within the package (for example, a web application scanner, source code analysis tool and a web server scanner)
2) You don't have to buy several licenses of the same product - you can mix, and this mix can useful in a couple of ways - pricing, technology support, versatility, accuracy, etc.
3) In case you insist on buying several licenses of the same product, pay attention to the fact that many vendors offer discounts on bundles of several licenses - discounts that can be significant.
4) If no other choice remains (and some would say immediately), barter - nothing is set in concrete and the vendor might offer you a better deal.
*Phase III - Summary* - Make sure the product/s you intend to test are within the frame of your budget, verify that you get an adequate benefit, and move the next phase .
Phase IV - Testing the Remaining Scanners
It's best not to rely only on statistics, and test the products yourself... or at least the few products that remained after the previous elimination phases.
You can use designated evaluation environments such as WAVSEP, ZAP-WAVE and WIVET, sample vulnerable applications such as BodgeIt, Insecure Web App and DVWA, and in addition, real applications that are implemented in one of the technologies that the scanners should test in the future.
In either event, you should test the remaining products against these applications, and start wrapping all the results to something that can help you decide.
Phase V - Additional Considerations
There is a number of elements that were not covered in this guide, such as stability, user interface, simplicity, performance, product updates, service level agreements, and so on.
Note that stability is a key issue, and in future version of this guide, will probably justify a segment of its own.
Like the rest of the product's aspects, these aspects should be as good as possible, and ideally, these features, standards and perquisites should be verified in phase IV, when the products are evaluated.. but until some additional information will be published on these aspects, it's really up to you to evaluate the solutions, and get to the final decision.
*Phase IV & V - Summary* - Make sure the product/s actually manage to scan the technology you need to test, check it's performance and stability against other applications, and verify that the various other aspects that are important to you are met.
In general, when choosing the right scanner for the task, pay attention to the following guidelines:
1) The scanner should be able to scan the vast majority of the target applications (and thus support the main types of input delivery methods, be able to handle scan barriers such as authentication requirements/required parameters/anti-CSRF tokens - assuming they are used in the tested applications, and be able to detect relevant vulnerabilities in the crucial platforms). A great price, perfect accuracy and shiny graphics won't compensate for the inability to detect vulnerabilities, or even scan your applications.
2) The scanner should match the technical level of the employee that is going to use it (a complex configuration interface without a good automated crawling mechanism will require some level of expertise in order to "train" it to properly scan the target application, while a simple configuration interface with a good crawling mechanism can be used by... well, humans).
3) The scanner should be stable, support as many relevant vulnerability detection plugins as possible, be as accurate as possible, and support any additional perquisites.
4) Finally, after all these conditions are met, it's time for the major criteria - it should be within the limit of your budget.
Although many other features can be useful as well, and assuming the technology support features being a necessity, the barter should revolve around the cost (in time or money) vs. the scanner's vulnerability detection features / accuracy / coverage / stability.
But buying the right product does not necessarily mean that the employee using it will do it properly. Using scanners inadequately can lead to false negatives, false positives, unnecessary load, and in extreme cases even damage to production systems.
But that is a subject for another post...