• Stefan Frei



Over the past eighteen months, NSS Labs has tested the security effectiveness of typical defense technologies, such as next generation firewall (NGFW), intrusion prevention systems (IPS), and endpoint protection (EPP – also referred to as antivirus/malware detection). A comparison of exploit block performances within and across these defense technologies has revealed a significant correlation of failures to detect exploits. Such detection failures present a serious challenge to the security industry as they allow an attacker to bypass several layers of defense using only a small set of exploits.

In multiple independent group tests carried out at the NSS testing facilities in Austin TX, NSS engineers determined the ability of 37 security products from 24 different security vendors to block exploits in real world attack scenarios. The 1,711 exploits used in these tests target 816 software products from 208 different software vendors, thereby covering 21% of all vulnerabilities published against these software products in the last 10 years.

None of the 37 tested security devices managed to detect all exploits and only 3% of 606 unique combinations of two security products managed to detect all exploits. Further, there is a large diversity in the security performance between individual security products or combinations of security products.

The analysis of these test results documents a significant correlation of failures to detect exploits between security products. The number of exploits found to bypass multiple security devices, as well as the number of security devices simultaneously bypassed by these exploits, is significantly higher than the common expectation, or than the predictions of those risk models that ignore the effects of this correlation.

This can lead security professionals to overestimate the combined security effect of deploying multiple different protection technologies. This significant correlation of detection failures indicates that deploying multiple products within a security category (such as IPS), or even multiple products across multiple categories (such as EPP behind IPS behind NGFW), does not always provide the “defense in depth” that we are led to believe exists from studying vendor claims for the efficacy of their products. This is because most vendors use the same sources of threat intelligence and the same vulnerability research feeds as each other, and this means that they will, more often than not, have the same deficiencies in their coverage.

Layered security, e.g. the deployment of layers of different security technologies, is beneficial when looking to secure the enterprise. However the choice of security devices to be combined is key to realizing substantial security gains and offsetting the increase in complexity, management, and cost.

Naïve risk modeling that ignores correlation, however, will result in a basic lack of understanding of the scope of exploits currently in common use that are able to bypass multiple security products.

The identification and analysis of exploits that escape detection by the majority of the security devices/products in a group test is globally relevant, as these exploits present a significant challenge to the security industry. This analysis shows that, while it is helpful to adopt a layered approach to security, the real key to effective protection against threats lies in an organization’s choice of protection technologies to be combined.