How to Compare Cybersecurity Products

A guide to testing antivirus software

The government ban on Kaspersky Lab products has spurred private companies, government-sponsored agencies, and the company’s 400 million users to re-evaluate their antivirus protection. The mass exodus brings up a valid issue in the crowded antivirus space: How can a company find the most effective product?

Self-reporting can’t be trusted: Vendors have recently come under fire for testing methods, missed malware, and other shortfalls. In the interest of full transparency, and to empower users to conduct their own testing, the cybersecurity experts at SparkCognition have put together a guide to testing AV products in a safe environment. Summarized here, we advocate for using third-party services to ensure results are unbiased. As always, our cybersecurity team is available to answer any questions.

Create a Safe Environment

To truly test the capabilities of an AV, the use of real malware is required, which can do real damage to a network if the testing goes awry. Thus, the easiest solution for a safe and effective environment is to use virtual machines.

A virtual machine (VM) is a fully operational computer system that runs from within another system. The most popular virtual machine software programs available are VMWare Workstation and Oracle Virtualbox, with each one being available on Windows, Macintosh, and Linux. Installing virtual machine software and an operating system similar to the actual systems used will provide the greatest test accuracy.

After setting up the virtual machine software and OS, the next step is to install an AV product to test (if installing multiple products, take a snapshot of the VM to revert to). Though the easiest solution for transferring a large number of files between a guest and a host is utilizing a shared folder, exercise caution with this option: It could mean that real malware will be accessible on the host and could infect the computer.

Types of Threats and Finding Samples

Although most people hear about all the malware that is lying in wait on the internet, it can be very difficult to find, particularly in the quantities needed for testing an AV. A comprehensive test will include as many files as possible.

We advocate for testing three types of malware: common malware (prevalent), malware that has been slightly changed to evade AV products (polymorphic), and malware unfamiliar to the system that would replicate a real threat (zero-day).

Prevalent malware: Several commercial companies have offerings for sale, the most reputable of which is Virustotal. Below are a few options for collecting malware samples for free.

Open Malware:

Polymorphic malware: One option available to test this type of malicious tactic is to use software (a packer) that will compress the original program and then insert it into another program that will decompress it at runtime. Hyperion and UPX packer can be used for this. Another option is to use a mutator, which alters different parts of the file to make it look different to AV.

Zero-day malware: The real test of an AV product is to see how it performs against never-before-seen threats. First-generation signature-based AV products become essentially ineffective here, while products powered by machine learning shine.

While zero-day malware, by definition, doesn’t exist in a test environment, several websites offer new malware that can provide a close approximation of AV response. The best source of close-to-zero-day software is the multiscanner website Virustotal, which can provide new malicious samples from the last 24 hours. Other sites such as Dasmalwerk and theZoo offer recent files for free as well.

However, since the files are new, their classification as malicious may be somewhat subjective to the AV product. The suggested query for Virustotal is all PE executables from the last 24 hours that are marked as malicious by 30% or more of vendors. To confirm the nature of the downloaded files, check back occasionally to see how the classifications of the files have changed over time.

Test the AV Products

Testing malware falls into two categories, active and passive. Passive testing means having an AV product scan a file, while active testing is executing the file and trusting the AV to prevent that execution. The best place to start is with a passive approach, since there is less chance of causing harm to the virtual machine and having to revert to a previous snapshot.

To start out the passive scanning, point the AV product at the folder of malicious files and run a folder scan. The product should provide the following information in its output:

  • Whether the file is a threat
  • Type of threat
  • Actions to protect the system (delete, block, quarantine)
  • Links or details about the threat and threat type

Next, perform a variety of actions that should engage the AV product. Authors of malicious software are constantly coming up with new execution methods and it is important for the AV product to be able to handle multiple avenues. For example:

  • Copy the file to a folder with a unique name
  • Use an email client to save the file
  • Right click and open the file
  • Put the file in the startup directory

The AV product should be able to alert the user and intercept execution. Run the active and passive tests for each set of malware (prevalent, polymorphic, and zero-day).

In addition, multiscanning websites like Virustotal are a great place to see how many AV products stack up against each other. It’s particularly good for comparing before/after performance on polymorphic malware samples.

In the connected, lightning-paced world of essential cybersecurity, where systems like electrical grids, manufacturing plants, traffic cameras, and other critical infrastructure are at stake, it’s better to be safe than sorry.

At SparkCognition, we encourage clients to compare security products for themselves. We particularly encourage them to test out the technologies that power our own security solution, DeepArmor, which has market-leading efficacy against zero-day and polymorphic threats.

SparkCognition is committed to compliance with applicable privacy laws, including GDPR, and we provide related assurances in our contractual commitments. Click here to review our Cookie & Privacy Policy.

Request a Partnership

Already a partner? Sign in.