In This Insight


Our customer really wanted us to create a lab in the Advanced Technology Center (or ATC) that would help them evaluate Fortinet's capabilities around virtual firewalls, specifically the FortiGate-VM04.  The uCPE being used in this evaluation where the FortiGate-VM04 VNF would operate was Cisco's Enterprise Network Compute System (or ENCS) Platform. 

Our testing consisted of several use cases that our customer was interested in evaluating for their solution in regards to Fortinet's virtual firewall capabilities

ATC Insight

We tested several scenarios as I mentioned in the Overview Section, and we will go into depth on several different testing scenarios that were important to our customer to obtain relevant results.  The higher-level testing scenarios were:

  • Bandwidth Traffic Testing
  • Firewall Efficacy Testing
  • Zero Day Detection Efficacy Testing
  • Redundancy and Disaster Recovery Testing
  • Upgrade Testing
  • Automated Deployment Testing

Bandwidth Traffic Testing

The maximum throughput of the IXIA Traffic Profile AppMix through the FortiGate VM-04 HA pair was 353.8 Mbps. The traffic flow used to achieve that rate was a combination of relevant directional Application Flow and transaction sizes defined in the initial scope of the Proof of Concept (or POC).   Additionally, we ran the IXIA Flow Index for Internet traffic alone and increased the flow rate. Due to the larger transaction sizes associated with this Flow Index comprised of HTTP, HTTPS, and DNS, we observed a maximum throughput of 403.6 Mbps. SR-IOV was enabled on the ISRv Gi2 interface. Data Plane Development Kit (or DPDK) was enabled globally on both Cisco ENCS chassis. The FortiGate VM-04 did not have DPDK enabled. 

Firewall Efficacy Testing

While sending strikes (only) from our IXIA toolset, our team observed that the FortiGate VM-04 firewall was able to block almost all the strikes that we threw at it via our Ixia BreakingPoint Malware Strike Lists.  The number of strikes that were allowed or skipped was minimal.  Additionally, when we sent strikes alongside the IXIA Traffic Profile AppMix (traffic that mimics customer traffic) the firewall was able to block the same majority of strikes which indicated that the FortiGate VM-04 would perform quite efficiently in a production customer scenario. All strikes were sent in the clear from IXIA without an SSL/TLS evasion profile enabled. SR-IOV was enabled on the ISRv Gi2 interface. DPDK was enabled globally on both ENCS chassis. The FortiGate VM-04 did not have DPDK enabled.

Zero-Day Detection Efficacy Testing

The Zero Day Detection capability is Fortinet's ability to detect new mutations of threats by performing additional analysis of test files traversing the FortiGate VM-04 VNF.  This type of testing showcased if threat analysis extends beyond standard signature-based methods local to the firewall to return a disposition on particular test files.

Our testing team leveraged a custom IXIA FTP flow to send the test file through the FortiGate VM-04. The test file properly unpacked and was detected as a threat in both the FortiGate cluster and the connected FortiAnalyzer. The FortiGate VM-04 successfully detected and blocked the nested infected file without dependency on off-box malware analysis.

Redundancy and Disaster Recovery Testing

There were a plethora of tests that were involved to cover the redundancy and disaster recovery scenarios in the mimicked customer environment.  

Here is a high-level list of the test cases that were necessary:

  • WAN failure on ENCS-A and how traffic is effected to ENCS-B
  • WAN restoration from ENCS-B back to ENCS-A and how traffic is effected
  • ISRv-A failure or out of service and how traffic is effected to ISRv-B
  • Restoration from ISRv-B back to ISRv-A and how traffic is effected
  • ISRv-A LAN virtual interface shutdown on ENCS-A
  • Restoration of ISRv-A LAN virtual interface and how traffic is effected
  • FortiGate VM-04-Pri failure or taken out of service and how traffic is effected to FortiGate VM-04-Sec
  • Restoration from FortiGate VM-04-Sec back to FortiGate VM-04-Pri and how traffic is effected
  • Physical shutdown or disconnect of cables to ENCS-A LAN interfaces
  • Restoration or connect cables to ENCS-A LAN interfaces
  • Shutdown or disconnect the Interconnect Link between ENCS-A and ENCS-B
  • Restoration of Interconnect Link between ENCS-A and ENCS-B
  • Power off ENCS-A
  • Restore Power to ENCS-A

The good news is that all of these tests were successful in terms of redundancy and disaster recovery.  The ATC Lab Services team leveraged a specialized app mix of 1000 UDP packets per second for these tests in order to show any granular loss, if any, with what we consider sensitive traffic flow (UDP packets).  In most of the test cases above normal UDP packet loss was encountered and expected during failover and restoration.  

Additional observations in traffic behavior were around "Split Brain" scenarios.  For example, when the Interconnect Link between ENCS-A and ENCS-B was disconnected, the team observed an HSRP "Split Brain" condition in which both ISRv hosts went active for the associated HSRP groups. No loss was observed as a result of the "Split Brain" condition.  When restored, no loss was observed and ISRv-A became the primary for all groups again. 

Upgrade Testing

The upgrade testing performed was designed to show the customer what happens when an HA pair of FortiGate VM-04s VNFs are upgraded.  We ran an IXIA 1000 UDP packet per second traffic mix during the upgrade process that traversed the firewall pair just to understand traffic impact.

We followed the proper upgrade path and procedure via Fortinet's best practice.  The upgrade process successfully completed and the FortiManager task log below illustrates that the cluster upgrade procedure completed approximately in 5 minutes.  The same procedure can be used to initiate concurrent upgrades on all or some of the FortiGate models managed by a FortiManager.  Additionally, upgrades can be easily incorporated into an automation framework using the FortiManager API.

Automated Deployment Testing

The automated deployment testing was important for the customer because it allowed them to understand if Cisco's ENCS Platform API's could be leveraged with Fortinet's FortiManager to deploy the FortiGate VM-04 onto a Cisco ENCS Platform.  The customer believed that this use case could be essential in cutting down the time and effort for Cisco ENCS and virtual firewall deployments.  We successfully demonstrated the automated deployment of a fully activated FortiGate VM-04 to the Cisco ENCS platform and confirmed the connectivity to FortiManager after deployment.

Final Thoughts On The Solution

Our customer was happy with the expansive testing we did around the FortiGate VM-04 running on Cisco's ENCS platform.  They were able to truly understand the true capabilities of the FortiGate VM-04 and how it could potentially be integrated into their integrated solution for their branch rollout strategy.  Here are some additional thoughts after the testing:

  • The redundancy and disaster recovery testing helped our customer understand exactly how chained NFV components will behave upon the loss of various resources. Fortinet met all of the normal requirements around HA and redundancy, and the customer was able to get comfortable with what truly happens in disaster recovery and production traffic failover scenarios.
  • Fortinet performed very well in the bandwidth tests.  The bandwidth traffic testing allowed our customer to truly understand the firewall's throughput capabilities with "everything turned on", like the IPS profile which is running to protect from malware and zero-day attacks.  They could gauge current production bandwidth traffic and how introducing the FortiGate VM-04 would fit into the overall design for bandwidth requirements.
  • Fortinet performed very well around the firewall efficacy testing and zero-day detection testing that was put forth in the Proof of Concept.  Our customer was able to truly get a sense of how well malware is detected and blocked and how additional threat analysis via FortiAnalyzer is used in zero-day attacks.
  • Finally, Fortinet scored high marks with the customer around automated deployment capability using the Cisco ENCS platform APIs with FortiManager.  As previously mentioned, this feature was important to the overall deployment and rollout strategy of our customer's initiative.  They believe it will be an efficiency gain that will save a ton of time and ultimately fit nicely into their standardized deployment model.


Our customer at a high level wanted to gain visibility into the overall functionality of the Cisco ENCS appliance that will host both and ISRv router VNF and a firewall VNF.  At a deeper level, they specifically wanted to understand how the FortiGate VM-04 Virtual firewall would function and perform under production like circumstances.  Our testing in the Advanced Technology Center (or ATC) around Fortinet's Virtual firewall would help our customer save time and money by accelerating the tests and observations with this Proof of Concept (or POC).

Testing Components

Devices Under Test
VNF Software Versions

We used the current Cisco NFVIS release version 3.12.2-FC. Release notes can be found here. The customer selected the OS versions for the VNFs per direction from Cisco and Fortinet in regards to the solution. See the above table for OEM-recommended software versions at the time of this writing (August 2020).

VNF Features Enabled
  • Cisco ISRv - Flexible Neflow, NBAR, NAT
  • FortiGate VM-04 - Anti-virus, App Control, IDS/IPS, URL Filtering, local logging, DNS inspection
VNF Resource Requirements

Below are the resource allocations to run each VNF.

  • Cisco ISRv - datasheet here
    • vCPU: 4, RAM: 4GB, Disk: 8GB
  • FortiGate VM-04 – datasheet here
    • vCPU: 4, RAM: 9GB, Disk: 60GB
Supporting Infrastructure and Tools
IXIA Application Profiles Used In Testing

The sample target distribution below that was used to mimic customer traffic production flows and model the IXIA application mix.  

Lab Diagram (Some physical and logical depiction)

This is a physical and logical depiction of some of the components and environment that was built specifically for this Proof of Concept (or POC) in the ATC for this effort.  You can see the positioning of the Cisco ENCS platform as well as some of the logical breakouts of the VNFs needed in this solution.

Technology Under Test

uCPE technology
Cisco Enterprise Network Compute System (or ENCS platform)

Fortinet Next-Generation Firewalls (Virtual) technology
FortiGate Virtual Appliances VM-04 for Virtual Network Function (or VNF)

Test Tools

For this Proof of Concept (or POC) we utilized the BreakingPoint tool from IXIA because of its capabilities around simulating real-world traffic up to layer 7, malware, and exploits.  Using this tool gives the ATC Lab Services team a very valid and industry-standard way to help our customers test out security solutions. If you would like to learn more about this product please visit Ixia's BreakingPoint page.


Figure A

(Figure A) A quick depiction of just a slice of reporting and metrics that we gather when executing POCs with our customers in the Advanced Technology Center (ATC).  We used the reporting and metrics out of the BreakingPoint tool quite extensively so our customer would have recorded metrics to evaluate the Palo Alto NGFW solution running on Cisco's Enterprise Network Compute System Platform (or ENCS Platform).