Fortinet Secure NetworkingATCFortinet Secure SD-WANFortinetATC InsightSoftware Defined WAN (SD-WAN)Network SecuritySASENetworkingSecurity Transformation
ATC Insight • • 18 minute read

Putting Fortinet Secure SD-WAN Through the Paces in the ATC

Many know Fortinet as one of the leaders in the security space but not many have looked at them through the lens of SD-WAN. Within the Advanced Technology Center (ATC) we recently took a look at Fortinet's Secure SD-WAN offering and put it through the paces with our Lab Services SD-WAN test plan template. Through our testing, we found that overall Fortinet is a serious contender in the SD-WAN space and should be in the discussions when considering an SD-WAN solution for your environment.

In this ATC Insight

 ATC Insight

The combination of SD-WAN, security, automation, reporting, and SD-Branch make Fortinet's Secure SD-WAN a compelling full-stack solution for branches and campuses.  In the following sections, we'll touch on each of these topics.

 SD-WAN

Fortinet's Secure SD-WAN is flexible in the deployment models that it supports.  You can build a traditional hub and spoke topology, partial mesh topology, or auto-discovery VPN (ADVPN) topology.  ADVPN is a Fortinet proprietary IPSEC technology that allows traditional hub and spokes to establish dynamic, on-demand, direct tunnels between each other without having to route traffic through the hubs.  This is like having a full mesh topology but without having to have the IPSEC tunnels up between all sites at all times.  There is one caveat to an ADVPN topology though; you are unable to load balance traffic across all available transports for spoke to spoke traffic.  In this situation, you can either specify the primary transport and backup transport for traffic or let the SLA determine the optimal transport.  All these topologies are built on two components; IPSEC VPNs and a routing protocol (no centralized controllers used here).  You can find an example of how to configure a hub and spoke deployment here or ADVPN deployment here.  In this lab environment, an ADVPN topology was configured.

The routing protocol support on the FortiGate is strong and feature-rich.  For the overlay, you can run either BGP, OSPF, or RIP.  In this lab environment iBGP was used in the overlay.  In the underlay and LAN zones, you can utilize BGP, OSPF, ISIS, or RIP.  In this lab environment eBGP was used in the underlay and eBGP and OSPF was used in the LAN zones.

Once the underlay and overlays are up you can begin to define SLAs and put SD-WAN rules in place to dictate the flow of traffic.  An SLA is used to monitor the health of the links.  Within the SLA you define the protocol (ICMP or HTTP) to be used to monitor, the remote server (IP or FQDN) to monitor, and the SLA targets (latency, jitter, and/or packet loss thresholds).

After defining the SLAs the next step is to create SD-WAN rules.  The SD-WAN rules have three components; source, destination, and outgoing interfaces.  The source is optional and can be configured with an address or user group (Active Directory based).  The destination can be configured with an address or internet service/application. Within outgoing interfaces you define a strategy (manual, best quality, lowest cost, or maximize bandwidth), interface preference, and measured SLA.  Additional information on SLAs and SD-WAN rules can be found here.  If SD-WAN rules are not defined an implicit rule is used.  In this lab environment, SD-WAN rules were implemented to have all traffic prefer the MPLS transport unless latency goes above 100ms.

 Security

At the heart of the FortiGate is a NGFW and everything else is built around this.  You cannot pass any traffic through the FortiGate without first defining the security policy to permit it.  This firewall has all the NGFW features that you would come to expect.  The FortiGate NGFW has been evaluated many times in the ATC and has always shown well.

 Automation

The FortiGate was developed with an automation mindset.  From the start you can bring your FortiGates online through the Zero Touch Provisioning (ZTP) process.  When the FortiGate comes online it will reach out to FortiCloud which has a database of your inventory.  FortiCloud will redirect the FortiGate to your instance of FortiManager.  Within FortiManager you can have templates staged for your devices.  When the FortiGate checks in you can have it automatically provisioned by FortiManager or it can go through an approval process and then be provisioned.  Templates within FortiManager can contain global settings that apply to all or groupings of your FortiGates.  The templates also support custom variables that can be unique per FortiGate.  Once devices are up and online you can continue to manage them via FortiManager to push out configuration changes or perform code upgrades.

The FortiGate has a full rest API built in and anything that can be configured within the web UI can be configured from the API.  There are also many Ansible modules that have been developed for the FortiGate.

 Reporting

Through FortiManager and FortiAnalyzer you can generate reports that will give you insights into the state of your environment.  By default, there are pre-canned reports that should satisfy most needs.  Custom reports can be created using the templates that are defined within FortiManager.  For those that need to get even more granular custom SQL queries can be run against the database to extract the exact data you are looking for.

 SD-Branch

The Fortinet SD-Branch is comprised of the FortiGate, FortiSwitch, FortiAP, and FortiNAC.  All of these can be managed from a single pane of glass and each has deep integrations with one another to create a comprehensive secure access solution for your branch and campus environments.  We did not get a chance to test this but hopefully in the future we can.  There are few vendors in this space that offer a full-stack solution to this degree.

 Fortinet Documentation

Something that often does not get mentioned but is critical is documentation and the Fortinet website has excellent documentation, you can find the documentation for the release we tested here.

 Test Plan/Test Case

 Management Test Cases

  1. Cloud Managed - Pass - Fortinet does not relay on controllers for control plane.  Fortinet does have FortiManager and FortiAnalyzer for management of the FortiGates.  Both FortiManager and FortiAnalyzer can be deployed in AWS.
  2. On-Premise Managed - Pass - Fortinet does not rely on controllers for control plane.  Fortinet does have FortiManager and FortiAnalyzer for management of the FortiGates.  These can either be hardware or virtual.
  3. Role Based Access Controls - Pass - Fortinet supports different priviledge levels for user accounts.  Fortinet supports multiple back-end systems for authentication; radius, LDAP, TACACs+, PKI, and SSO.  Fortinet also supports 2 factor authentication.  For multi-tenancy options Fortinet supports VDOMs (virtual domains) that split a single FortiGate / FortiManager into isolated virtual instances.

 Automation Test Cases

  1. RESTful API  - Pass - The FortiGate has a RESTful API and any item that can be configured in the web UI can be configured via the API.  The API documentation is not publicly available and you need to request permission to access.  There is also support for Ansible; there are many modules developed to automate the FortiGate. 
  2. Built-In Capabilities - Pass - Through FortiManager you are able to create templates that can be applied to devices.  These templates can contain global variables that are the same across all devices (i.e. ntp) or can contain variables that are unique per device (i.e. IP addresses).
  3. Configuration Rollback - Pass - The FortiGate supports configuration rollback when used in conjunction with FortiManager.  The FortiGate can be configured to follback to a previous configuration in the event that connectivity is lost with FortiManager.  Additional details can be found here.
  4. Zero Touch Provisioning - Pass - For ZTP when the FortiGate initially comes online it reaches out to FortiCloud which then redirects the FortiGate to the customers instance of FortiManager (based on serial numbers). The customer could already have the serial number associated in FortiManager with a template assigned so that the device is automatically provisioned. If the FortiGate serial is not pre-populated in FortiManager it will register in the queue notifying the customer that it is online and ready to be approved and configured.

 Routing Test Cases

  1. BGP IPv4 Support - Pass - The FortiGate supports IPv4 BGP in all scenarios; WAN, overlay, or LAN.  The IPv4 BGP protocol on the FortiGate is feature rich and should accommodate most if not all scenarios.
  2. BGP IPv6 Support - Pass - All features supported in IPv4 BGP are available in IPv6.
  3. OSPFv2 Support - Pass - The FortiGate supports OSPFv2.  The OSPF protocol on the FortiGate is feature rich and should accommodate most if not all scenarios.
  4. GRE/IPSEC Tunneling - Pass - The FortiGate supports building both GRE and IPSEC tunnels to 3rd party devices
  5. Multicast Routing Integration - Pass - The FortiGate supports multicast traffic traversing the SD-WAN fabric.
  6. Routing Integration with a Non SD-WAN Site - Pass - The FortiGate is able to integrate with legacy non SD-WAN sites.  The FortiGate is able to leak SD-WAN LAN routes into the underlay so that a non SD-WAN site is able to talk to an SD-WAN site by traversing the underlay.  If you prefer not to leak the routes into the underlay it is also possible to backhaul the non SD-WAN sites to a data center and have the non SD-WAN sites ride the overlay to get to SD-WAN sites.

 LAN Test Cases

  1. Router on a Stick - Pass - The FortiGate supports 802.1q tagging.  This can be configured on any interfaces; WAN or LAN.
  2. First Hop Redundancy Protocol - Pass - The FortiGate supports VRRP as the the first hop redundancy protocol.  For HA deployment an active/standby setup is recommended and that is what was tested.  In the active/standby HA deployment both FortiGates share an identical configuration.

 Security Test Cases

  1. Segmentation of User Traffic - Pass - The FortiGate supports multiple options for segmenting user traffic.  This can be achieved through VLANs, VRFs, VDOMS, or the firewall.
  2. Built-In Firewall - Pass - The FortiGate is first and foremost a NGFW and supports all the features that you would expect from one.  In order for any traffic to pass through the FortiGate firewall policy must be configured to permit the traffic.
  3. Service Chaining - Unsupported - Service chaining is unsupported on the FortiGate.
  4. ZScaler Integration - Pass - The FortiGate supports integrations with ZScaler.  Details on the integration can be found here.

 Application Test Cases

  1. Identification of Known Applications - Pass - The FortiGate was able to identify applications that are flowing through the topology.
  2. Identification of Custom Applications - Pass - On the FortiGate custom applications profiles can be created and SD-WAN rule sets can be configured around them.  You can define custom applications that IP based (subnet, IP range, FQDN, geography, dynamic) or you can define based on application signature.
  3. Quality of Service - Pass - The FortiGate supports the ability to classify traffic based on DSCP markings and apply a QoS policy based on that.  Additional details can be found here.
  4. Traffic Shaping and/or Policing - Pass - Within a QoS policy configured on the FortiGate you have the ability to police to the defined bandwidth.
  5. Brownout Detection and Re-Routing - Pass - On the FortiGate SD-WAN policy was configured to have MPLS be the preferred path.  The SD-WAN policy was configured with an SLA to monitor an IP based in the data center to ensure latency remained under 100ms.  Impairment was placed on the Branch 1 MPLS transport and traffic immediately redirected over to the Internet transport.  There was a brief hit to Ixia traffic but all flows recovered and quickly resumed their normal levels.  Impairment was removed from the MPLS transport and traffic migrated back to MPLS once it was deemed healthy.  There was no impact to traffic as it migrated back to the healthy transport.
  6. Blackout Detection and Re-Routing - Pass - On the FortiGate SD-WAN policy was configured to have MPLS be the preferred path.  In this test the MPLS transport was disconnected from the FortiGate.  During this outage the FortiGate immediately migrated traffic over to the Internet transport.  There was a brief hit to Ixia traffic but all flows recovered quickly and resumed their normal levels.  When the MPLS transport was restored traffic was migrated back to MPLS (per the SD-WAN policy).  During the migration traffic was not impacted.
  7. Path of Last Resort - Pass - The FortiGate supports integrations with LTE devices that can be treated as a path of last resort.  Fortinet has it's own solution, FortiExtender, that can be managed by the FortiGate.

 Resiliency Test Cases

  1. Isolate Date Center 1 WAN Connections - Pass - At data center 1 there are two transports; MPLS and Internet.  One at a time each of these transports was taken offline.  Upon total WAN failure of data center 1 all branches were still able to get to data center 1 via data center 2 over the DCI connecting the two.  There was a brief hit to Ixia traffic as routing tables updated to reflect the new path for data center 1 applications via the DCI.  When the WAN transports were restored at data center 1 the routing tables were updated and data center 1 became the preferred path again for it's applications.
  2. Hardware Failure of a Device in HA - Pass - The FortiGates at Branch 1 were setup in an HA pair (active/standby).  Policy was configured on the FortiGate to initiate failover in the event that a transport went down.  The MPLS transport was disconnected from the active FortiGate which immediately triggered a failover event to the standby unit.  We are running BGP in the underlay and overlay, during an HA failover BGP drops and reconnects for all neighbors.  There was a brief impact to Ixia during the failover but resumed to normal levels.  On the HA setup preempt was configured so if the previously active unit becomes healthy again HA will fallback.  The MPLS transport was reconnected to the previously active unit.  Once the unit was deemed healthy traffic migrated back to that unit.  There was no impact to traffic during this migration.
  3. Break HA Link Between SD-WAN Devices - Pass With Comments - The FortiGate will operate erratically if the HA link is lost between the two units, causing a split brain scenario.  The FortiGate supports multiple interfaces simultaneously acting as an HA link.  Under that configuration if would be difficult to create a split brain scenario.
  4. Loss of Control Connections - Out of Scope - FortiGates do not rely on controllers.

 Operations Test Cases

  1. Configuration Templates - Pass - Through FortiManager you are able to create templates that can be applied to devices.  These templates can contain global variables that are the same across all devices (i.e. ntp) or can contain variables that are unique per device and/or site.
  2. One-off Configurations - Pass - Through FortiManager you are able to create templates which can contain unique variables per device and/or site.
  3. SD-WAN Monitoring and Reporting Tools - Pass - Through FortiManager you can generate reports to show you the utilization and health of your SD-WAN sites.  There are stock reports that can be generated or created.  You can also get very granular with the reporting by creating custom SQL queries against the database to extract data to create custom reports.
  4. Third Party Integration with Netflow - Pass - The FortiGate supports netflow.
  5. Third Party Integration with SNMP - Pass - the FortiGate supports SNMPv2 and SNMPv3.
  6. Third Party Integration with Syslog - Pass - The FortiGate supports syslog.

 The Lab Environment

In this topology we have the following sites: 2 data centers, 2 SD-WAN sites, and 1 legacy site.
Data Center 1

At this data center we have two FortiGate Secure SD-WAN units setup in HA (active/standby).  In this type of HA deployment the units share an identical configuration (hence the same IP addresses used between the two on the diagram).  Both FortiGates are connected to the Internet and MPLS transports via the data center WAN edge routers.  The active FortiGate is eBGP peering with both data center WAN edge routers.  The active FortiGate forms overlay tunnels with the other SD-WAN units in the fabric via this WAN connectivity.

On the LAN side both FortiGate units have a routed interface to a LAN side router.  The active FortiGate unit is eBGP peering with the LAN side router.  Through BGP these peers are advertising prefixes to one another.  The active FortiGate is redistributing prefixes learned via the SD-WAN fabric and the LAN router is redistributing prefixes from the data center.  The LAN router has a DCI connection to the other data center LAN router so that in the event that WAN connectivity is lost at data center 1 the branches are able to reach resources in data center 1 via the DCI.

To support connectivity to the legacy site (Branch 3) the active FortiGate is redistributing the data center prefixes into the MPLS underlay and also permitting Branch 3 prefixes to be added to it's routing table and redistributing those to the LAN side router.  This allows Branch 3 to be able to reach resources in Data Center 1 through the FortiGates without having to cable around the FortiGates (i.e. Data Center LAN router to Data Center MPLS Edge router).

Data Center 2

This data center is setup similarly to data center 1 except it has a single FortiGate Secure SD-WAN unit and a single WAN edge router terminating both WAN service providers.

Branch 1 (SD-WAN Site)

At this branch we have two FortiGate Secure SD-WAN units setup in HA similar to data center 1.  Each FortiGate has a connection to each WAN service provider (Internet and MPLS).  In an active/standby deployment it is preferred to have both WANs connected to both units.  In an active/active deployment you could have WAN A terminate on FortiGate-01 and WAN B terminate on FortiGate-02 and both units would be able to use both WANs (as documented here).  The active FortiGate is eBGP peering with both WAN service providers.  The active FortiGate forms overlay with the other SD-WAN units in the fabric via this WAN connectivity.

On the LAN side the active FortiGate is the gateway for the LAN side networks and is redistributing these directly connected networks into the SD-WAN fabric.  This lab environment was primarily virtual and due to the nature of our setup we had to create separate interfaces per network on the FortiGate.  The FortiGate does support subinterfaces and would have been the configuration we had used if we did not have our limitations.

To support connectivity to the legacy site (Branch 3) the active FortiGate is redistributing it's LAN side prefixes into the MPLS underlay and also permitting Branch 3 prefixes to be added to the it's routing table.  This allows Branch 1 and Branch 3 to talk directly via the MPLS without having to go through the data centers.

Branch 2 (SD-WAN Site)

At this branch we have a single FortiGate Secure SD-WAN unit that is connected to each WAN service provider (Internet and MPLS).  The FortiGate is eBGP peering with both WAN service providers.  The FortiGate forms overlays with other SD-WAN units in the fabric via this WAN connectivity.

On the LAN side the FortiGate has a routed interface to a LAN router.  The FortiGate and the LAN router are OSPF peering.  Through OSPF these peers are advertising prefixes to one another.  The FortiGate is redistributing prefixes learned via the SD-WAN fabric and the LAN router is redistributing it's directly connected networks. 

To support connectivity to the legacy site (Branch 3) the FortiGate is setup similarly to Branch 1 so that Branch 2 can talk directly to Branch 3 via the MPLS without having to go through the data centers.

Branch 3 (Legacy Site)

At this branch we have a traditional router that is connected to the MPLS service provider.  This router is eBGP peering with the MPLS service provider and is redistributing it's directly connected LAN side networks so that that other sites connected to the MPLS service provider are able to reach it.

 Technologies Under Test

Devices Under Test

  • (6) FortiGate VM - Model: FortiGate VM02 - OS: 6.2.4
  • (1) FortiManager VM - Model: FortiManager-VM - OS: 6.2.4
  • (1) FortiAnalyzer VM - Model: FortiAnalyzer-VM - OS: 6.2.4

Supporting Infrastructure

  • (1) Cisco UCS-C240 M4S - Model: UCSC-C240-M4s - OS: ESXi 6.5
  • (1) Cisco Catalyst 3650 - Model: WS-C2650-24PD-E- OS: 16.06.05
  • (5) Cisco CSR1000v (DC Edge & LAN Routers - Model: Cisco CSR1000V Chassis - OS: 16.06.04
  • (5) Windows 10 VM (Site Endpoints) - Model: Windows 10 - OS: 10
  • (2) FRRouting VM (Branch 2 LAN Router & Branch 3 Edge Router) - Model: FRRouting - OS: 7.3
  • (2) Traffic Jam VM (Internet & MPLS Clouds) - Model: Traffic Jam - OS: 2.0.7

 Test Tools

 Ixia IxLoad - Simulated User Traffic

  • (1) Ixia Traffic Generator - Model: XGS12 - OS: 9.00.1900.17
  • (12) Ixia 1Gb Interfaces - Model: IxLoad XCellon-Ultra NP - OS: 9.00.115.99
Traffic Profiles

These traffic profiles represents a small subset of common applications (HTTP/HTTPs, FTP, Voice) taht we see in an enterprise environment.  At a high level the following traffic patterns will be flowing throughout the topology:

  • All branches to all data centers: HTTP/HTTPs, FTP, Voice
  • Branch 1 to Branch 2: Voice

Outlined below are the traffic profiles that were used during the testing.

 

 Addtional Content

Since the initial testing by the ATC Lab Servies team, the Global Solutions Development team has partnered to create an on-demand lab that provides hands-on experience with the Fortinet SD-WAN Solution.  The environment that supports this lab is completely virtual, removing any hardware dependencies associated with physical infrastructure. This lab itself uses FortiManager as the single pane of glass for configuration, monitoring and deployments. 

The modules included in this lab are:

  • Central Management and Logging
  • SD-WAN
  • Branch Provisioning

Technologies