VLAN-backed VPCs without NSX TEPs

Why this matters…

NSX Virtual Private Clouds (VPCs) are the network model behind VCF Automation and vSphere Supervisor. Until now, standing one up meant preparing the physical network for NSX overlay networking: a dedicated host overlay VLAN and an IP pool or DHCP scope for tunnel endpoints (TEPs) on the hosts.

VCF 9.1.1 adds a second option. A VLAN-backed VPC delivers VPC services without TEPs and without overlay encapsulation. Traffic flows directly on physical VLANs. This post covers how the feature works, the advantages, and the limitations Broadcom documents, so you can decide where it fits.

Scope note: every claim in this post comes from the official VCF 9.1.1 release notes, VCF 9.1 TechDocs, and Broadcom knowledge base articles.

What is a VLAN-backed VPC

The NSX 9.1.1 release notes introduce "support for a VLAN-backed VPC (networking model without TEPs) under the Distributed Transit Gateway." In this model, all traffic flows directly on physical VLANs without overlay encapsulation. Because of that, it does not support private subnets: the IP scheme is limited to the CIDRs backed by physical VLANs.

A Distributed Transit Gateway on its own does not require Edge nodes, dynamic routing, or specific configuration on the physical network. It scales out as you add hosts because traffic goes directly from the host to the VLAN, and services such as DHCP and 1:1 NAT for external IPs are distributed too.

Full stack VPC versus VLAN-backed VPC

The 9.1.1 installer and workload domain wizards ask you to choose a VPC network configuration. The table summarizes the two choices as documented.

Advantages

Fewer physical network prerequisites

With VCF 9.1.1, the dedicated NSX host overlay VLAN is no longer required for VLAN-backed VPCs in the VCF Installer. The Create a New Workload Domain page likewise drops the static IP pool or DHCP requirement on the host overlay VLAN, and states that in TEP-less mode "the Host TEP infrastructure is not required."

No larger MTU for overlay traffic

Broadcom documents MTU minimums that exist to carry TEP traffic. The VCF Installer requires at least 1600 on the distributed switch used for host overlay, and at least 1700 when NSX overlay runs over the ESXi management VMkernel. A VLAN-backed VPC has no TEPs and no overlay encapsulation, so there is no tunnel traffic that needs the extra headroom. Your physical network can keep its standard MTU for this model, which removes one of the most common overlay deployment hurdles.

Simpler, distributed connectivity

Because the model sits under the Distributed Transit Gateway, you avoid NSX Edge nodes and dynamic routing for external connectivity. In the installer, if you select VLAN-backed VPC, the VPC Gateway Connectivity and VNA Cluster Details sections are not displayed, so you do not need a Virtual Network Appliance (VNA) cluster for the VPC itself. VNAs are available for traffic requiring centralized services. For example: they reappear if you also select a Supervisor configuration, because Broadcom documents that distributed connections supporting a Supervisor use a VNA cluster for services such as load balancing. Private CIDRs are not required for the Supervisor configuration.

Access to VPC-based services

VCF Automation 9.1.1 AllApps Org adds support for VLAN-backed VPCs consumed through VCF Automation, and Supervisor supports the model as well. The model uses the existing VLAN infrastructure instead of overlay networking.

It is the default for import

The NSX release notes state that VLAN-backed VPC "is the default model during a VCF import." The import documentation adds that with VCF 9.1.1 and later, the import workflow attempts to configure the cluster for VLAN-backed VPC by default, and the overlay networking option is no longer available. You can configure TEP-based overlay with dedicated TEPs through the SDDC Manager API by using the BrownfieldOverlaySpec.

Limitations and things to plan for

Several of these come straight from the documentation. Read them before you commit to the model, because some can only be changed by redeploying.

Documented design limitations

• No private subnets. The IP scheme is limited to the CIDRs backed by physical VLANs, so workload IPs come from your physical network address space.
• No SNAT. The Supervisor documentation says SNAT is not supported in models without TEPs, and tells you to deselect the default outbound SNAT when creating a connectivity profile.
• No automatic default network. In namespaces with VLAN-backed VPC, the NSX operator does not create a default network for pods or VMs. You must specify a shared subnet explicitly.
• Homogeneous regions. The region a Supervisor belongs to must be homogeneous, with a network stack that is fully VLAN-backed.
• Greenfield only for Supervisor and VCF Automation. In the 9.1.1 release, the VLAN-backed VPC model supports only greenfield deployments of both.
• Version dependencies on import. For a vCenter, NSX and ESX combination at 9.1 or later, a cluster is skipped if ESX is 9.0.x while NSX is 9.1 (version mismatch). Plan your ESX versions before importing.

vSphere Kubernetes Service

The Supervisor documentation states that VKS 3.7.1 and earlier releases do not support the VLAN-backed VPC model, and that on a 9.1.1 Supervisor running those versions you cannot configure it for VKS workloads.

Known issues in 9.1.1

The SDDC Manager and VCF Automation release notes list items worth knowing about before you build.

• Adding a cluster to a VLAN-backed VPC workload domain creates a TEP-less cluster with no network span or transit gateway associated. Broadcom KB 454415 has the workaround. Associating the span re-prepares the cluster for NSX, and the KB warns to expect host downtime during that process.
• Imported or synced clusters under one vCenter are associated with the same VLAN-backed VPC span. If VLAN configuration differs between them, or the vSphere Distributed Switch is not shared, you can see connectivity issues.
• Inventory sync for an out-of-band workload domain can fail with an error that the transit gateway network stack cannot change from FULL_STACK_VPC to VLAN_BACKED_VPC. This happens when a shared NSX instance already has full stack VPCs attached to the default transit gateway. See KB 452453.
• Region creation in VCF Automation can fail with "Following zones of the specified supervisors do not have a network stack configured." Refreshing the vCenter connection in Provider Management and retrying resolves it.
• Cosmetic: the workload domain wizard still shows NSX-Overlay traffic type and transport zone labels, and the VCF Installer can show an incorrect VLAN count on the requirements page (use View Details). Broadcom says these labels do not mean overlay or TEPs are in use.

How to enable it

There are three paths in 9.1.1.

• VCF Installer wizard: choose VLAN-Backed VPC under VPC Network Configuration when you deploy a new fleet or instance.
• Workload domain creation: choose VLAN-Backed VPC on the networking step of the Create a New Workload Domain workflow. If you join an existing NSX Manager, the full stack VPC is shared across all full stack VPC or TEP-configured domains on that instance.
• Import: it is applied by default, as described above.

Choosing between the two models

Choose a VLAN-backed VPC when your priority is using existing VLAN infrastructure, you cannot or do not want to meet overlay requirements, and your workloads can live with externally routable addressing and no SNAT. It also suits a greenfield Supervisor or VCF Automation deployment where the physical network is the source of truth.

Choose a full stack VPC when you need private subnets, outbound SNAT, a Centralized connectivity model with Edge nodes, automatic default networks for pods and VMs, or you are running VKS 3.7.1 or earlier and need a supported path. Because the transit gateway network stack cannot change from full stack to VLAN-backed while VPCs are attached, make this decision early.

Bottom line

VLAN-backed VPCs lower the barrier to entry for VCF networking. For environments held back by overlay prerequisites, they open the door to VCF Automation and Supervisor with a much smaller physical network ask. The trade is functionality: no private subnets, no SNAT, greenfield-only for Supervisor and VCF Automation, and no VKS support in 3.7.1 and earlier. As with any 9.1.1 networking decision, test it against your workload requirements first.

World Wide Technology helps organizations design, validate, and deploy VCF environments, from greenfield builds to complex brownfield imports. Our Advanced Technology Center gives you hands-on access to real VCF infrastructure to test choices like this one before production.

Ready to plan your VCF network design? Let's make it happen.

Visit wwt.com or reach out to your WWT account team.