Microsoft Azure ExpressRoute provides dedicated, private circuits between your WAN or datacentre and private networks you build in the Microsoft Azure public cloud. There are two types of ExpressRoute connections – Network (NSP) based and Exchange (IXP) based with each allowing us to extend our infrastructure by providing connectivity that is:
- Private: the circuit is isolated using industry-standard VLANs – the traffic never traverses the public Internet when connecting to Azure VNETs and, when using the public peer, even Azure services with public endpoints such as Storage and Azure SQL Database.
- Reliable: Microsoft’s portion of ExpressRoute is covered by an SLA of 99.9%. Equinix Cloud Exchange (ECX) provides an SLA of 99.999% when redundancy is configured using an active – active router configuration.
- High Speed speeds differ between NSP and IXP connections – but go from 10Mbps up to 10Gbps. ECX provides three choices of virtual circuit speeds in Australia: 200Mbps, 500Mbps and 1Gbps.
Microsoft provided a handy table comparison between all different types of Azure connectivity on this blog post.
ExpressRoute with Equinix Cloud Exchange
Equinix Cloud Exchange is a Layer 2 networking service providing connectivity to multiple Cloud Service Providers which includes Microsoft Azure. ECX’s main features are:
- On Demand (once you’re signed up)
- One physical port supports many Virtual Circuits (VCs)
- Available Globally
- Support 1Gbps and 10Gbps fibre-based Ethernet ports. Azure supports virtual circuits of 200Mbps, 500Mbps and 1Gbps
- Orchestration using API for automation of provisioning which provides almost instant provisioning of a virtual circuit.
We can share an ECX physical port so that we can connect to both Azure ExpressRoute and AWS DirectConnect. This is supported as long as we use the same tagging mechanism based on either 802.1Q (Dot1Q) or 802.1ad (QinQ). Microsoft Azure uses 802.1ad on the Sell side (Z-side) to connect to ECX.
ECX pre-requisites for Azure ExpressRoute
The pre-requisites for connecting to Azure regardless the tagging mechanism are:
- Two Physical ports on two separate ECX chassis for redundancy.
- A primary and secondary virtual circuit per Azure peer (public or private).
Buy-side (A-side) Dot1Q and Azure ExpressRoute
The following diagram illustrates the network setup required for ExpressRoute using Dot1Q ports on ECX:
Tags on the Primary and Secondary virtual circuits are the same when the A-side is Dot1Q. When provisioning virtual circuits using Dot1Q on the A-Side use one VLAN tag per circuit request. This VLAN tag should be the same VLAN tag used when setting up the Private or Public BGP sessions on Azure using Azure PowerShell.
There are few things that need to be noted when using Dot1Q in this context:
- The same Service Key can be used to order separate VCs for private or public peerings on ECX.
- Order a dedicated Azure circuit using Azure PowerShell Cmdlet (shown below) and obtain the Service Key and use the this to raise virtual circuit requests with Equinix.https://gist.github.com/andreaswasita/77329a14e403d106c8a6
Get-AzureDedicatedCircuit returns the following output.
As we can see the status of ServiceProviderProvisioningState is NotProvisioned.
Note: ensure the physical ports have been provisioned at Equinix before we use this Cmdlet. Microsoft will start charging as soon as we create the ExpressRoute circuit even if we don’t connect it to the service provider.
- Two physical ports need to be provisioned for redundancy on ECX – you will get the notification from Equinix NOC engineers once the physical ports have been provisioned.
- Submit one virtual circuit request for each of the private and public peers on the ECX Portal. Each request needs a separate VLAN ID along with the Service Key. Go to the ECX Portal and submit one request for private peering (2 VCs – Primary and Secondary) and One Request for public peering (2VCs – Primary and Secondary).Once the ECX VCs have been provisioned check the Azure Circuit status which will now show Provisioned.
Next we need to configure BGP for exchanging routes between our on-premises network and Azure as a next step, but we will come back to this after we have a quick look at using QinQ with Azure ExpressRoute.
Buy-side (A-side) QinQ Azure ExpressRoute
The following diagram illustrates the network setup required for ExpressRoute using QinQ ports on ECX:
C-TAGs identify private or public peering traffic on Azure and the primary and secondary virtual circuits are setup across separate ECX chassis identified by unique S-TAGs. The A-Side buyer (us) can choose to either use the same or different VLAN IDs to identify the primary and secondary VCs. The same pair of primary and secondary VCs can be used for both private and public peering towards Azure. The inner tags identify if the session is Private or Public.
The process for provisioning a QinQ connection is the same as Dot1Q apart from the following change:
- Submit only one request on the ECX Portal for both private and public peers. The same pair of primary and secondary virtual circuits can be used for both private and public peering in this setup.
Configuring BGP
ExpressRoute uses BGP for routing and you require four /30 subnets for both the primary and secondary routes for both private and public peering. The IP prefixes for BGP cannot overlap with IP prefixes in either your on-prem or cloud environments. Example Routing subnets and VLAN IDs:
- Primary Private: 192.168.1.0/30 (VLAN 100)
- Secondary Private: 192.168.2.0/30 (VLAN 100)
- Primary Public: 192.168.1.4/30 (VLAN 101)
- Secondary Public: 192.168.2.4/30 (VLAN 101)
The first available IP address of each subnet will be assigned to the local router and the second will be automatically assigned to the router on the Azure side.
To configure BGP sessions for both private and public peering on Azure use the Azure PowerShell Cmdlets as shown below.
Private peer:
Public peer:
Once we have configured the above we will need to configure the BGP sessions on our on-premises routers and ensure any firewall rules are modified so that traffic can be routed correctly.
I hope you’ve found this post useful – please leave any comments or questions below!
Read more from me on the Kloud Blog or on my own blog at www.wasita.net.
Reblogged this on wasita.net.
Your comment about Dot1Q “The same Service Key can be used to order separate VCs for private or public peerings on ECX” contradicts what it says in the Equinix Deployment Guide.
In the Equinix Deployment Guide it says “If the customer chooses 802.1Q (Dot1Q) on A-side, the customer is required to order two separate dedicated circuits on Azure each with a unique Service key. One service key for a pair of VCs (primary and secondary VC) for private peering and another service key for a pair of VCs (primary and secondary VC) for public peering.”
Hi Marc,
You may have an old version of the Equinix Cloud Exchange Deployment Guide, although I don’t even recall ever requiring 2 separate service keys for private and public – given it would need 2 ExpressRoute circuit purchases which is an additional cost.
Check the version number on the last page of the guide in the footprint, the latest is v1215 from December 2015. You can find it on the Equinix Customer Portal or hit me up via email and I can send a copy.
Bottom line, Andreas was correct up here, the section from page 25 of the current ECX Deployment Guide notes:
“When using 802.1Q on ECX to connect to Azure ExpressRoute, enterprises are required to place separate VC requests on CXP for private and public peerings towards Azure ExpressRoute. When using 802.1Q on ECX, CXP allows up to three VCs to be ordered using the same service key (one VC per public, private and Office 365 peering).”
On top of this, the Dot1Q is also commercially equivalent to QinQ so we don’t penalise customers due to the single tagging technical limitation. Hence there is a business logic in place detecting the use of the same service key for the 3 VCs for different peers – and only 1 VC is billed monthly in this scenario.
Hope this clarifies,
Tamas Horvath
Senior Global Solutions Architect, Equinix