Notes from the Field

I have recently been involved with an implementation of Microsoft Peering for Expressroute with a large Australian customer and thought I would share the experience with you.

Firstly, and secondly, make sure that you read the specific guidance from Microsoft regarding prerequisites for Microsoft Peering. (See below)

Configure Microsoft peering for the circuit

Make sure that you have the following information before you proceed.

  • A /30 subnet for the primary link. This must be a valid public IPv4 prefix owned by you and registered in an RIR / IRR.
  • A /30 subnet for the secondary link. This must be a valid public IPv4 prefix owned by you and registered in an RIR / IRR.
  • A valid VLAN ID to establish this peering on. Ensure that no other peering in the circuit uses the same VLAN ID.
  • AS number for peering. You can use both 2-byte and 4-byte AS numbers.
  • Advertised prefixes: You must provide a list of all prefixes you plan to advertise over the BGP session. Only public IP address prefixes are accepted. You can send a comma separated list if you plan to send a set of prefixes. These prefixes must be registered to you in an RIR / IRR.
  • Customer ASN: If you are advertising prefixes that are not registered to the peering AS number, you can specify the AS number to which they are registered. This is optional.
  • Routing Registry Name: You can specify the RIR / IRR against which the AS number and prefixes are registered.
  • An MD5 hash, if you choose to use one. This is optional.

https://azure.microsoft.com/en-us/documentation/articles/expressroute-howto-routing-classic/#microsoft-peering

Let me address the items in bold above.

AS Number for Peering: A public AS number is required in order to implement Microsoft Peering. The customer I was working with, didn’t own a public AS, and as a result, were required to apply to APNIC for a public AS. APNIC are the registrar for AS numbers in Asia Pacific

The prerequisites for applying for a public AS number in Australia are as follows, and have been taken from the APNIC site.

Your organization is eligible for an AS Number assignment if:

  • it is currently multihomed, or
  • it holds previously-allocated provider independent address space and intends to multihome in the future.

An organization will also be eligible if it can demonstrate that it will meet the above criteria upon receiving an AS Number (or within a reasonably short time afterwards).

I have heard along the grapevine from colleagues of mine, that this process can be quite time consuming, particularly when the above requirements are not clear, however this wasn’t the case for the customer in question, and the AS number was approved in a timely manner.

Advertised prefixes: This is the list of pubic IP prefixes which azure will receive requests from.

Important Note: Microsoft will run a validation process once the dedicatedcircuit for Microsoft Peering is established. I attempted to find out detailed information regarding the validation process however I was not able to obtain this information. What I was able to find out, is that a validation check is run against the Public AS number to verify that the customer who owns the tenant, also owns the AS. A similar validation check is completed against the AdvertisedPrefixes.

In our case, even though the customer owned both the AS, and the AdvertisedPrefixes, automatic validation failed.

In order to fix this issue, we opened a support call with Azure support who were able to perform a manual validation within about 10 mins of being in contact.

Once the virtual circuit was up, and the AdvertisedPrefixes value showing “configured”, we moved onto configuring the virtual circuit in the Equinix Cloud Exchange.

The customers’ ISP configured the virtual circuits correctly, however we were not able to successfully establish BGP Peering. Another support call was created, with all vendors involved and after some troubleshooting, it was discovered the Primary and Secondary Peer subnets had been configured in reverse and this was causing a mac address mismatch with Azure.

Fortunately, the solution to this issue was to simply recreate the virtual circuit via the cloud exchange portal using the same service key as was used originally. Once this was completed, BGP peering was successfully established.

Premium Sku

Although not documented anywhere (that I could find), Microsoft Peering requires a Premium sku type in order to be enabled.

If you attempt to enable Microsoft Peering on a virtual circuit with a “standard” sku type, you will be promptly rewarded with a powershell error telling you exactly this.

Quick Shoutout: Thanks to our Microsoft Technology Strategist “Scott Turner” for point this out to me ahead of time.

Implementation Process

Create Virtual Circuit

#Import Azure Expressroute modules
Import-Module 'C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1'
Import-Module 'C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\ExpressRoute\ExpressRoute.psd1'

#Creating a new circuit for Sydney O365
$Bandwidth = 500
$CircuitName = "EquinixO365Sydney"
$ServiceProvider = "Equinix"
$Location = "Sydney"
New-AzureDedicatedCircuit -CircuitName $CircuitName `
-ServiceProviderName $ServiceProvider `
-Bandwidth $Bandwidth -Location $Location -Sku Premium

#Creating a new circuit for Melbourne O365
$Bandwidth = 500
$CircuitName = "EquinixO365Melbourne"
$ServiceProvider = "Equinix"
$Location = "Melbourne"
New-AzureDedicatedCircuit -CircuitName $CircuitName `
-ServiceProviderName $ServiceProvider `
-Bandwidth $Bandwidth -Location $Location -Sku Premium

Get your Service Keys

Get-AzureDedicatedCircuit

Enable BGP Peering

Set-AzureBGPPeering -AccessType Microsoft `
-ServiceKey "SYD Service Key" -PrimaryPeerSubnet "x.x.x.x/30" `
-SecondaryPeerSubnet "x.x.x.x/30" -VlanId "Enter Vlan Id" `
-PeerAsn "Enter Public ASN" `
-AdvertisedPublicPrefixes "Enter Advertised Prefixes" -RoutingRegistryName "APNIC"

Set-AzureBGPPeering -AccessType Microsoft `
-ServiceKey "MEL Service Key" -PrimaryPeerSubnet "x.x.x.x/30" `
-SecondaryPeerSubnet "x.x.x.x/30" -VlanId "Enter Vlan Id" `
-PeerAsn "Enter Public ASN" `
-AdvertisedPublicPrefixes "Enter Advertised Prefixes" -RoutingRegistryName "APNIC"

Important Note: Advertising default routes

Default routes are permitted only on Azure private peering sessions. In such a case, we will route all traffic from the associated virtual networks to your network. Advertising default routes into private peering will result in the internet path from Azure being blocked. You must rely on your corporate edge to route traffic from and to the internet for services hosted in Azure.

To enable connectivity to other Azure services and infrastructure services, you must make sure one of the following items is in place:

  • Azure public peering is enabled to route traffic to public endpoints
  • You use user defined routing to allow internet connectivity for every subnet requiring Internet connectivity.

Note: Advertising default routes will break Windows and other VM license activation. Follow instructions here to work around this.

https://azure.microsoft.com/en-us/documentation/articles/expressroute-routing/

Do not advertise the default route to Microsoft Peering as this will break things!

I hope this helps some of you down the track as at the time we completed implementation, there was very little documentation available on the Web for Microsoft Peering.

 

Category:
Azure Infrastructure, Cloud Infrastructure, Office 365
Tags:
, ,

Join the conversation! 12 Comments

  1. Post has been updated to include the Powershell (classic) code which was used to implement the Microsoft Peering for ExpressRoute solution

    Reply
  2. Hi Shane, if I use the NSP for the Microsoft peering, do I need to provision the BGP configuration in my azure portal or the NSP will help me to do that?

    Reply
    • Hi Alan,

      You will still need to configure BGP peering using the Azure Powershell cmdlets however you would not to configure the cloud exchange portal as this is a different ExpressRoute solution to the NSP.

      Thanks,
      Shane.

      Reply
  3. We have multihomed private peering with Microsoft express route. How can i force to use only primary link as preferred path. I’m wondering how routing works at Microsoft side for primary and secondary connection. We loose connectivity to Microsoft sometimes and it comes back up on Secondary and vice versa.

    Reply
    • Hi Manish,

      During implementation testing, we were able to force a failover between the primary/secondary links by shutting BGP neighbourhood on the current primary link.

      Cheers,
      Shane.

      Reply
  4. Thanks for writing this Shane, I’ve found it very helpful. I have a question – I see from your Powershell you configure connections to SYD and MELB, are they two separate ExpressRoute services or is one the primary and one the secondary for the same service?

    Reply
    • Hi Aaron,

      These are effectively primary and secondary connections to the same circuit.

      O365 datacenters can fail over between these two locations and for ExpressRoute to be effective, we require virtual circuits in each location.

      Cheers,
      Shane.

      Reply
  5. Thanks for this informative article. I found the information about using XR premium for office 365 peering in the FAQ.
    https://azure.microsoft.com/en-us/pricing/details/expressroute/

    Can I use Office365 (including Exchange and Skype for Business) on ExpressRoute?
    Yes. Please refer to the Office 365 page for guidance, recommendations, and implementation steps for using ExpressRoute with Office 365. Office 365 license must be separately obtained. You will need to purchase an ExpressRoute Premium circuit to use Office 365 with ExpressRoute. Office 365 license must be separately obtained. Moreover, you will be charged for data transfer (based on consumption) if you choose the ExpressRoute metered pricing plan.

    Reply
  6. Hi,

    How to do NATing. Does Azure public and O365 requires separate pool. Can anyone share an example config on NATing on Juniper routers

    Reply
  7. So I am working for a finance firm and we are moving some workload to the Azure Cloud. We have security concerns on having the Private, Public and O365 connection terminating in the same router. Our network manager wants to use the same router we use for internet traffic and just use a VRF for the Private VLAN. I and a consultant are thinking that we should drop the Telco ethernet circuit into a Layer 2 switch as a trunk and split the VLAN’s so that we can send the Public and O365 VLAN to the Internet Router and the Private VLAN to a Dedicated Azure Router. This would allow us to have a completely separate device that is not accessible from the internet terminating the Azure Private VLAN and BGP routing. We would then run it through a firewall before entering our network for an extra security layer cause we just don’t fully trust Microsoft or anyone for that matter as best practice.

    What is your take on this design/plan.

    Reply
    • Hi Erik,

      The solution you are describing is exactly what was implemented for the customer I was working with.

      Microsoft and Public Peering are terminated on the Internet Router while private peering is terminated on the internal switching environment.

      We have separate layer 2 connections per peer with vrf’s configured on the Internet Router. Each peer is configured on an individual vlan as well. (This may be a requirement depending on whether QinQ or Dot1Q is being used).

      As you eluded to, there is a firewall securing the Microsoft and Public peers. The added benefit of configuring these peers on the Internet Router, is that in the event of an ER peer failure, we can automatically fail over to the internet route without losing service.

      I hope this helps.

      Cheers,
      Shane.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: