Exchange 2010 Hybrid Auto Mapping Shared Mailboxes

Migrating shared mailboxes to Office 365 is one of those things that is starting to become easier over time, especially with full access permissions now working cross premises.

One little discovery that I thought I would share is that if you have an Exchange 2010 hybrid configuration, the auto mapping feature will not work cross premises. (Exchange 2013 and above, you are ok and have nothing to worry about).

This means if an on-premises user has access to a shared mailbox that you migrate to Office 365, it will disappear from their Outlook even though they still have full access.

Unless you migrate the shared mailbox and user at the same time, the only option is to manually add the shared mailbox back into Outlook.

Something to keep in mind especially if your user base accesses multiple shared mailboxes at a time.

 

 

Complex Mail Routing in Exchange Online Staged Migration Scenario

Notes From the Field:

I was recently asked to assist an ongoing project with understanding some complex mail routing and identity scenario’s which had been identified during planning for an upcoming mail migration from an external system into Exchange Online.

New User accounts were created in Active Directory for the external staff who are about to be migrated. If we were to assign the target state, production email attributes now, and create the exchange online mailboxes, we would have a problem nearing migration.

When the new domain is verified in Office365 & Exchange Online, new mail from staff already in Exchange Online would start delivering to the newly created mailboxes for the staff soon to be onboarded.

Not doing this, will delay the project which is something we didn’t want either.

I have proposed the following in order to create a scenario whereby cutover to Exchange Online for the new domain is quick, as well as not causing user downtime during the co-existence period. We are creating some “co-existence” state attributes on the on-premises AD user objects that will allow mail flow to continue in all scenarios up until cutover. (I will come back to this later).

generic_exchangeonline_migration_process_flow

We have configured the AD user objects in the following way

  1. UserPrincipalName – username@localdomainname.local
  2. mail – username@mydomain.onmicrosoft.com
  3. targetaddress – username@mydomain.com

We have configured the remote mailbox objects in the following way

  1. mail – username@mydomain.onmicrosoft.com
  2. targetaddress – username@mydomain.com

We have configured the on-premises Exchange Accepted domain in the following way

  1. Accepted Domain – External Relay

We have configured the Exchange Online Accepted domain in the following way

  1. Accepted Domain – Internal Relay

How does this all work?

Glad you asked! As I eluded to earlier, the main problem here is with staff who already have mailboxes in Exchange Online. By configuring the objects in this way, we achieve several things:

  1. We can verify the new domains successfully in Office365 without impacting existing or new users. By setting the UPN & mail attributes to @mydomain.onmicrosoft.com, Office365 & Exchange Online do not (yet) reference the newly onboarded domain to these mailboxes.
  2. By configuring the accepted domains in this way, we are doing the following:
    1. When an email is sent from Exchange Online to an email address at the new domain, Exchange Online will route the message via the hybrid connector to the Exchange on-premises environment. (the new mailbox has an email address @mydomain.onmicrosoft.com)
    2. When the on-premises environment receives the email, Exchange will look at both the remote mailbox object & the accepted domain configuration.
      1. The target address on the mail is configured @mydomain.com
      2. The accepted domain is configured as external relay
      3. Because of this, the on-premises exchange environment will forward the message externally.

Why is this good?

Again, for a few reasons!

We are now able to pre-stage content from the existing external email environment to Exchange Online by using a target address of @mydomain.onmicrosoft.com. The project is no longer at risk of being delayed ! 🙂

At the night of cutover for MX records to Exchange Online (Or in this case, a 3rd party email hygiene provider),  We are able to use the same powershell code that we used in the beginning to configure the new user objects to modify the user accounts for production use. (We are using a different csv import file to achieve this).

Target State Objects

We have configured the AD user objects in the following way

  1. UserPrincipalName – username@mydomain.com
  2. mail – username@mydomain.com
  3. targetaddress – username@mydomain.mail.onmicrosoft.com

We have configured the remote mailbox objects in the following way

  1. mail
    1. username@mydomain.com (primary)
    2. username@mydomain.onmicrosoft.com
  2. targetaddress – username@mydomain.mail.onmicrosoft.com

We have configured the on-premises Exchange Accepted domain in the following way

  1. Accepted Domain – Authoritive

We have configured the Exchange Online Accepted domain in the following way

  1. Accepted Domain – Internal Relay

NOTE: AAD Connect sync is now run and a manual validation completed to confirm user accounts in both on-premises AD & Exchange, as well as Azure AD & Exchange Online to confirm that the user updates have been successful.

We can now update DNS MX records to our 3rd party email hygiene provider (or this could be Exchange Online Protection if you don’t have one).

A final synchronisation of mail from the original email system is completed once new mail is being delivered to Exchange Online.

WPAD and Proxy Auth Cause Exchange HCW to Fail

A recent conversation with a colleague reminded me of an issue I’ve faced a number of times (and forgotten to blog about) when running the Exchange Hybrid Configuration Wizard (HCW) on Exchange 2010 or 2013 in an environment where Web Proxy Autodiscovery Protocol (WPAD) is used.

The Problem

The most common scenario where I’ve seen this come into play is along the lines of this:

  1. WPAD is used to distribute Proxy.PAC to client machines
  2. Customer permits direct connection from Exchange servers to Internet
  3. From an elevated command prompt, run “netsh winhttp reset proxy” to ensure a direct connection
  4. Change Internet Options settings from “Automatically detect settings” to “Disabled”
  5. Browse to a site restricted by the proxy to confirm proxy bypass is working
  6. Can connect to Exchange Online using Remote PowerShell
  7. Run the HCW but it fails with the following error in the logs:
    ERROR : System.Management.Automation.RemoteException: Federation information could not be received from the external organization.
    ERROR : Subtask NeedsConfiguration execution failed: Configure Organization Relationship
    Exchange was unable to communicate with the autodiscover endpoint for your Office 365 tenant. This is typically an outbound http access configuration issue. If you are using a proxy server for outbound communication, verify that Exchange is configured to use it via the “Get-ExchangeServer –InternetWebProxy” cmdlet. Use the “Set-ExchangeServer –InternetWebProxy” cmdlet to configure if needed.

At this point, you’d probably be scratching your head wondering where it’s going wrong. I certainly was. It can be difficult to troubleshoot issues with the HCW as it is over 50 configuration steps all wrapped up in a ‘friendly’ GUI. The first error gives a good indication of where to look: Federation information could not be received from the external organization.

Check whether federation information can be retrieved from Office 365 by running the cmdlet from your Exchange server:

Get-FederationInformation –DomainName YourTenant.mail.onmicrosoft.com - BypassAdditionalDomainValidation: $true -Verbose

In my case it failed and I found this little gem in the verbose output:

Exception=The remote server returned an error: (407) Proxy Authentication Required

So where is it getting this proxy configuration from? As it turns out, the HCW (and the individual cmdlets) actually run in the user context of the Local System account, rather than the current user. The default configuration for Internet Options is for “Automatically Detect Settings” to be enabled on a per-user basis, which also applies to the Local System account. This default setting combined with WPAD and client’s PAC file is directing the HCW out through the proxy server, which is always going to fail given that Local System is not a credential that the proxy server will validate.

The Workarounds

There are a few ways to address this, although the first two may be harder to implement if it’s a larger organisation:

  • Permit the server to use the proxy without authentication
  • Modify the PAC file settings being distributed by WPAD

Alternatively it is possible to change the default Internet Options settings within the Local System profile:

  1. In your own profile, configure the default Internet Option setting from “Automatically Detect Settings” to “Disabled” and export the registry key from HKCU. Import this key into the “Local System” (HKEY_USERS\.DEFAULT) hive. Always remember to back up your registry before making changes.
    HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
  2. Use PsExec to launch Internet Explorer as “Local System”, disable the setting and save the changes
    psexec.exe -i -s -d “C:\Program Files\Internet Explorer\iexplore.exe”

Exchange Server hybrid “edition” myths and misunderstandings

Originally posted on Lucian’s blog at clouduccino.com (click here to view). Follow Lucian on Twitter @LucianFrango.


There’s a common misunderstanding that Exchange Server hybrid (whichever version you may be running) is needed to be kept on-premises forever if you have Azure AD Connect. AADC syncs on-premises Active Directory with Azure AD. When AADC and federated identity is enabled, MOST of the cloud attributes in Azure AD are READ ONLY. From that statement it’s been understood that hybrid is needed to be maintained to do all that Exchange Online remote management goodness. Wrong!

I hate to burst the bubble here, but, I’m going to burst the bubble.

Exchange Server Hybrid

Being a consultant, I’m going to do that frustrating thing and say those famous words: “it depends on the situation”. I love being ambiguous sometimes as it affords room for different options and ideas which is great for brainstorming and architecting.

Looking at Exchange Server hybrid functionality independently of thinking about the common tech phrase “hybrid”, what does Exchange Server hybrid do? Put simply, which isn’t very clear on TechNet or other publications, hybrid creates send and receive connectors between on-premises and Office 365 EXO. It’s now just a wizard / setup application that completes a few commands that can be achieved manually through powershell. It’s not even an Exchange Server role anymore.

Read More

Exchange Server 2016 Hybrid upgrade considerations

Originally posted on Lucian’s blog at clouduccino.com. Follow Lucian on Twitter via @LucianFrango.


 

Exchange Server 2016, RTM as of October 2015, is still very much freshly baked having just come out of the oven from Redmond. Two recent projects that I’ve worked on have required me to consider deploying it as the “Hybrid” server (not an actual role- I’ll get to that later) for integration and coexistence with Office 365 Exchange Online.

With anything new there is a learning curve as to how the new product now works (not that dissimilar from previous versions of Exchange Server) and what will work with the existing environment to not compromise service.

There is an unwritten assumption that is made in our hybrid guidance that you have already properly deployed and completed the coexistence process with the current versions of Exchange in your on-premises environment.

– The Exchange Team

Read More

Skype for Business Online to On-Premises Migration

Okay guys – you’ve been told “lets move everyone back from the cloud! We need Enterprise Voice for our users” This will go against most Microsoft sales materials as we should be looking towards cloud.

If you are part of an organisation that has been birthed out of Skype for Business Online (SFBO) as part of your Office 365 subscription, it would make sense that you would have never had on-premises Lync or SFB servers in your Active Directory domain. Very little configuration is needed in SFBO and a busy administrators would have loved enabling the license SKU for SFBO for each users and then wiped their hands of it. Its just that easy, enable and forget.

The main limitation around SFBO is the need for an IP-PBX and/or PSTN connectivity. The time may have come for your organisation to leverage your Microsoft agreement even further and look to your existing technology/application catalogue and see that Skype for Business can fill the requirements of your aging PBX. This trigger point is usually when the PBX asset has reached capacity and there is a cost trade off;

  • Throw more money at the dusty old PBX box for extra expanision cards and possibly cabling to terminal
  • Spend the money to start new in the world of VOIP, but look to an existing technology that can provide this functionality (and hopefully more) before looking elsewhere

Telephones have been around for a long time, its nothing new. Picking up, making, transferring calls is all pretty standard stuff we have been doing for few decades now. If I’m going to invest money in something I should be asking for more! more! more! How do I make sure that my choice keeps my organisation staying relevant in the way it communicates for the next chapter?

Hello Skype for Business as PBX replacement.

That’s enough of ramble. You have come here to understand the process of moving users back from the cloud because there is less documented about this procedure than too the cloud.

Current Environment

  • On-premises Domain Services
  • DirSync/AADSync Server
  • Office 365 Tenant
    • Skype for Business Online SKU enabled
  • DNS for my domain.com.au
    • Lyncdiscover.domain.com CNAME webdir.online.lync.com
    • SIP.domain.com. CNAME sipdir.online.lync.com
    • _sipfederationtls._tcp.domain.com SERVICE sipfed.online.lync.com
    • _sip._tls.domain.com SERVICE sipdir.online.lync.com

Current_Environment

In this scenario we will look at the steps needed to specifically enable Hybrid and move users back to on-premises.

  1. Add on-premises infrastructure
  2. Connect Hybrid SFB with Office 365 and on-premises
  3. Move Enterprise Voice users back

Add On-premises Infrastructure

  • Add your Front End, Edge and Reverse Proxy Infrastructure
    1. Build the Servers as per TechNet, but leave the SIP Address’s DNS zones to not effect internal and external clients just yet
    2. All discover records should still point to Microsoft (sipdir, webdir etc)
  • Confiure your Edge and Reverse Proxy with Public Certificates
    1. Test the port authentication as best you can;
      1. Telnet to Edge Ports
      2. Test Reverse Proxy URLs
      3. Remote Connectivity Analyzer for Edge
  • Configure Edge for Federation
    1. Assign your Edge as the Federation Route in your Topology Builder
    2. Configure Edge Specific Configuration
Set-CSAccessEdgeConfiguration -AllowOutsideUsers 1 -AllowFederatedUsers 1 -UseDnsSrvRouting -EnablePartnerDiscovery $true
  • Recreate Allowed Federated Domains in On-premises
    1. If you do a get-csalloweddomain in Office 365 you may not get all the correct information specific for your tenant back to you.
      1. If you have federation with only allowed/block lists, you may need to recreate these as there is no nice way of piping the cmdlets from ‘get’ to ‘new’.
      2. Allow open federation to accommodate for all traffic is the simplist approach for migration
  • Set the Global Remote Access and Federation User Policy to Allow
Get-CsExternalAccessPolicy -Identity Global | Set-CsExternalAccessPolicy -EnableFederationAccess $True -EnableOutsideAccess $True

 

Connect Hybrid Skype for Business

  • Remove existing Lync/Skype for Business Hosting Rule
Get-CSHostingProvider -Identity <SFB/Lync Online> | Remove-CSHostingProvider
  • Recreate the Provider with Hybrid Specific Configuration
New-CSHostingProvider -Identity SFBOnline -ProxyFqdn "sipfed.online.lync.com" -Enabled $true -EnabledSharedAddressSpace $true -HostsOCSUsers $true -VerificationLevel UseSourceVerification -IsLocal $false -AutodiscoverUrl https://webdir.online.lync.com/Autodiscover/AutodiscoverService.svc/root
  • Update External/Public DNS Records
    1. Remember that only updating external DNS records means that your internal users can functions ‘as-is’ until your happy with the progress
      1. Edge Names (SIP Access/Web Conference/ AV  FQDNs)
      2. External Web Services FQDN
      3. Dialin FQDN
      4. Meeting FQDN
      5. LyncDiscover FQDN
      6. SRV _sipfederationtls._tcp.domain.com
      7. SRV _sip._tls.domain.com
    2. Remote users that aren’t previously authenticated could have an issue logging in at the time of the change

Test Process

Join On-premises Pilot User with Online Account

This makes your on-premises deployment aware of active directory accounts that are currently cloud enabled.

  • Run the following cmdlet in SFB Management Shell connected to on-premises servers to test a user
Enable-CsUser -Identity &lt;accountname&gt; -SipAddress "sip:<sipaddress>" -HostingProviderProxyFqdn "sipfed.online.lync.com" -verbose
  • Synchronise AADSync/DirSync
    1. Login to Directory Sync Server
    2. Run >  Delta Import And Delta Sync on the Active Directory Connector
    3. You will see an update count that includes your object

Move a Pilot Users back

This step will actually move the user from SFB Online back to your on-premises pool with contact lists intact. This is initiated from the on-premises server and will need authentication for the Office 365 tenant to perfom the task.

  • Run the following cmdlet in Powershell connected to both on-premises and online sessions
Import-Module LyncOnlineConnector
$credential = Get-Credential
$session = New-CsOnlineSession -Credential $credential
Import-PSSession $session -AllowClobber
  • Get the Online Admin URL for your tenant
    1. Log into Office 365 Portal
    2. Check the URL presented in the address bar, will be admin0x. Where x = a letter specific for you
  • Move the User back
Move-CsUser -Identity <UPN> -Target <FE Pool Name> -Credential $cred -HostedMigrationOverrideURL https://admin0f.online.lync.com/HostedMigration/hostedmigrationservice.svc

Enable All Users

If the above to Pilot Tests worked we need to scale up our migration batches. We need to mass produce the following cmdlet in SFB Management Shell connecting on-premises user accounts to the corresponding online account;

Enable-CsUser -Identity <accountname> -SipAddress "sip:&lt;sipaddress&gt;" -HostingProviderProxyFqdn "sipfed.online.lync.com" -verbose

To do this practically I used the UPN value which I knew that would resolve to the correct users values in on-premises and Office 365 because they are synced from the source. I also could then understand the logic that the users UPN is in this case the primary SMTP/Mail value and therefor matching SIP Address for Skype for Business that I needed.

  • Get all the Office 365 users that are enabled for Skype for Business
Get-CSOnlineUser | ? {$_.SipAddress -notlike $Null} | Select SipAddress, DisplayName | Export-CSV -Path C:\temp\OnlineUsers.csv -NoTypeInformation
  1. This will give you a list of ‘real’ SFBO users that are licensed but are also registered SFB logins
  2. Review the list for deleted users that haven’t been removed properly, there SIPAddress will include GUID style login, these lines can be removed as we do not wish to migrate them.

Lets leverage this list of known online users and enable their joining in on-premises with a ForEach loop example below;


$Users = Import-Csv C:\updates\OnlineUsers.csv

ForEach($User in $Users)
{
$SipAddress = $user.sipaddress
$UPN = $SipAddress.replace("sip:", "")
$Enable = Enable-CsUser -Identity $UPN -SipAddress $SipAddress -HostingProviderProxyFqdn "sipfed.online.lync.com"
}
  • Update Azure Active Directory of the changes by another AADSync/DirSync > Delta Import & Delta Sync
  • Update Internal DNS to point all associated SFB records to on-premises Skype for Business Server(s)
    1. SRV _sipinternaltls._tcp.domain.com.au
    2. Lyncdiscoverinternal.domain.com.au
    3. SIP.domain.com.au
  • Add Additional A Records
    1. Meet.domain.com.au
    2. Dialin.domain.com.au
    3. Pool Name
    4. SFB Web Service URL Names
    5. Admin URL

Visual Indication of Success

Log into your on-premises SFB Admin Control Panel and run a blank user search to discover all users. Noticed the ‘Homed’ field should say ‘SFBOnline’ 

Move All Users

Leveraging the same list of users, run the move cmdlet like the example;


ForEach($User in $Users)
{

$Displname = $user.displayname
$SipAddress = $user.sipaddress
$UPN = $SipAddress.replace("sip:", "")
$Move = Move-CsUser -Identity $UPN -Target &amp;amp;amp;amp;amp;amp;lt;FE Pool Name&amp;amp;amp;amp;amp;amp;gt; -Credential $credential -HostedMigrationOverrideURL https://admin0f.online.lync.com/HostedMigration/hostedmigrationservice.svc -Confirm:$false
if($Move -eq $False)
{
Write-host "User $SipAddress didn't move!!"
}
}

Status

To get visual status while you move all the users, log into your Office 365 Skype for Business Administration Portal and view the details. Continually refresh the page to see the value for “users synced and homed online” go down as each user becomes enabled on-premises.

SFB_User_Stats

 

 

 

Log into your on-premises SFB Admin Control Panel and run a blank user search with a additional filter for Homed or Registrar Pool / is equal to / <registrar server name>.

Client Experience

The client should be unknowning of your changes being made in Office 365 and on-premises until you perform the move-csuser request for their account. During this period a redirect message will be sent to the client with a new registrar server FQDN and a automated logout and login will happen. If the user doesn’t have their client in the forground of their desktop, then this will happen silently in the background. The redirect in my move request had the users logged out and back in within about 1-2 seconds.

Hybrid Exchange Connectivity with Azure Traffic Manager

Does your exchange hybrid architecture need to have redundancy? How about an active/passive solution using Azure Traffic Manager elimating the need for a HLB device in your DMZ.

Currently there is a few topologies for configuring Hybrid Exchange with Office 365;

  1. Single Hybrid Server
  2. 2+ Hybrid Server behind a load balancer
  3. 2+ Hybrid Server with DNS round robin

A simple solution to make a redundant Hybrid Exchange design without using a HLB is to leverage Azure Traffic Manager to monitor and service the DNS namespace configured in on-premises Exchange and Office 365 configuration.

Traffic Manager offers:

  • It works at the DNS level, routing traffic between one or more hybrid public endpoints that sit behind the common DNS service name
  • Traffic management policy profiles like failover and performance metrics rather than DNS round robin and TTL.

In my scenario we will make the Primary Hybrid Server the responding IP until the event of a failure. We do this via pro-actively probing the health check page on a web services virtual directory on each Hybrid Server where NTLM challenge authentication is not required for the anonymous probe. If we think about all the services offered across exchange virtual directories that are exposed, Outlook Web Access with forms based authentication seems to be the standout. OWA has a health check page that can be used by load balancers to check for service outages and degredation.

“/owa/healthcheck.htm”

If in the event that this service becomes unresponsive, Traffic Manager will start responding with the secondary server endpoint IP until services resume. We have comfort in knowing that if the OWA page is unresponsive we can be assured that the Exchange Web Services (EWS) virtual directory probably is aswell.

New-AzureTrafficManagerProfile -Name "HybridExchange" -DomainName "KloudExHybrid.trafficmanager.net" -LoadBalancingMethod "Failover" -Ttl 30 -MonitorProtocol "Https" -MonitorPort 443 -MonitorRelativePath "/owa/healthcheck.htm" | Add-AzureTrafficManagerEndpoint -DomainName "hybrid1.kloud.com.au" -Status "Enabled" -Type "Any" | Add-AzureTrafficManagerEndpoint -DomainName "hybrid2.kloud.com.au" -Status "Enabled" -Type "Any" | Set-AzureTrafficManagerProfile

Create a CNAME for the hybrid namespace configured with Office 365 to KloudExHybrid.trafficmanager.net. For the example outlined in this blog, I would create a CNAME in my public DNS Zone for kloud.com.au with ‘exhybrid.kloud.com.au’ to ‘kloudexhybrid.trafficmanager.net’.

Exchange_Hybrid_TrafficManager

 

In this solution you will know that the primary Exchange Hybrid Server will always respond to Office 365 web services requests until the event of a failure. This will include free/busy lookups and also mailbox migrations. The benefits of using an active/passive hybrid namespace rather than a HA pair is that the through put of mailbox migration is much higher as EWS requests maintain a persistent connection with the originating Hybrid Server and no load balancer is intercepting/redirecting the traffic. We also have control on the failover/failback timings versus true DNS load balancing with round robin which could take longer to resolve.

This is a simple solution using a cloud service for eliminating the need of any fancy layer 7 load balancing and improving on the simplicity of DNS round robin.

Hot Tip: If you have a Database Availability Group (DAG) on-premises with mailboxes being migrated to Exchange Online. It would be of benefit for your active Hybrid Server to be closer to the mailbox server with the active copy to reduce mailbox move traffic on your local network links during synchronisation.