UAG 2010 – Problems with Custom Trunk Ports and Failing Redirects

UAG 2010 prior to Service Pack 1 Update 1 did not support publishing trunks on custom ports – only 80 and 443 were supported. That meant each UAG trunk required a separate IP address per trunk. With SP 1 Update we could publish UAG trunks on custom ports on a single IP address, although it doesn’t seem many people actually did this. For a customer recently where UAG 2010 was required with 5 trunks, there was an existing network architecture restriction that required the UAG servers to use public IP addresses. We needed high availability and using the existing Hardware Load Balancing (HLB) solution meant 15 public IP addresses in a default 1 trunk per IP configuration – 5 per UAG server and 5 HLB virtual IP addresses.

5 trunks were required to meet a combination of different domain names, certificates), different authentication mechanisms, different access methods and authorisation requirements. We could not give the UAG servers private IP addresses for the External interfaces, so the compromise was to publish each trunk on the same public IP address using a custom port. The HLB would publish the trunks on port 443 and connect to UAG on the relevant custom port. This meant 7 public IP addresses used (5 x HLB and 2 x UAG servers) which was the best we could do.

Environment Overview

The diagram above (click for a larger version) shows a high level overview with dummy trunk names and IP addresses showing that each HLB VIP is a different IP address and each UAG trunk is on the same address using a custom port.

Trunk 3 is a non-authenticated trunk for Lync 2013 referenced in my previous blog post Publish Lync 2013 Including Mobility and Office Web Apps with UAG 2010

Trunks 1, 2, 4 and 5 all publish the same applications for Exchange 2010, SharePoint 2013 and FIM 2010 on different domain names and from different locations.

The table below has more detail with dummy trunk and published names.

Issues Encountered

With UAG 2010 Service Pack 3, all applications on trunk 1, 4 and 5 worked correctly (trunk 3 worked in all scenarios due to not doing authentication). The problem was that all applications on trunk 2 (except for SharePoint) would authenticate but not be able to connect to the application. Using Outlook Web App as an example, the client would:

1. Browse to https://mail.showcase.kloud.com.au/owa
2. be taken to the following for authentication https://mail.showcase.kloud.com.au/uniquesig0c4449dcaa9c2c7cb80a00574fc3ea78/uniquesig0/InternalSite/OWA/Login.asp?resource_id=5C0BCBBECE4E49BDA6859D1CD5DDD09C&login_type=2&site_name=trunk1&secure=1&URLHASH=7fc49ee2-2e27-4518-8fb8-b54b0ff77b87&orig_url=https%3a%2f%2fmail.showcase.kloud.com.au%2fowa
3. After authentication the client browser would appear to be stuck on a validation page https://mail.showcase.kloud.com.au/uniquesig0c4449dcaa9c2c7cb80a00574fc3ea78/uniquesig0/InternalSite/Validate.asp
4. After about 40 seconds the connection would timeout with the following in the address bar https://mail.showcase.kloud.com.au/uniquesig0c4449dcaa9c2c7cb80a00574fc3ea78/uniquesig0/InternalSite/RedirectToOrigURL.asp?site_name=trunk1&secure=1

If I went to the original URL again https://mail.showcase.kloud.com.au/owa I could get to the Outlook Web App page without having to authenticate. Sometimes I needed to go to the original URL 2 or 3 times for it to work.

Troubleshooting

UAG 2010 Service Pack 3 tracing symbols had not been released at the time of troubleshooting (they were released on 30 April) so I rolled back to Service Pack 2. With Service Pack 2, only trunk 1 worked correctly so that was making the situation worse. The UAG trace logs were massive and did not get me any closer to a solution.

Fiddler and HttpWatch showed the same thing, that after the validate.asp page it was trying to:

1. Redirect to https://mail.showcase.kloud.com.au/uniquesig0c4449dcaa9c2c7cb80a00574fc3ea78/uniquesig0/InternalSite/RedirectToOrigURL.asp?site_name=trunk1&secure=1
2. and then to one of the following addresses depending on which Service Pack I had
   1. https://trunk2.showcase.kloud.com.au:2443/uniquesigacde95fbf87ff4ca921a1aca1d5b6d510f79207ec5f596b4b80c06329b09f2591329bdfadfac0bdd33fa95f57bf2ef36/uniquesig1/owa/
   2. https://mail.showcase.kloud.com.au:2443/owa

which contained the trunk name and custom port or the published name with a custom port. A sample of the logon experience from HttpWatch is shown below (click for a larger version).

Cause

After a lot of testing and troubleshooting I eventually raised a Microsoft PSS call. After a couple of days of testing, my very knowledgeable PSS engineer eventually worked out the cause of the issue:

• UAG has a problem resolving the names internally when custom ports are in use and the ‘Public host name’ matches the ‘Addresses’ entry. In this example both are set to ‘mail.showcase.kloud.com.au’.

Trunk 1 worked as it was using the default port 443. Trunk 4 and 5 worked as the ‘Public host name’ ‘mail.contoso.com’ was different to the ‘Addresses’ published name ‘mail.showcase.kloud.com.au’. It turns out UAG 2010 Service Pack 2 has another bug with the custom port publishing which is why only trunk 1 worked when using SP 2.

Resolution

Using SP3, the solution for trunk 2 was to change the ‘Addresses’ name UAG connected to the internal servers on to another name e.g. ‘mail-internal.showcase.kloud.com’. This caused another issue – a new name was required on the certificate used by the internal web servers. The internal certificate had to be regenerated with the additional FQDN added to the Subject Alternate Names and was applied to all Exchange Client Access Servers. For FIM we thought we could get away with changing ‘Replace the host header with the following’ as you see below (which is not an option on the OWA application template) but encountered what seems like another bug where UAG does not pass the host header, but rather uses the original ‘Addresses’ host name. So FIM needed a SAN added to the certificate too.

SharePoint worked in all scenarios so I can only assume it is because of the AAM (Alternate Access Mapping) already catering for the different names being published and the host header worked as expected.

Summary

The publishing scenario used here is probably an edge case. Most environments would use private IP addresses and not need custom ports so would not come across this issue. Others using custom ports may have different internal and external DNS namespaces, like mail.showcase.kloud.com.au externally and mail.kloud.local internally which would not hit this issue. I seem to have hit the perfect set of requirements for causing problems. Maybe some karmic payback – although just having to work with UAG is punishment for something. I miss the relative simplicity and flexibility of TMG like being able to publish multiple domain names on a single publishing rule!

Even though others will probably not see this problem, it is good to be aware that there are some issues with custom ports, using the same name internally and externally and that host headers do not always work as expected.

The next blog will cover how to configure and publish Outlook Anywhere NTLM authentication with UAG to provide seamless Single Sign On from domain joined computers.