Publish Lync 2013 with 2012 R2 Preview Web Application Proxy

I discussed the new Windows 2012 R2 Preview Web Application Proxy (WAP) remote access role in a previous post Windows 2012 R2 Preview Web Application Proxy – Exchange 2013 Publishing Tests. I showed how to publish Exchange 2013 (except for Outlook Anywhere which isn’t working) and a claims based application.

In this post I am going to cover:

• how to publish (reverse proxy) Lync 2013 and Office Web Apps Server 2013 (OWAS) using the Web Application Proxy
• why the Office Web Apps Server published URLs must be the same
• fix to make Lync mobile work (updated 19 July 2013 with information about Windows Phone 8 SNI support)
• the solution to a problem I had testing this on Azure IaaS Virtual Machines

Publishing Lync Applications

Lync has a few different namespaces that need to be published:

• Lync External Web Services (which includes the Lync Web App and Lync Scheduler)
• Lync meeting join
• Lync dialin page
• Lyncdiscover for client autodiscover
• Office Web Apps Server for PowerPoint sharing

Lync applications cannot use Preauthentication and have to use Pass-through which lets the backend server provide authentication. Setup the Web Application Proxy by installing the Remote Access Web Application Proxy role service as mentioned in the previous post. Install the certificate and run the configuration wizard to setup the server as an ADFS proxy.

Note that there is a dependency on a Windows 2012 R2 Preview ADFS farm. Even though we are not using Preauthentication, WAP cannot be setup without being configured as an ADFS Proxy.

The Lync applications can be published manually like this – note the port 4443 on the Backend server URL:

Or using a script:

[sourcecode language=”powershell”]
$domain = "marc.kloud.com.au"
$webServices = "lyncws."
$certificate = "F7F6B300FE4D569FD598E1C9722571CF9AD780DD"

Add-WebApplicationProxyApplication -Name ‘Lync Web Services’ -ExternalPreAuthentication PassThrough -ExternalUrl "https://$webServices$domain/" -BackendServerUrl ("https://"+$webServices+$domain+":4443/") -ExternalCertificateThumbprint $certificate
Add-WebApplicationProxyApplication -Name ‘Lync Lyncdiscover’ -ExternalPreAuthentication PassThrough -ExternalUrl "https://lyncdiscover.$domain/" -BackendServerUrl ("https://lyncdiscover."+$domain+":4443/") -ExternalCertificateThumbprint $certificate
Add-WebApplicationProxyApplication -Name ‘Lync Dialin’ -ExternalPreAuthentication PassThrough -ExternalUrl "https://dialin.$domain/" -BackendServerUrl ("https://dialin."+$domain+":4443/") -ExternalCertificateThumbprint $certificate
Add-WebApplicationProxyApplication -Name ‘Lync Meet’ -ExternalPreAuthentication PassThrough -ExternalUrl "https://meet.$domain/" -BackendServerUrl ("https://meet."+$domain+":4443/") -ExternalCertificateThumbprint $certificate
Add-WebApplicationProxyApplication -Name ‘Office Web Apps Server’ -ExternalPreAuthentication PassThrough -ExternalUrl ‘https://owas.marc.kloud.com.au/’ -BackendServerUrl ‘https://owas.marc.kloud.com.au/’ -ExternalCertificateThumbprint $certificate
[/sourcecode]

Important Configuration

There are three main things to be aware of when publishing Lync and Office Web Apps Server:

• The Office Web Apps Server published application must have the same host name for ‘ExternalURL’ and ‘BackendServerURL’
• The Lync 2013 mobile clients do not support Server Name Indication (SNI)
• As we are publishing a full domain, not a sub path, you cannot use the same backend host name. So make sure that the backend servers have certificates for all the public names too

Office Web Apps Server URLs

Web Application Proxy supports header rewriting/translation in order to publish different internal names to the external names. It does not yet support body rewriting which is what I think the warning at the top of the diagram above is all about. It links to http://technet.microsoft.com/library/cc732148(WS.10).aspx but that content does not exist yet. I initially thought the problem may be the Office Web Apps Farm InternalURL configuration which had the internal server name shown below:

But that was not an issue. If I set the OWAS publishing rule on the WAP server with different external and internal URLs:

The client gets a referral to the Office Web Apps Server external URL https://owas.marc.kloud.com.au/m/ but the response body contains some X-Headers and javascript URLs for the backend URL of https://marc-owas1.marc.kloud.com.au/m/ URLs as you can see on the right hand side in the diagram below (click for a larger version). The client then tries to open a connection to the internal hostname which does not exist in DNS.

In order to make this work, the Web Application Proxy publishing rule for Office Web Apps Server must have the same ‘ExternalURL’ and ‘BackendServerURL’.

Lync mobile clients do not support SNI

Server Name Indication (SNI) is a feature that is supported by just about every client browser (but not at all on Windows XP), but has only been supported by Microsoft web servers since IIS 8 in Windows 2012. SNI allows multiple certificates to be used on a single IP and port combination similar to the way host headers work with HTTP. With HTTPS traffic, the client needs to perform the certificate handshake before it sends the host header, so the server does not know which site is being requested and therefore which certificate to present. Lync 2013 and the Lync Windows Store App support SNI (I think because they use Internet Explorer underneath) however the Lync mobile clients do not.

Update 19 July 2013: After additional testing it was confirmed that Windows Phone 8 Lync Mobile 2013 and ActiveSync clients do support SNI, so the following problem will not be seen if you only use Windows Phone 8. iOS has supported SNI since version 4 and Android since version 3 (Honeycomb) and you can test this in a browser by going to an SNI test site like https://alice.sni.velox.ch/ – however the Lync and ActiveSync clients do not provide the SNI extension when connecting on either mobile OS (iOS does send the SNI name for autodiscover but not when it connects to the Microsoft-Server-Activesync URI).

The Problem

As I was testing this on a lab I did not have public DNS or a public certificate. I used the Lync 2013 client on my Android phone, modified the hosts file and imported the root certificate. The Lync mobile client would not connect and I did not see any traffic hitting the WAP application in the WAP performance counters. When viewing the client side logs I saw this:

HttpConnection: javax.net.ssl.SSLException: SSL handshake aborted: ssl=0x5886b8e0: I/O error during system call, Connection reset by peer

I installed Wireshark to see what was happening on the WAP server and what I saw pointed me at the problem:

Working: A web browser sends the ‘server_name’ extension containing the host name as can be seen below.

Not working: The Lync mobile client does not send the ‘server_name’ extension (neither does the Exchange ActiveSync client) as can be seen below:

The problem is that the client does not specify a server name and WAP does not specify a default certificate. I can’t change the client function (although I hope Microsoft will add SNI support soon), so the only option is to change the server.

The Solution

Finding the solution took a lot of searching. This TechNet blog Server Name Indication (SNI) with IIS 8 (Windows Server 2012) pointed me in the right direction when talking about how IIS 8 has a way to add a legacy SSL binding to support non-SNI compliant clients. Web Application Proxy however is not based on IIS. That blog mentioned using netsh to view the HTTP SSL bindings:

[sourcecode language=”powershell”]

netsh http show sslcert

[/sourcecode]

This lists all bindings including Remote PowerShell and ADFS

As well as the Web Application Proxy published applications

The MSDN article How to: Configure a Port with an SSL Certificate showed how to add a new binding. The trick is to add an IP:port binding in addition to the Hostname:port which acts as a legacy non-SNI binding. As all of the WAP applications have the same certificate and Application ID, I reused those and created the new binding:

[sourcecode language=”powershell”]

netsh http add sslcert ipport=0.0.0.0:443 certhash=f7f6b300fe4d569fd598e1c9722571cf9ad780dd appid={f955c070-e044-456c-ac00-e9e4275b3f04}

[/sourcecode]

This assumes that all names are on the single certificate as it sets the default binding on all IP addresses. If you had multiple certificates you would need multiple IP addresses and set a binding for each as you would have in TMG or UAG. This really reduces the effectiveness of SNI, so in order to not complicate the configuration or waste IPv4 addresses the clients need to be updated.

Bonus Feature

As mentioned by my colleague in An Overview of Server Name Indication (SNI) and Creating an IIS SNI Web SSL Binding Using PowerShell in Windows Server 2012, it should be possible to script and automate all of our configuration. I thought it would be a quick process to automatically create the legacy binding, but had to use good old PowerShell text parsing to get it working. This is not the most elegant script, but I am a self-taught scripter and this works for me. It does the following:

• takes the netsh output and pulls out the IP:port or Hostname:port as well as the Certificate Hash and Application ID
• removes the split and all spaces
• creates an object for each existing binding
• matches the existing WAP binding that has the correct certificate
• creates the netsh command for creating the binding
• runs the netsh command (netsh creation cannot be run directly in PowerShell which is why the command is piped to netsh)
• shows the new binding information

[sourcecode language=”powershell”]
$hostName = "mail.marc.kloud.com.au" # Name of existing hostname to match
$IPport = "0.0.0.0:443" # IP and port to bind to. 0.0.0.0:443 matches all

$certsList = @()
$certs = netsh http show sslcert | where {$_ -match ":port" -or $_ -match "Certificate Hash" -or $_ -match "Application ID"}
$certs = $certs.replace(" : ","#").replace(" ","")
$certs | foreach {
if ($_ -match ":port") {
$tempObject = New-Object -Type PSObject
$tempObject | Add-Member -Type noteproperty -Name Binding -value $_.split("#")[1]
}
elseif ($_ -match "CertificateHash") {
$tempObject | Add-Member -Type noteproperty -Name CertificateHash -value ($_.split("#")[1]-replace " ")
}
else {
$tempObject | Add-Member -Type noteproperty -Name ApplicationID -value ($_.split("#")[1]-replace " ")
$certsList += $tempObject
}
}

$match = $certsList | where {$_.Binding -match $hostName}
$CertificateHash = $match.CertificateHash
$ApplicationID = $match.ApplicationID

cls
write-host -ForegroundColor Gree "`n`nDefault bindings before changes"
netsh http show sslcert | where {$_ -match "ip:port"}

$command = "http add sslcert ipport=$IPport certhash=$CertificateHash appid=$ApplicationID"
write-host -ForegroundColor Yellow "`n`nAdding Certificate"
$command | netsh

write-host -ForegroundColor Yellow "`n`nDefault bindings after changes. Check that $IPport exists"
netsh http show sslcert | where {$_ -match "ip:port"}
[/sourcecode]

I save this as ‘Set-DefaultCertificate.ps1’ and the result of running the script is shown below.

Lync on Azure

I originally setup my lab on Azure as it was the fastest way for me to get up and running with Server 2012 R2 Preview. I then wanted to test publishing Lync, so installed a Lync Standard Edition server. Note that Lync is not listed as supported software on Azure Microsoft server software support for Windows Azure Virtual Machines. Lync Front End servers can run on Azure but Lync Edge servers cannot because they need two distinct network interfaces – one for internal communication and one for external. Azure does not support more than one network interface and also does not support adding additional IP addresses to a network interface, so there is no way to install a Lync Edge server.

I had setup Lync 2013 before in Azure, but this time I hit an error installing the SQL databases as part of step 2 of the ‘Install or Update Lync Server System’ in the Lync Deployment Wizard where it tries to create and attach the rtcxds database:

InstallDatabaseInternalFailure: An internal error has occurred while trying to create or update the database

Error: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding

InstallDatabaseInternalFailure: An error occurred while applying SQL script for the feature BackendStore

I could see the rtcxds.mdf and rtcxds.ldf files being created, but after 10 minutes it would fail with a timeout error and the databases would be removed. I initially thought it was an environmental issue as I had tried to use Pat Richard’s Set-Cs2013Features script from http://www.ehloworld.com/1697 which preinstalls the SQL Express instances and patches. So I installed another Windows 2012 server and just let Lync do the SQL install – same problem. I went back to a Windows 2008 R2 server and still had the same problem. Sorry to doubt your script Pat J

I tried a whole lot of things to do with the NT AUTHORITY\NETWORK SERVICE as there were some errors regarding that, but eventually I discovered the reason based on the last comment on this TechNet support forum post http://social.technet.microsoft.com/Forums/lync/en-US/f8d960d3-eaea-442a-8a03-da8b38ca7e1a/backendstore-database-creation-fails-lync-2013

Azure Operating System disks have read/write host cache enabled – primarily aimed at helping read performance. This means that a write is cached and not written to disk immediately. When using Active Directory Domain Controllers on Azure Placement of the Windows Server AD DS database and SYSVOL states:

Unlike Operating System disk drives, data disk drives do not cache writes by default. Data disk drives that are attached to a VM use write-through caching. Write-through caching makes sure the write is committed to durable Windows Azure Storage before the transaction is complete from the perspective of the VM’s operating system. It provides durability, at the expense of slightly slower writes.

SQL has a similar requirement for write-through caching. The rtcxds database and log files are each created as 4 GB and it seems that creating the 8 GB of rtcxds files puts too much pressure on the write cached network storage.

So the solution is to attach a new empty disk with at least 10 GB (this is suitable for a test) to the Azure VM with no read or write cache

Initialise the disk, format it and give it a drive letter. Then install the databases from PowerShell, specifying the data disk:

[sourcecode language=”powershell”]

Install-CsDatabase -LocalDatabases -DatabasePaths F:\CsData

[/sourcecode]

Run step 2 from ‘Install or Update Lync Server System’ in the Lync Deployment Wizard again and it succeeds.

Summary

2012 R2 Preview Web Application Proxy fully supports publishing Lync 2013 and Office Web Apps Server 2013 as long as the default binding is set and the OWAS publishing rule internal and external URLs are the same. The following was tested successfully:

• Meeting join
• Dialin page
• Lync Web App
• Lync Web Scheduler
• Lyncdiscover
• Lync 2013 mobile client
• Lync Windows Store App (Lync MX)
• Office Web Apps upload and presentation
• Lync 2013 client Address book web search (tested with an Edge server outside of Azure)
• Lync 2013 client Address book group expansion (tested with an Edge server outside of Azure)

The Web Application Proxy looks very promising as a Reverse Proxy for the primary Microsoft applications I work with. SharePoint has not been tested yet, but that should work in either ADFS Preauthentication or Pass-through mode as long as the URLs match all the way through. I am looking forward to hopefully not having to use UAG again.