I had promised a couple of blogs on dealing with the challenges of distance that are unavoidable as we adopt a variety of dispersed Cloud deployed services. The first was using a WCF Custom Channel to cache SharePoint content which is now a bit old school. This is the second.

The rate of change at the moment is astonishing. I’ve been sitting on blog number two for quite some time, but when I go to build an example and type it up, something new has come along that is a slightly better way of doing it. But then that’s why we are in this business, right? So it is with much haste that I blog this while it is still relevant, (I think it still is) or at the very least it is buzz word (Web API, SignalR, Owin and Katana) compliant.

Something in one of the previous blog comments got me thinking.
“Thanks peter for nice blog. It helps allot. I am planning to write Routing service which can route both SOAP and OData requests. I took your solution and tried to debug. I am getting empty string while reading message before sending to client.”

Turns out that while WCF is designed to be transport agnostic, it’s hard to get it to be message format agnostic. What would be good is something that doesn’t get in the way but sits off to the side of the messaging but can be used to get in and affect messages, like adding authentication, caching or some other functionality. What I need is a forward proxy, not a router.

From the great book “Succinctly” book series: A proxy server is a computer that sits between a client and server. A proxy is mostly transparent to end users. You think you are sending HTTP request messages directly to a server, but the messages are actually going to a proxy. The proxy accepts HTTP request messages from a client and forwards the messages to the desired server. The proxy then takes the server response and forwards the response back to the client. Before forwarding these messages, the proxy can inspect the messages and potentially take some additional actions.

Which got me wondering, what is it that happens when you tick that proxy box in your browser? Is there something special about those servers we use to browse traffic through?

So I built a simple webserver, set my local IIS as the proxy server and with some config and fiddling the IIS web server was receiving traffic on IIS when I was trying to browse sites on the Internet. OK, so what? What if we capture that traffic in a web service and send it on and hand back the replies to just as a proxy would?

Web API

Web API is great simple technology for building simple APIs that serve the HTTP protocol. How hard would it be to build a simple server using Web API that proxies traffic straight through? Turns out it’s easy. Way, way easier that in WCF! All you need to do is build a Web API DelegatingHandler like this:

public class ProxyHandler : DelegatingHandler
{
  protected override async System.Threading.Tasks.Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
  {
    UriBuilder forwardUri = new UriBuilder(request.RequestUri);
    //strip off the proxy port and replace with an Http port
    forwardUri.Port = 80;
    //send it on to the requested URL
r
equest.RequestUri = forwardUri.Uri;
    HttpClient client = new HttpClient();
    var response = await client.SendAsync(request,HttpCompletionOption.ResponseHeadersRead);
    return response;
}
}

It’s worth taking a moment to understand what this code is doing. Every request that comes in on the proxy port (by convention I use port 8080) change it to port 80 and make the request then return the response. Simple. The other thing to note and the big change over my previous WCF version is the use of async. This enables the proxy to listen for new requests while previous requests are still pending responses, perfect for a proxy. Next is to configure the proxy:

config.Routes.MapHttpRoute(name: “Proxy”, routeTemplate: “{*path}”handler: HttpClientFactory.CreatePipeline
  (
    innerHandler: new HttpClientHandler(), // will never get here if proxy is doing its job
    handlers: new DelegatingHandler[]
    { new ProxyHandler() }
  ),
  defaults: new { path = RouteParameter.Optional },
  constraints: null
);

For all requests ({*path}) use the default HttpClientHandler but put the ProxyHandler in the pipeline before it. All requests are actually handled by the ProxyHandler and should never fall through to the inner handler. There are couple of additional tricks to get it working in all cases. You’ll need this for some reason that I haven’t yet figured out but I think it’s a bug in Web API.

 //have to explicitly null it to avoid protocol violation
 if (request.Method == HttpMethod.Get)
   request.Content = null;

You’ll also need this in your web.config if you are going to host it in IIS (more on hosting options later) to tell IIS that you want the proxy code to handle all requests, no matter what!

<system.webserver>
<!–needed if you want to run in IIS and handle everything!–>
<handlers>
 <remove name=ExtensionlessUrlHandler-Integrated-4.0 />
<remove name=OPTIONSVerbHandler />
<remove name=TRACEVerbHandler />
<add name=ExtensionlessUrlHandler-Integrated-4.0” path=*.” verb=*” type=System.Web.Handlers.TransferRequestHandler” preCondition=integratedMode,runtimeVersionv4.0 />
</handlers>
</system.webServer>

And, most importantly, this will tell your proxy not to use a proxy when making requests to the Internet (to avoid an endless loop of proxy requests)

<system.net>
<defaultProxy enabled=false />
</system.net>

Proxy done!

Hosting

The above code is hosting platform agnostic. This is one of the areas undergoing a lot of change in the industry right now. In what appears to be a new era of openness, open source and interoperability (even by Microsoft!) there are now a range of ways to host web services. Long story short .NET Web != IIS which is great. IIS is a very big, cumbersome engine if you don’t really need it. And this is a perfect example, as a simple forward proxy we don’t want or need any of that web serving functionality getting in the way, in fact we had to put special configuration into IIS to get it to leave our requests alone.

Since Web API is not tied to the web hosting framework in .NET, it supports self-hosting which is useful for running cheap API’s in Console apps or WorkerRoles. But now there is another option, Owin. Owin defines an open API for building web applications over a hosting platform by delivering well defined pluggable architecture. Katana is Microsoft’s implementation of OWIN. This article has a great explanation of how we got to such a complicated place with the pair of ASP.NET and IIS and how we are now finally winding it back out again now with Owin, Katana and various other supporting frameworks.

Like anything so new, this area is still in flux. The APIs are a bit green and changing and some simple modules (like Static File delivery) are still in beta. But it’s a great idea, and we can use it right away for our proxy with just a simple line of code we can spin up the proxy in a Console app anywhere.


public
static
void Start(string proxyAddress)
{
  try
  {
   // Start OWIN proxy host
   apps.Add(WebApp.Start<Proxy>(proxyAddress));
   Trace.TraceInformation(“Listening on:” + proxyAddress);
   Trace.WriteLine(“Set your IE proxy to:” + proxyAddress);
  }

The abstraction of Owin also enables us to just as easily deploy into Azure WorkerRole, again I prefer using a worker role so we don’t get all that annoying complexity and default modules that come with IIS and a Web Role.

foreach (var rie in RoleEnvironment.CurrentRoleInstance.InstanceEndpoints)
{
 var endpoint = rie.Value;
 if (endpoint.Protocol == “http”)
 {
  string baseUri = string.Format(“{0}://{1}”, endpoint.Protocol, endpoint.IPEndpoint);
  Trace.TraceInformation(String.Format(“Starting OWIN at {0}”, baseUri));
  Proxy.Start(baseUri);
 }
}
OK so now we have built a proxy server which can be used to channel all requests from your machine through the Web API proxy which is pretty powerful. We can deploy the proxy to Azure in US, UK or Asia and use it to browse sites unavailable in Australia. Try it out yourself by setting your IE proxy to busker.cloudapp.net port 8080. [The choice of the name “Busker” will become apparent in the next blog entry]

And try browsing www.google.com to see the results through the proxy (my US based proxy makes Google think I’m in Kendale Lakes Florida)

and direct ( that’s me with the with the proxy off getting Australian results)

But while that’s fun for tricking Google or for viewing NetFlix we could do the same with a publically available proxy or by deploying a commercial proxy server (TMG, Squid etc) into a virtual machine in the Cloud. The difference and power is in how we can inject custom behaviours using the WebAPI pipeline to get the proxy to affect the traffic.

Domain constrained handling

The idea of a proxy is to handle all traffic no matter what the domain. A web site or Web API is targeted at providing many urls under a single domain and it shows in the frameworks. They have great url mapping and filtering capability but largely ignore the domain. What if we want our proxy to behave differently for different domains? The answer is in this line:

constraints: null

We can add a custom constraint to ensure a handler is only invoked if the constraint is satisfied and in the constraint we can implement any sort of filtering we like.

constraints: new { isLocal = new HostConstraint { Host = “table.core.windows.net” } }

The HostConstraint class has a single method Match() which does something simple like:

public
bool Match(HttpRequestMessage request, IHttpRoute route, string parameterName, IDictionary<string, object> values, HttpRouteDirection routeDirection)
{ return request.RequestUri.Host.Contains(Host);}

So now I can make the proxy plug in different handlers and therefore behave differently for different domains!

Domain Specific Handlers

Previously I demonstrated using a WCF Router to modify the Azure Table API and make it a true oData interface which can be consumed by Excel. To do it we needed to create the Azure Table Storage authentication tokens and inject the oData paging elements over the vanilla table.core.net service. That same functionality has been ported to the proxy and plugged in as another handler which is constrained to just requests to the Azure Table Service (table.core.windows.net)

//now plug in some AzureAuth config handling using a Controller
//http://backtester.table.core.windows.net/config?connectionstring=DefaultEndpointsProtocol=https;AccountName=backtester;AccountKey=

config.Routes.MapHttpRoute(
  name: “ConfigAzureAuth”,
  routeTemplate: “config”,
  defaults: new { controller = “ConfigAzureAuth” },
  constraints: new { isLocal = new HostConstraint { Host = “table.core.windows.net” } }
);

Try setting the browser proxy to busker.cloudapp.net:8080 and use a browser or Excel to browse to http://backtester.table.core.windows.net/Quote(). This account holds a table with some stock quotes in it. Without the proxy in play this would fail because the AzureTableHandler injects the necessary authentication headers and paging over the straight Azure table API. Or submit your own storage keys to the Proxy to browse your own table storage in a browser or Excel using this config url


http://backtester.table.core.windows.net/config?connectionstring=DefaultEndpointsProtocol=https;AccountName=backtester;AccountKey=

Note here something a bit unusual, it looks like the Azure url http://backtester.table.core.windows.net supports a url path “config” but actually it doesn’t the Proxy picks up that url and handles it instead of sending on to Azure.

Client Specific Handlers

So now we can configure the Proxy to behave differently for different destinations, but what about different requesting clients? Yes we can, one (very simple) way is through ipaddress although we could also write domain specific cookies back to the user from the Proxy. I’ve used this ability to add some additional functionality using a handler in the pipeline if a user has registered for it. Try this (when your proxy is set to busker.cloudapp.net port 8080) http://www.buskerimage.com/flip or this http://www.buskerimage.com/bw. This website (www.buskerimage.com) doesn’t actually exist but the proxy is listening for that domain and using it to configure the proxy. In both cases the Proxy is remembering your ipaddress and manipulating the images on the way through either flipping them or going old school black and white (or both). Here’s what Bing Images looks like with both on.

Note only requests from and ip address registered for black and white and flipped images will see this view. OK so that’s all fun, what about the serious stuff? That’s in next blog when these building blocks are put to good use. But it’s fun to play with for now.

Running on Azure for now here: busker.cloudapp.net:8080

Available on github here: https://github.com/petermreid/buskerproxy

Category:
Application Development and Integration, Azure Platform
Tags:
, , , ,

Join the conversation! 13 Comments

  1. Great post! I can imagine using the proxy for service authentication when using oAuth between a SPA and an API. I have always wanted the clientid and key to be a true ‘secret’ key that is not present on the client side applications for anyone to hack into. If you did use this as that, how would you configure the proxy service as a host that comes between the SPA and the API for just the initial token access?

  2. Yes that should be fine. This is a generic HTTP proxy, doesn’t care whether its SOAP or REST

  3. Great Post!
    Im rewritting our main api that forwards requests to out internal microservices. I noticed that i had to handle a cors issue getting duplicate origins.
    But i wrote because i seem to be getting a weird 503 error when i deploy it to the server. Dont seem to have an issue locally. Just wondering if you have an idea on a cause. Im still debugging my way through.

  4. Thank you for providing this concise example.

    However, I got some very bizarre results when trying to use this in the real world. Web servers often use the Host header to determine which particular site you are trying to view on a box that is hosting several.

    Can I suggest an improvement?

    Set the Host header of the forwarded request to the Host of the website/service you are visiting. This will ensure that any web servers that are binding web sites to host names are still accessible.

    E.g.
    https://github.com/orangudan/HttpProxyHandler/blob/c32b0314c1ce7357b1702917029c7967db524d42/HttpProxyHandler/Models/HttpProxyHandler.cs

  5. Does this proxy work with SignalR? My SignalR clients always throw a Error 500 Internal Server error when I try to connect through this proxy. I’ve enabled tracing at the server and client side and cannot find where the problem is. I do not see any requests even hitting the proxy, but my REST calls come through from the same client just fine. Any help would be greatly appreciated.

    • I traced it down to this:
      https://192.168.1.110:4445/signalr/negotiate?clientProtocol=1.4&connectionData=%5B{“Name”:”ConnectItHub”}]
      if I omit the parameter for connectionData. I can paste this line in the address of Chrome to reproduce the error. It seems to fail because it might be trying to parse the JSON data. My actual server is running port 4443 and the proxy is running on 4445. This same line works fine when I change the port to 4443. This is not a SignalR issue per se since I can trigger the error in the browser even when the server is not running, still throws HTTP ERROR 500

  6. Ok, never mind. I needed the SignalR calls to be sent through to my SignalR server and this project sets it’s own SignalR server up. I removed the appBuilder.MapSignalR(); and all is working for me now.

  7. Hi Peter,

    Excellent article it has been really helpful and I have a working Proxy. Quick question, how would you handle cookies?

  8. I havn’t had any luck with this for https:// connections. Am I doing something wrong or is it just not supported?

  9. Not supported unfortunately
    I spent quite a bit of time trying to handle the low level HTTPS stream/connect messaging and gave up. You can use the FiddlerCore library

    • Thanks – yes I’ve fiddled around with it quite a bit and not been able to get it working. I got something rigged up using Fiddler, but wasn’t aware about FiddlerCore – looks promising.

      Still I have a number of none-https use cases where this will come in very handy, so thanks!

Comments are closed.