Using Microsoft Azure Table Service REST API to collect data samples

Sometimes we need a simple solution that requires collecting data from multiple sources. The sources of data can be IoT devices or systems working on different platforms and in different places. Traditionally, integrators start thinking about implementation of a custom centralised REST API with some database repository. This solution can take days to implement and test, it is very expensive and requires hosting, maintenance, and support. However, in many cases, it is not needed at all. This post introduces the idea that out-of-the-box Azure Tables REST API is good enough to start your data collection, research, and analysis in no time. Moreover, the suggested solution offers very convenient REST API that supports JSON objects and very flexible NoSQL format. Furthermore, what’s great is that you do not need to write lots of code and hire programmers. Anybody who understands how to work with REST API, create headers and put JSON in the Web request body can immediately start working on a project and sending data to a very cheap Azure Tables storage. Additional benefits of using Azure Tables are: native support in Microsoft Azure Machine Learning, other statistical packages also allow you to download data from Azure Tables.

Microsoft provides Azure Tables SDKs for various languages and platforms. By all means you should use these SDKs; your life will be much easier. However, some systems don’t have this luxury and require developers to work with Azure Tables REST API directly. The only requirement for your system is that it should be able to execute web requests, and you should be able to work with the headers of these requests. Most of the systems satisfy this requirement. In this post, I explain how to form web requests and work with the latest Azure Table REST API from your code. I’ve also created a reference code to support my findings. It is written in C# but this technique can be replicated to other languages and platforms.

The full source code is hosted on GitHub here:

https://github.com/dimkdimk/AzureTablesRestDemo

Prerequisites.

You will need an Azure subscription. Create there a new storage account and create a test table with the name “mytable” in it. Below is my table that I’ve created in the Visual Studio 2015.

Picture 1

I’ve created a helper class that has two main methods: RequestResource and InsertEntity.

The full source code of this class is here:

Testing the class is easy. I’ve created a console application, prepared a few data samples and called our Azure Tables helpers methods. The source code of this program is below.

The hardest part in calling Azure Tables Web API is creating encrypted signature string to form Authorise header. It can be a bit tricky and not very clear for beginners. Please take a look at the detailed documentation that describes how to sign a string for various Azure Storage services: https://msdn.microsoft.com/en-au/library/dd179428.aspx

To help you with the authorisation code, I’ve written an example of how it can be done in the class AzuretableHelper. Please take a look at the code that creates strAuthorization variable. First, you will need to form a string that contains your canonical resource name and current time in the specific format including newline characters in-between. Then, this string has to be encrypted with so-called HMAC SHA-256 encryption algorithm. The key for this encryption code is a SharedKey that you can obtain from your Azure Storage Account as shown here:

Picture 2

The encrypted Authorisation string has to be re-created on each request you execute. Otherwise, Azure API will reject your requests with the Unuthorised error message in the response.

The meaning of other request’s headers is straightforward. You specify the version, format, and other attributes. To see the full list of methods you can perform on Azure Tables and to read about all attributes and headers you can refer to MSDN documentation:

https://msdn.microsoft.com/library/dd179423.aspx

Once you’ve mastered this “geeky” method of using Azure Tables Web API, you can send your data straight to your Azure Tables without using any intermediate Web API facilities. Also, you can read Azure Tables data to receive configuration parameters or input commands from another applications. A similar approach can be applied to other Azure Storage API, for example, Azure Blob storage, or Azure Queue.

 

 

 

 

Cloud Strategy – do you need one?

Today’s organisations demand that their services perform better, cost less, are more reliable and offer greater flexibility than ever before. With high quality and quickly evolving consumer technologies, the expectation of IT services in the workplace is that they now reflect the same calibre of tools available to people outside of the office.

With common perceptions that business IT capabilities are too far behind, many departments look to acquire and provision cloud services themselves, without involving the IT organisation. These siloed initiatives complicate and create conflict with the responsibilities of IT such as governance, security, onboarding and offboarding, usage monitoring, corporate reporting, vendor management and purchasing and budgeting, just to name a few.

The result of this scenario is often inter-organisational struggles as teams demand to move faster while IT departments attempt to retain control to help manage operational risk. An important part of solving this problem to ensure these two parts of the business work together, is the creation of an effective cloud strategy that communicates supporting cloud policies.

Kloud consultants are assisting CIO’s and their teams to formulate the cloud computing component of their IT strategy with the engagement process outlined in the supporting brochure. Over the course of a few days to a few weeks, Kloud works closely with organisations to develop a roadmap that meets both business and staff motives.

The objective of our Cloud Ready Workshops are to ensure organisations are set on the right path with a clear understanding of how to manage technology costs more efficiently, deploy technologies faster, allowing management to focus on delivering maximum business value.

 

Do It Yourself Cloud Accelerator

This intro is unashamedly lifted from a Microsoft article but I couldn’t say it any better: “The cloud has enormous potential to reduce operational expenses and achieve new levels of scale, but moving workloads away from the people who depend on them can increase networking costs and hurt productivity. Users expect high performance and don’t care where their applications and data are hosted” Cloud is a journey, to get there takes more than just migrating your workloads to the cloud. In most organisations architecture, infrastructure and networking have grown around the assumption that business systems are internal and the internet is external. Big WAN connections to offices and well protected interfaces to the Internet is the default networking solution to that situation.

Who moved my Cheese?

So what happens when business systems move to the Internet? The need for branch offices to communicate with head office diminishes but the need to communicate efficiently with the Internet becomes more important. The paradox of this situation is employees are likely to be much better served by their own internet connection when accessing the cloud based business services than by a filtered, sanitised and latent connection provided by work. Cloud nirvana is reached when there are no internal business systems, WAN connections are replaced by managed devices and independent internet connections attached to employees. In fact a true end state for cloud and BYOD may be that device and connectivity are deferred to the employee rather than a service provided by the company.

Do It Yourself Cloud Accelerator

3 Stages of Cloud Enlightenment

So with that end state in mind its worth considering the situation most business find themselves in today. “Cloud services are upon us but we have a classic enterprise network”. They may have migrated to Office 365 for Internet and Email but still have a big centralised internet connection, heavy dependency on WAN links and treat the whole Internet, including the new migrated Email and SharePoint services as one big untrusted Internet. It is an interesting problem.  Cloud services (like Office 365) relieve the organisation of the concerns of delivering the service but that is matched equally by a loss of control and ownership which affects the way the service can be delivered to internal employees. On using the cloud service a secure tunnel is set up between the cloud service provider and the employee which disintermediates the employer who is paying the bills! The cloud service provider owns the ip addresses, the URL and the security certificate used to encrypt the traffic which is a significant change from the highly flexible on-premises model where the organisation controlled everything. The underlying problem here is that the SSL used to encrypt traffic between service provider and a single user is not flexible enough to support the new 3 way trust relationship that occurs when an organisation uses a cloud provider to deliver services to employees.

Take back the streets

All is not lost. Just because you’ve accepted cloud into your life doesn’t mean you have to totally submit to it. Cloud service providers do a great job using caching services such as Akamai to drop content right at the organisational door step, but the ability to efficiently transport that data through internal networks to the employee is now harder. In the following blog we’ll look at a strategy for taking back ownership of the traffic and deploying an efficient and cheap internal CDN network to serve content to branch offices with no direct internet connection. While the example is centered on Office 365 SharePoint Online, it is equally applicable to any cloud based service. There are commercial solutions to the problem by vendors like Riverbed and CISCO who will, by dropping an expensive device on your network, “accelerate” your Office 365 experience for internal employees. But can you do that with out-of-the-box software and commodity hardware? Yes you can. To build our own Cloud Accelerator we need the following:

  • 1 Windows Server (can be deployed internally or externally)
  • Access to makecert.exe through one of the Windows development kits
  • 1 Office 365 SharePoint Online Service (tenant.sharepoint.com) and an account to test with

Application Publisher

By using the Fiddler tool when browsing to your cloud application will reveal where the content is actually coming from. For our SharePoint Online site it looks something like this. https://gist.github.com/petermreid/8d2567b4703daff35ae2 Note that the content is actually coming from a mixture of the SharePoint Online server and a set of CDN URLs (cdn.sharepointonline.com and prod.msocdn.com). That is because Microsoft make heavy use of caching services like Akamai to offload much of the content serving and deliver it close to the organizational internet links as possible. The other thing to notice is the size. In fact the total size of the home page of our Intranet (without any caching) is a whopping 1MB which may be a significant burden on internet network links if not cached adequately across users. But to be able to do that we need to step in the middle to see the traffic and what can be cached on intermediate servers. Microsoft have a trusted certificate authority and can create secure certificates for *.sharepoint.com and *.sharepointonline.com which enables them to encrypt the traffic for your tenant’s Office 365 traffic. When you browse to Office 365 SharePoint Online from your browser an encrypted tunnel is set up between the browser and Microsoft but we can take back ownership of the traffic if we need to break the secure tunnel and republish the traffic using internal application publication.

Do It Yourself Cloud Accelerator App Publish

Application Publication

To do this we need to create our own certificate authority and deliver our own tenant.sharepoint.com and cdn.sharepointonline.com certificates and tell our users to trust those certificates by putting the new root certificate in their trusted root certificate store. This is no different to the trick that corporate proxy servers use to enable content filtering of your secure browser traffic. But here we don’t need to sniff all traffic, just the traffic that we already own coming from Office 365 which means we can pre create the certificates rather than generate them on the fly.

Trust Me

The first step is to create a new set of certificates that represent the trust between organization and employee. There’s a great article describing the process here but we’ll run through the steps here (replace the work tenant for your SharePoint Online tenant)

  • First find your copy of makecert.exe, mine is here C:\Program Files (x86)\Windows Kits\8.1\bin\x64.
  • Create a new certificate authority (or you can use one that your organisation already has). Mine will be called “Petes Root Authority” and it is from this root that we will build the new domain specific certificates.

Then use the certificate authority (stored in CA.pvk and CA.cer) to create a new certificate for tenant.sharepointonline.com

And one for the CDN domain cdn.sharepointonline.com https://gist.github.com/petermreid/4a2bee1b3b90dfef39d0

Following those instructions you should end up with a Certificate Authority and a server certificate for each of tenant.sharepoint.com and cdn.sharepointonline.com which we will install on our Office 365 Application Publisher. This is classic man-in-the-middle attack, where the man is the middle is you!

Application Publication

To publish the cloud service internally we will use a Windows Server and Internet Information Services (IIS) equipped with Application Request Routing (ARR). Normally this set up is used to publish content to the Internet but we will use it the other way around, to publish the Internet internally. The server can be anywhere, in the cloud or on-prem or even in a branch office but it just needs to have the opportunity to intercept our browsing traffic.

IIS Install

In the Server Manager->Local Server

  • IE Enhanced Security Configuration set to Off for Administrators
  • Manage->Add Roles and features->Role based->Add the Web Server(IIS) role.
  • Click through the “Features”
  • Click “Web Server Role”
  • Click on “Role Services”
  • Click Health and Diagnostics->Tracing (used to debug issues)

  • Click Install
  • Close the dialog when complete

Certificates

Install the certificates we created.

  • Install the CA into Trusted Root Certificate Authorities

  • And the server certificates into the WebHosting store by double clicking on each .pfx file,
  • choosing Store Location ->Local Machine and
  • Place all certificates in the following store ->Web Hosting

Click Start and type “Manage Computer Certificates”  and choose the Web Hosting branch to see the installed certificates

Web Platform Installer

  • Click Start and type IIS Admin and enter
  • Click on the machine name
  • Accept the dialog to install the Web Platform Installer and follow steps to download, run and install
  • Once installed, Click products and search for “Application Request Routing”
  • Then use the platform installer to install “Application Request Routing 2.5 with KB2589179”
  • Accept the Pre-requisites install and click Install and then Finish

  • Close IIS Admin and reopen using Start->IIS admin

Application Request Routing

  • Click on the top level server name and double click on “Application Request Routing Cache” icon
  • Under “Drive Management” select Add Drive and choose a new empty drive to store content in
  • Under “Server Proxy Settings” check the “Enable proxy” box
  • Clear the check box “Reverse rewrite host in respose headers” (we just want it to pass content right through without any modification)
  • and click “Apply”

Web Site

  • Click on the “Default Web Site” choose “Explore” and deploy the following web.config to that directory.
  • This config file contains all the rules required to configure Application Request Routing and URL Rewrite to forward the incoming requests to the site and to send the responses back to the requestor.
  • There is an additional filter (locked down to sharepoint and cdn) to ensure the router doesn’t become an “open proxy” which can be used to mask attacks on public web sites.

Site Binding

The default site will listen for HTTP requests on port :80, additional bindings are required to serve the https traffic.

  • In IIS Admin Click Bindings…Add Site Binding
  • Choose https and add bindings for all the  certificates we created earlier (make sure to select Require Server Name Indication and type the corresponding host name so IIS knows which certificate to use for which request)

URL Rewrite

Just one simple rule drives it all. All it does is takes any incoming request and re-requests it! (Note browsing direct to the website will cause an infinite loop of requests which IIS kindly breaks after about 20 requests) Double click the URL Rewrite icon in IIS to see the rule represented in the GUI…

Or the same as seen in web.config…

Network Configuration

DNS

So now all we have to do is change the mapping of the URL to point to our web server instead of Microsoft’s one. If this is an organisation with an internal DNS then it is easy to repoint the required entries. Otherwise, and for testing we can use a local DNS override, the “hosts” file (this file is located at [Windows]\System32\drivers\etc) by adding the following entries. ipaddress.of.your.apppublisher tenant.sharepoint.com ipaddress.of.your.apppublisher cdn.sharepointonline.com (Note if you’ve done this using an Azure server then the ip address will be the address of your “Cloud Service” ) Now when you browse to tenant.sharepoint.com the traffic is actually coming via our new Application Publisher web site but the content is the same! If you want to check if the content is actually coming through the router, have a look at the content headers (using Fiddler) since ARR inserts an additional header X-Powered-By: ARR/2.5

But remember that any user who intends to use this accelerator to receive content must have the root certificate installed in their “Trusted Root Certificate Authorities). By installing that root the user is acknowledging that they trust the acceleration server to intercept traffic on their behalf.

Accelerate

So, that’s an awful lot of effort to get the same content on the same URL. But there is something subtly different, we now “own” the traffic and we can introduce speed, bandwidth and latency improving strategies. Out of the box the Application Request Routing module we used implements caching which will honour the cache control headers on the requests to provide an intermediate shared cache across your users of the service. Simply click on the top level web site, choose Application Request Routing Cache, Add a Drive that it can use for caching and it will immediately start delivering content from the cache (where it can) instead of going back to the source thereby providing immediate performance improvements and bandwidth savings. Now we “Own” the traffic there are many other smarter ways to deliver the content to users especially where they are isolated in branch offices and low bandwidth locations. I’ll be investigating some of those in the next blog which will turn our internal content delivery network up to 11!

Are My Cloud Services Vulnerable to “Heartbleed”?

I have been receiving questions from a number of customers about the “Heartbleed” vulnerability that has been widely reported by the media.  Many customers are concerned as to whether they are at risk by using cloud services from Microsoft and other providers.  There a reasonable concern with any IT service when it comes to security.  Your provider should be able to answer simple questions about whether a service is vulnerable or not to Heartbleed and what steps are being taken to mitigate the risk.

Let’s take a moment to discuss what “Heartbleed” actually is.  It is a flaw in the OpenSSL encryption software library used by many websites to protect customers’ data. The vulnerability, known as “Heartbleed,” could potentially allow a cyberattacker to access a website’s customer data along with traffic encryption keys.

After a thorough investigation, Microsoft determined that Microsoft Azure, Office 365, Yammer and Skype, along with most Microsoft Services, are not impacted by the OpenSSL “Heartbleed” vulnerability. Windows’ implementation of SSL/TLS is also not impacted.

If you have any concerns about the security of a cloud service, please contact Kloud Solutions at the following URL:

http://www.kloud.com.au/contact-us/

Why You Should Move Your IT to the Cloud

​Cloud Computing is a revolutionizing the way IT is delivered.  Today, business of all sizes rely on IT to operate effectively.  IT is mission critical.  Unfortunately, very few enterprises can afford the operational costs required to deliver a highly available IT environment.

The Cloud is changing the economics of delivering IT.  Now, businesses of all sizes can subscribe to a highly reliable, elastic, cloud service for a fraction of the cost of running the infrastructure on premises.  The Cloud has transformed many complex IT services into commodity offerings with low monthly fees.  Using IT as a competitive advantage can allow an enterprise to become more agile and differentiate itself verses other competitors with bigger budgets.

To help illustrate the magnitude of this change, think back 100 years to the Industrial Revolution.  If you wanted to build a factory in the early 1900’s, the first thing you had to do was build a power plant.  Because your factory could not succeed without access to cheap, reliable electric power.  If you wanted to build a factory in the year 2013, you would be considered crazy to try to build a power plant to serve only your factory.  Why?  Because building one power plant for each factory is massively inefficient.  It means that every factory has to take on the high fixed costs of building and maintaining a power plant.  A more efficient way to deliver power to a large number of factories is to build one massive power plant in each local area.  This approach spreads the fixed costs across hundreds of factories which lowers the total cost of delivering power for everyone.  Each factory pays for power as a monthly utility cost.

The same type of transformation is occurring in IT.  The Cloud is transforming IT into a utility business.  It no longer makes sense for companies to build their own individual infrastructure to deliver the majority of IT services.  A more cost effective model is to pay for compute and storage as a utility cost.  This model will make IT more reliable and more affordable for consumers and companies of all sizes.  This is not a statement about the future of technology.  It is an observation about the reality that exists in the market today.

Kloud Solutions believes that the Cloud will fundamentally transform the IT industry.  We exist to help enterprises migrate mission critical IT services to the Cloud such as email, websites, voice, mobile applications, identity, security, integration, and management.  If you would like to discuss how Kloud Solutions can help your business, please contact us using the following URL:

http://www.kloud.com.au/contact-us/

 

Simulate moving to the Cloud with NEWT

I’ve blogged a bit in the past about the unique challenges encountered when moving to the cloud and the unavoidable consequence of introducing new network hops when moving workloads out of the data centre. I’m currently working for a unique organisation in the mining industry who are quite aggressively pursuing cost saving initiatives and have seen cloud as one of the potential savings. The uniqueness of the IT operating environment comes from the dispersed and challenging “branch offices” which may lie at the end of a long dedicated wired, microwave or satellite link

Centralising IT services to a data centre in Singapore is all very well if you’re office is on a well serviced broadband Internet link but what of these other data centres with more challenged connectivity. They have until now been well served by the Exchange server, file server or application server in a local rack and quite rightly want to know “what is the experience going to be like for my users if this server is at the other end of a long wire across the Pilbara.

Moving application workloads to best serve these challenging end user scenarios depends heavily on the application architecture. The mining industry, more so than other industries have a challenging mix of file based, database, and high end graphical applications that might not tolerate having tiers of the application spanning a challenging network link. To give the equivalent end user experience, do we move all tiers of the application and RDP the user interface, move the application server and deliver http, move just the database or files and leave the application on site?

Answering these questions is not easy, and depends on hard to measure qualitative user experience measures like scrolling, frame rate, opening times, report responsiveness and can only be determined once the workload has been moved out of the local data centre. What we need is a way to synthesise the application environment as if there are additional network challenges injected into the application stack without actually moving them.

What we need is a “Network Link Conditioner” available on Mac and UNIX environments but no Windows. Introducing NEWT or Network Emulator Toolkit for Windows. A product of the hardworking Microsoft Research team in Asia. And for such a powerful and important tool in demonstrating the effects of moving to cloud, it’s not so easy to find!

Installing

It’s been around for a while and there are a few odd web sites that still host copies of the application but the most official place I can find is in the new and oddly named Chocolatey gallery download site. Once you have Chocolatey installed it’s just a simple

So what can it do? It can help to simulate various network challenges by using a special, configurable network adaptor inserted into the network stack. After you install the application on a machine, try opening “Network and Sharing Centre” and then “Change adapter settings” and then right click “Properties” on one of your Network Adapters and you should see this.

A lap around NEWT

The documentation that comes with NEWT is actually very good so I won’t recover it here. There is particularly detailed information on the inner workings of the network pipeline and the effect of configuration options on it. Interestingly the NEWT user interface is just one of the interfaces as it also supports a .NET and COM interface also for building your own custom solutions.

Running

  • Add a Filter to one or All of your network adaptors and choose the ip addresses and or ports you want to affect
  • Add up and/or down Links and set the desired effect on bandwidth restriction, additional latency, packet loss and reordering.
  • Try Action->Toggle Trace, Push Play and have a look at the trace windows to see the packets being sniffed

To quickly check if it is working as expected try this ping google and note the latency (32mS) and ip address (74.125.237.209), set a Filter with 100mS of up & down link latency, Run NEWT and ping again

There’s now an additional 200mS (100 up and 100 down plus some NEWT filter overhead) added to Google home page. And you will notice it if you open a browser window to Google and hit Refresh (to refresh the browser cache). If you want to go back in time, try using the pre created filters like the “Dialup56K” setting! So enough of messing around what about the serious stuff.

So here are a few scenarios I’ve used it in recently:

Throttling Email

Why: When moving from an in house hosted exchange environment to Office 365 Email, email mail box synchronisation traffic must pass through corporate Internet facing networking infrastructure. Outlook (with the use of Cached Exchange mode) is actually extremely good at dealing with the additional latency, but some customers may be concerned about the slower email download rates now that Exchange is not on the same network (don’t know why since Email is inherently an asynchronous messaging system). To demonstrate the effect of slower downloads:

  • Download a file using the browser and note the download speed (this is to estimate the bandwidth available to an end user across the corporate Internet facing infrastructure)
  • Determine the ip address of the corporate exchange server and add a NEWT Filter for that ipaddress
  • Add a Downlink and limit the bandwidth to the previously noted download speed available to the browser (optionally also add the latency to the Office 365 data centre)
  • Send a large email to your Inbox

Migration to SharePoint Online

Why: When you have an existing Intranet served internally to internal users it can be difficult to describe what the effect will be of moving that workload to SharePoint Online. There are certainly differences but browsers are remarkably good at hiding latency and tools like OneDrive become more important. Demonstrating these effects to a customer to ease concerns about moving offshore is easy with NEWT.

  • Estimate the latency to the SharePoint Online data centre (Australia to Singapore is around 130mS each way)
  • Add a NEWT Filter for the internal Intranet portal
  • Add an Uplink and Downlink and add on your estimated Singapore latency
  • Browse and walk through the Intranet site showing where latency has an effect and the tools that have been built to get around it.

And remember for web applications like this your other favourite tool, Fiddler still as expected works as it operates at a different layer of the network stack.

Migrating an Application Database to the Cloud

Why: There are many cases where an application is deployed to local machines but communicate to a central database which may be worth considering moving to a cloud based database for better scalability, performance, backup and redundancy. Applications that have been built to depend on databases can vary significantly in their usage of the database. Databases are extremely good at serving needy applications which has encouraged inefficient development practices and chatty database usage. For this reason database bound applications can vary significantly in their ability to tolerate being separated from their database. NEWT can help synthesis that environment to determine if you’ve got an application that is tolerant or intolerant to a database move to the cloud.

  • Determine the connection string used by the application to connect to the database
  • Add a NEWT Filter for the ipaddress of the database on the database connection port (1433)
  • Inject as much latency as you need to simulate the distance between this desktop application and the intended location of the database.

Migrating an Application to the Cloud

Why: The question often comes up on lifting and shifting an entire application to the cloud along with all dependant tiers and present the application back to the user over RDP. Will an RDP interface be effective for this application?

  • Set up two machines, one with the application user interface (with remote desktop services enabled) and one with remote desktop
  • Inject the required latency using a NEWT filter on the application machine ipaddress and on the RDP port 3389.
  • Allow the user to operate the application using the synthesised latent RDP interface

There are many other possible uses, including synthesising the effects of packet loss and reordering on video streaming and its use in demonstrating the effect of network challenges on Lync or other VOIP deployments is invaluable. In a corporate IT environment where a cloud migration strategy is being considered NEWT is an invaluable tool to help demonstrate the effects of moving workloads, make the right decisions about those workloads and to help determine those interfaces between systems and people that have , and have not been designed and built to tolerate a network split across a large and unpredictable network.

Ideal Workloads for Cloud

Working in the fast changing world of IT, sometimes is good to stop, take a breather and reflect on where we are and how we got here. Cloud computing is certainly no exception and has had an enormous amount of hype over the past 5 years, but has it lived up to the promise?

Looking Back

I and many others at PDC 2008 listened to the announcement of Azure and the follow up communications as Microsoft went “all in” on the cloud. I particularly remember a set of slides from this presentation that found its way into thousands of other Microsoft sponsored presentations (including mine) around the planet.

The abridged message of this early marketing was based on the theory that the elastic nature of cloud should lend itself to a certain set of workloads that are uneconomic to serve on fixed infrastructure. Acronym heavy concepts like TCO, ROI and SLM are used to identify those workloads in your organisation and help build a business case to move them to the cloud.

Reality

So 5 years on where are we? Working on the frontline of cloud computing the reality is somewhat different and surprising. While the optimal workloads are certainly still the most economical to move off your on-premise infrastructure, cloud economics isn’t the most important factor, not yet anyway. Here’s my new 20-20 hindsight on that same slide, “Workload Patterns Practical for Cloud”.

Turns out the ideal workloads for Cloud are somewhat less ambitious and fall into one of the following categories:

IT Avoidance

Avoiding the internal IT department. We see a lot of this and it’s a conversation started by the business who just need to get stuff done but have been at the mercy of IT departments, project management prioritisation queues and bloated risk averse estimation processes. While all good IT departments should have systems in place to ensure they are serving the business (the money makers) in reality we are seeing a failing of that process and the resulting spill over of IT execution is being picked up by cloud based services outside of the control of IT departments. For example:

  • Business departments directly engaging services from SAAS providers
    (Social, Collaboration, Accounting, Blogs, Video hosting etc)
  • Reaching a dispersed and mobile workforce who haven’t been well served by traditional internal, IT departments with their firewalls and standard operating environments.

Predictable and Boring

While the original cloud sales pitch was all about quickly auto scaling your computations and web sites to solve new problems and serve huge numbers of customers, the reality is quite different. It is the plain vanilla workloads of Email , Intranet, Timesheets and Accounting with its fixed number of users and steady state traffic that is moving to cloud.

Why? It’s easy, the migration path is well trodden and quantifiable. The CIO who needs to reduce CapEx on the balance sheet can fund a large and risky migration of a core line of business system or alternatively take a fixed price migration, remove a rack of servers and a couple of IT head count in the deal. This really makes sense when you think about cloud computing as a utility like electricity. Organisations should be migrating out to cloud services those workloads that are not core business. This is just the beginning of an IT commoditisation process, the end result of which is organisations not in the business of making money from IT shouldn’t have an IT department nor a server room just as they don’t generate their own electricity.

That will take time, but evidentally it will start with the boring workloads first.

Dev & Test

At Kloud we moved to a new office space with a dedicated air conditioned server room which now looks like this. The only cold server containment is the beer fridge and the only server is the xBox (actually there is one server left but its life is likely to be short lived).

We arguably are a company that makes money from IT and we do indeed have lots of development, test and build servers but they all run elsewhere and only when needed for an individual client or project. The short lived environments are used to replicate or fake everything needed to build a client solution and just as quickly removed.

It has become the norm for organisations to deploy much larger server footprint to support multiple independent development and test streams than that deployed to run production environments. These server farms are typically underutilised and run much longer than they need to.

While not glamorous nor business critical, these environments represent a large proportion of the corporate data centre and are an obvious starting point for moving workloads to the cloud. And since by definition these environments are meant to be isolated, there is no better isolation than the other side of the world in someone elses data centre.

Disaster and Backup

The final workload pattern that is blazing the cloud trail is disaster and backup. Hosting a replica of the corporate production environment and backing up the data to a remote data centre capable of being switched to in the event of disaster. This is a great strategy for that CIO who has until now had to support 2 data centres >50 kms apart for the 1 day in 4 years when a backhoe cuts the Internet link.

This is potentially the most interesting of all the cloud adoption strategies since the disciplines, processes and deployment challenges of running a DR site in the cloud requires the same disciplines as those to run production. Cloud vendors know this and are encouraging the use of cloud platforms for DR and backup strategies and cloud backed storage solutions.

Looking Forward

So what now moving forward? Arguably we are at a tipping point for cloud I think for the following reasons:

  • Hardware refresh cycle: Most hardware accounting factors in a 3-4 year hardware refresh cycle. Hardware that was purchased before cloud was a viable option are now up for renewal. We are now realistically at a point where a CIO with vision could quite justifyably say “we are buying no more hardware” (which is what startups have been doing for years)
  • Money wins most arguments: The formative years of cloud have been dominated by one persistent question; Security! But the rate of uptake of corporate Email and Intranet compared to other workloads suggests that in the mind of the CIO the prospect of fixed cost and low risk migration trumps the risk of hosting your sensitive internal data in the public cloud.
  • Moats are not a great defence: Traditional IT Security principles have followed the Middle Ages mindset of putting a ring of confidence around the castle where all the good people are on the inside and all the bad people safely on the outside. The adoption of SaaS as a service provider outside and BYOD mobility as a consumer have forced a challenge to that mindset and an acknowledgement by many IT professionals that the security provided by a top tier cloud provider is most likely better than something you can knock together under the stairs at work.
  • Identity first: One of the great side effects of the adoption of “boring” workloads is that even boring applications need logins and ideally you want those logins to be the same regardless of where the application runs. While the ultimate solution “Federation” appears to still be an afterthought for most SaaS provders the next best thing is identity management and provisioning to manage the potential explosion of identities. This valuable initiative being undertaken by organisations today is paving the way for much more flexible “apps without borders” model tomorrow.

So Cloud adoption happened but in a different way than would be predicted purely on economics. But it happened. And the formative work is paving the way for a new generation of agile, competitive organisations relieved of the burden of IT.

Latency, Caching and Cloud

Q: What is the one unavoidable issue when moving services to the cloud?

Security? Reliability? Ownership? Performance? Nope

A: Latency!

Physics 101

Latency is the amount of time required to communicate between one point and another and is limited by the speed of light. Using high school physics and geography:

Circumference of planet: 40,000km
Longest distance point to point: 20,000km
Speed of Light: 300,000 km/s
Send time: 66mS
Ping time (send and reply): 133mS
That is the theoretical minimum for sending a message to the other side of the planet and getting an answer. This can be confirmed by using the network ping tool available on most operating systems by pinging a server on the other side of the planet.

Pinging http://www.yammer.com [204.152.18.206] with 32 bytes of data:
Reply from 204.152.18.206: bytes=32 time=222ms TTL=241

But that says 222! In reality, the story is a little more complicated as there are many intermediate hops and decisions to be made along the way which can be revealed using a command like the following which reveals some 13 (for me) hops on the way:

tracert www.yammer.com
1 1 ms 1 ms 2 ms 192.168.2.1
2 19 ms 17 ms 17 ms nexthop.vic.iinet.net.au [203.215.7.251]
3 18 ms 17 ms 17 ms te7-2.mel-pipe-bdr1.iinet.net.au [203.215.6.10]
4 27 ms 28 ms 29 ms xe-7-1-11-0.syd-ult-core1.on.ii.net [203.215.20.102]
5 105 ms 26 ms 27 ms unknown.telstraglobal.net [134.159.126.189]
6 31 ms 31 ms 32 ms i-0-2-4-0.sydo-core01.bi.telstraglobal.net [202.84.223.5]
7 189 ms 188 ms 187 ms i-0-4-0-1.paix-core01.bx.telstraglobal.net [202.84.249.57]
8 187 ms 186 ms 188 ms i-0-0-0-5.paix02.bi.telstraglobal.net [202.84.251.18]
9 184 ms 252 ms 185 ms ge4-0.mpr2.pao1.us.mfnx.net [198.32.176.11]
10 186 ms 229 ms 207 ms xe-1-0-0.mpr2.pao1.us.above.net [64.125.31.62]
11 186 ms 186 ms 186 ms 64.125.195.94.t00969-02.above.net [64.125.195.94]
12 195 ms 186 ms 185 ms 204.152.18.38
13 187 ms 188 ms 187 ms 204.152.18.206

The effect of this time lag on end users is normally much worse than a few mS because chatty solutions that worked well on the internal infrastructure are stretched across a latent link multiplying the lag effect. So what strategies can be used to minimise the impact of latency? Essentially all solutions involve some means of bringing the content closer to the end user and putting in place a mechanism for keeping it up to date. That is otherwise known as “caching”.

Everybody wants the all the latest data immediately, however when dealing with a global networked architecture as we do with Cloud, that is not possible.

Brewers CAP Theorem

The above is an example of Brewers CAP theorem at play.

To cut a long story short, we need to strike a trade-off between which of Consistency, Availability or Partitioning and decide which is going to be sacrificed so we can scale a system globally.

  • “Partitioning”: tradeoff-Break up the data set and keep local data local.
  • “Availability”: tradeoff-Responses may take longer or outage windows experienced.
  • “Consistency”: tradeoff-Sometimes the data you get won’t be exactly right.

Turns out the last strategy, although it sounds dangerous is happening all the time, from your browser to your proxy server to the ISP provider, to Akamai (or similar) the web server and beyond. Our view of the Internet is blurred as we are always browsing with our cache-coloured glasses that presents a view time-shifted from the one at the source. As users our expectation of performance and immediacy has overruled correctness.

We can see the result when you choose items in Amazon, drop them in your shopping cart only to find they are “on back order” once at the checkout. When you put an item in the shopping basket, you’re not taking a lock on your item and decrementing a great big global product counter. You’re viewing a delayed estimation of what is actually available in the warehouse.

Q: So what’s all this got to do with Cloud?

A: When you choose to move to Cloud you’re making a trade-off between the cost of providing the service and one or more of the “CAPs”.

And that’s where caching comes in. It is very good at giving the illusion of Availability by quietly delivering Inconsistency.

Caching

Caching is happening all the time without you even being aware of it.

A hidden but very effective cache is the one that was quietly deployed between Outlook and Exchange. Starting life as an option for laptop users who wanted to do emails while disconnected on the plane has become the default for all Outlook users regardless of connectivity. When was the last time you saw “server not available” error in Outlook stopping you from opening or answering any emails? This is also the same strategy (and for the same purpose) which has emerged underpinning SharePoint Workspace and now SkyDrive Pro.

Another cache is quietly doing its work every time you browse to a site. The requesting URLs and the resulting responses are being saved away in your browser cache, in your proxy server, at your Internet Service Provider and beyond for next time you visit the site.

Those two examples are examples of two broad way of solving the caching problem:

Distributed Query Engine

The first (Outlook, SkyDrive Pro) is to use a single source of data with partitions of that data replicated to distribute the data and query engine to remote locations. This architecture hides the effect of network latency for consuming users and applications and provides a useful scalability point.

This model can be broadly classed as a distributed query engine. That is, not only is the data replicated to the regions, but the entire query engine too (Outlook mail sorting, filtering and searching). The benefit here is the clients can continue to operate off line with a full copy of the (albeit old) data. The problem is it requires access to the original data source for replication.

Cached Query and Results

The other model for solving the distribution problem is to use a single data source and distributing the content as stored query and results. That is; query for and then cache the results into local caches. The benefit is, only content that is actually required at a client is cached and it can be implemented at a higher level of abstraction such as web services.

Web Services

Looking at web services in general is it possible to implement a useful cache generically over a variety of data sources? Is it possible to find points into which we can inject and manage some caching delay (Inconsistency) to achieve a scalable, performant, available globally deployed system?

As it turns out, the guys who built the “World Wide Web” invented a highly cacheable protocol called HTTP. A lot of effort has gone into making HTTP cacheable by providing ways to express how much tolerance a client has for old content and how long the server recommends it should be stored. HTTP actually defines a number of different verbs that were created to express different the different actions that could be performed over HTTP.

OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT and MERGE

Each has different verb has different cacheability characteristics. Actions like POST are considered to be an action that changes data (along with PUT and DELETE) and therefore is not to be cached (look at 13.10 here) while GET has the full suite of cacheability options available.

SOAP Dupe

However the meaning, purpose and usefulness of those verbs has become blurred overtime as new higher level protocols like SOAP/WCF have piggy backed solely on the HTTP POST verb as it provided better upstream data support but it has been used to implement updating and retrieving of data.

The side effect of that decision is that it’s very hard to cache web service calls because they use POST under the hood. Most software, frameworks and proxy devices all throw their hands up when it comes to caching of POSTs. So if you want to do it you need to either go below the HTTP protocol (think WAN acceleration) or add something on top. (I’ll implement one of these in a future blog).

The Rise of REST

This is why, in the era of Cloud, REST and higher level protocols like oData have become so popular. They are a return to the world of HTTP verb specific cacheability for web services. The cost is you need to cram all your request data on the URL!

https://klouddev.sharepoint.com/_api/web/lists/getbytitle(‘Documents’)/items(14)/File

Cloud Centralisation

Moving to Cloud involves outsourcing your IT infrastructure to someone and somewhere else. The implications of that can be far reaching. One that is unavoidable is that the economics of computing will force large clumps of computing resources to be located together, close to natural resources like hydroelectric schemes or in naturally cold places where cooling is free.

Q: What does that mean for us as developers?

A: Expect your infrastructure and your users to be separated by distance.

The App That Got Spat

Additionally we are now seeing economics driving cloud services to be provided “multi-tenant”. Which means you and potentially many other users and organisations are sharing the same application, servers and security space. To enable app customisation to work securely so that tenants don’t accidentally (or maliciously) tread on each others toes, a new app development model is emerging where the custom app runs away from the cloud data and services. Either as a separate site owned by the tenant or in the user’s browser or both. For SharePoint Online this is the new “auto-hosted” and “provider-hosted” app models.

Q: What does that mean for us as developers?

A: Expect your application and your data sources to be separated by distance.

The traditional 3 tier app model just got extended to at least 4 tiers, Data, Service, AppHosting and Browser. The effect is injecting further distance and hops between data and users and putting more reliance on the design of good web service APIs and the ability to call them efficiently and effectively.

What Now?

So caching will become more and more important to app developers working in the Cloud space. To continue to give users the experience they expect but from a more complex and challenging operating environment and geographically dispersed architecture. Over the next two blogs we’ll visit some options for delivering cacheable services over SharePoint which may be extended to other similar SAAS app development models.

Cloud Revolution – The Blind Entrepreneur

[Following from Cloud Revolution]

Profitable since the first quarter, 375 million page views per month, $4 million in annual revenue, 75 employees and $30 million in venture funding. Certainly sounds like a successful business, and it is.

I Can Has Cheezburger has leveraged the initial success of that hit web site by launching hundreds of other similar but slightly different sites based on the original. Many of these sites fail to attract traffic and are shut down within weeks, but every now and then one works, like “Fail Blog” or “Totally looks like“. With many copy cat sites launching web sites daily, chasing the latest “meme” and clipping the ticket on advertising for every one of your page views there’s plenty of motivation to continue.

Traditional innovation business models involve coming up with a great idea, getting seed capital, driving the business for a year only to watch it fail and leave the inventor bankrupt for his troubles. CEO Ben Hu is a new breed of trial-and-error innovator powered by the “scale fast” and “fail fast” business model enabled by the Cloud. By constantly trying out new variants of sites and seeing what works, and more importantly what doesn’t, he is tightening up and lowering the cost of the innovation cycle.

Could a cringe-worthy blog on cat pictures really change the world? Quite possibly, yes. Mr Hu is disappointed with the way newspapers have been presenting news online and is using all the lessons learnt in publishing cat pictures along with his new $30M of seed capital to reinvent the world of online journalism. He will no doubt try many new ways to present news, most will fail, but eventually something is going to stick and we could then be looking at the next Rupert Murdoch.

We know from Richard Dawkins “Blind Watchmaker” that many small incremental changes combined with a process of selection will eventually turn an amoeba to a human. When applied to the Internet and powered by the pay as you go model of the Cloud, the rate of change can be dialled up to 11 and the consequence of failure is near zero. The result will be revolutionary and highly effective ways of presenting information over the Internet for the benefit of humanity and especially Mr Hu!

Cloud Revolution – The Late Night Scientist

[Following from Cloud Revolution]

It’s late at night in January and an email just arrived that has someone very excited. A medical student has won an auction, but this is no eBay auction, it’s an Amazon EC2 instance auction. She has been preparing for this moment for over a year and will make or break her post-doctoral research paper.

She is continuing the valuable work done by her predecessors in the field of on Parkinson’s research and its relationship to dopamine levels in the brain. Previous studies were painstakingly conducted by carefully tracking, monitoring and documenting the lives of hundreds of thousands of patients presenting with dopamine affecting afflictions such as methamphetamine addictions over the course of 10-15 years.

Our scientist is hoping to show a relationship to other causes of dopamine depletion and to do it she’s has been scouring the huge catalogue of free and paid for databases from the likes of the WHO and Science Direct available in the Cloud. Bringing together datasets as wide as meteorology, geography, drug addiction, depression and death rates she has amassed 20Tb of data that needs to be processed to find correlations. Trial runs on small subsets have shown that it will take around 30,000 hours for one CPU to get through the full set. She can’t afford to wait the 3 years to run on her own machine, and limited by a $5,000 grant she can’t buy the hardware needed to do it herself. Instead she puts in a bid for 5000 Amazon EC2 Spot instances at 4c/hour.

Humming in the darkness under a football field sized roof in Ashburn Virginia are thousands of computers running Amazon.com, Instagram, Reddit, Quora and Foursquare. But now it is late in the evening, many shoppers and users are asleep, post Christmas sales are over and the GFC have all conspired to bring Internet traffic to a record low. One by one computers are being freed up into a pool until there are 5000 available and the deal is struck. 6 hours and $1200 later and she has her answer.

This is an example of the new paradigm of data-intensive scientific discovery and it’s happening right now. Effectively time-shifting 16 years of research into 6 hours of processing by utilising data and computing power that already exists. While many organizations are battling with the concept of how to secure their data in the cloud, others have seen the opportunity and make their data freely available or as a chargeable service.

There are many problems that don’t need to be solved now, or even today. In fact some problems have remained unsolved for years but may now tackled using enormous amounts of otherwise idle computing capacity at prices previously unheard of to scientists. This new tool of human evolution will be used to map the neurons in the brain, solve the riddle of Parkinson’s disease, chip away at the list of cancers and discover unexpected relationships and correlations across massive datasets of medical information.

And it’s all available to anyone with a hunch or a hypothesis they want to test.