Putting SQL to REST with Azure Data Factory

Microsoft’s integration stack has slowly matured over the past years, and we’re on the verge of finally breaking away from BizTalk Server, or are we? In this article I’m going to explore Azure Data Factory (ADF). Rather than showing the usual out of the box demo I’m going to demonstrate a real-world scenario that I recently encountered at one of Kloud’s customers.

ADF is a very easy to use and cost-effective solution for simple integration scenarios that can be best described as ETL in the ‘old world’. ADF can run at large scale, and has a series of connectors to load data from a data source, apply a simple mapping and load the transformed data into a target destination.

ADF is limited in terms of standard connectors, and (currently) has no functionality to send data to HTTP/RESTful endpoints. Data can be sourced from HTTP endpoints, but in this case, we’re going to read data from a SQL server and write it to a HTTP endpoint.

Unfortunately ADF tooling isn’t available in VS2017 yet, but you can download the Microsoft Azure DataFactory Tools for Visual Studio 2015 here. Next we’ll use the extremely useful 3rd party library ‘Azure.DataFactory.LocalEnvironment’ that can be found on GitHub. This library allows you to debug ADF projects locally, and eases deployment by generating ARM templates. The easiest way to get started is to open the sample solution, and modify accordingly.

You’ll also need to setup an Azure Batch account and storage account according to Microsoft documentation. Azure Batch is running your execution host engine, which effectively runs your custom activities on one or more VMs in a pool of nodes. The storage account will be used to deploy your custom activity, and is also used for ADF logging purposes. We’ll also create a SQL Azure AdventureWorksLT database to read some data from.

Using the VS templates we’ll create the following artefacts:

  • AzureSqlLinkedService (AzureSqlLinkedService1.json)
    This is the linked service that connects the source with the pipeline, and contains the connection string to connect to our AdventureWorksLT database.
  • WebLinkedService (WebLinkedService1.json)
    This is the linked service that connects to the target pipeline. ADF doesn’t support this type as an output service, so we only use it to refer to from our HTTP table so it passes schema validation.
  • AzureSqlTableLocation (AzureSqlTableLocation1.json)
    This contains the table definition of the Azure SQL source table.
  • HttpTableLocation (HttpTableLocation1.json)
    T
    he tooling doesn’t contain a specific template for Http tables, but we can manually tweak any table template to represent our target (JSON) structure.

AzureSqlLinkedService

AzureSqlTable

Furthermore, we’ll adjust the DataDownloaderSamplePipeline.json to use the input and output tables that are defined above. We’ll also set our schedule and add a custom property to define a column mapping that allows us to map between input columns and output fields.

The grunt of the solution is performed in the DataDownloaderActivity class, where custom .NET code ‘wires together’ the input and output data sources and performs the actual copying of data. The class uses a SqlDataReader to read records, and copies them in chunks as JSON to our target HTTP service. For demonstration purposes I am using the Request Bin service to verify that the output data made its way to the target destination.

We can deploy our solution via PowerShell, or the Visual Studio 2015 tooling if preferred:

NewADF

After deployment we can see the data factory appearing in the portal, and use the monitoring feature to see our copy tasks spinning up according to the defined schedule:

ADF Output

In the Request Bin that I created I can see the output batches appearing one at a time:

RequestBinOutput

As you might notice it’s not all that straightforward to compose and deploy a custom activity, and having to rely on Azure Batch can incur significant cost unless you adopt the right auto scaling strategy. Although the solution requires us to write code and implement our connectivity logic ourselves, we are able to leverage some nice platform features as a reliable execution host, retry logic, scaling, logging and monitoring that are all accessible through the Azure portal.

The complete source code can be found here. The below gists show the various ADF artefacts and the custom .NET activity.

The custom activity C# code:

Moving resources between Azure Resource Groups

The concept of resource groups has been around for a little while, and is adequately supported in the Azure preview portal. Resource groups are logical containers that allow you to group individual resources such as virtual machines, storage accounts, websites and databases so they can be managed together. They give a much clearer picture to what resources belong together, and can also give visibility into consumption/spending in a grouped matter.

However, when resources are created in the classic Azure portal (e.g. virtual machines, storage accounts, etc.) there is no support for resource group management, which results in a new resource group being created for each resource that you create. This can lead to a large number of resource groups that are unclear and tedious to manage. Also, if you do tend to use resource groups in the Azure preview portal there is no way to perform housekeeping or management of these resource groups.

With the latest Azure PowerShell cmdlets (v0.8.15.1) we now have the ability to move resources between resource groups. You can install the latest version of the PowerShell tools via the Web Platform Installer:

wpi azure powershell

After installation of this particular version we now have the following PowerShell commands available that will assist us in moving resources:

  • New-AzureResourceGroup
  • Move-AzureResource
  • Remove-AzureResourceGroup
  • Get-AzureResource
  • Get-AzureResourceGroup
  • Get-AzureResourceLog
  • Get-AzureResourceGroupLog

Switch-AzureMode AzureResourceManager

After launching a Microsoft Azure Powershell console we need to switch to Azure Resource Manager mode in order to manage our resource groups:

Switch-AzureMode AzureResourceManager

Get-AzureResourceGroup

Without any parameters this cmdlet gives a complete list of all resource groups that are deployed in your current subscription:

When resources are created in the classic Azure portal they will appear with a new resource group name that corresponds to the name of the object that was created (e.g. virtual machine name, storage account name, website name, etc.).

Note that we have a few default resource groups for storage, SQL and some specific resource groups corresponding to virtual machines. These were automatically created when I built some virtual machines and a Azure SQL server database in the classic Azure portal.

New-AzureResourceGroup

In order to group our existing resources we’re going to create a new resource group. It’s important to note that resource groups reside in a particular region which needs to be specified upon creation:

You’d think that resources can only be moved across resource groups that reside in the same region. However, I’ve successfully moved resources between resource groups that reside in different regions. This doesn’t affect the actual location of the resource so I’m not sure what the exact purpose of specifying a location for a resource group is.

Get-AzureResourceGroup

The Get-AzureResourceGroup cmdlet allows you to view all resources within a group, including their respective types and IDs:

Move-AzureResourceGroup

To move resources from the existing resource groups we need to provide the Move-ResourceGroup cmdlet a list of resource IDs. The cmdlet accepts the resource ID(s) as pipeline input parameters, so we can use the Get-AzureResource cmdlet to feed the list of resource IDs. The following script moves a cloud service, virtual machine and storage account (all residing in the same region) to the newly created resource group:

The Get-AzureResource cmdlet allows you to further filter based on resource type, or individual resource name. The Move-ResourceGroup cmdlet automatically removed the original resource group in case there are no resources associated after moving them.

Unfortunately at the time of writing there was an issue with moving SQL database servers and databases to other resource groups:

Trying to move the SQL server only does not raise any errors, but doesn’t result in the desired target state and leaves the SQL server and database in the original resource group:

The cmetlets Get-AzureResourceLog and Get-AzureResourceGroupLog provide a log of all the performed operations on resources and resource groups, but couldn’t provide any further information regarding the failure to move resources to the new group.

Now we have successfully moved our virtual machine and storage account to the new resource group we can get insight into these resources through the resource group:

Resource Group

Command and control with Arduino, Windows Phone and Azure Mobile Services

In most of our posts on the topic of IoT to date we’ve focussed on how to send data emitted from sensors and devices to centralised platforms where we can further process and analyse this data. In this post we’re going to have a look at how we can reverse this model and control our ‘things’ remotely by utilising cloud services. I’m going to demonstrate how to remotely control a light emitting diode (LED) strip with a Windows Phone using Microsoft Azure Mobile Services.

To control the RGB LED strip I’m going to use an Arduino Uno, a breadboard and some MOSFETs (a type of transistor). The LED strip will require more power than the Arduino can supply, so I’m using a 9V battery as a power supply which needs to be separated from the Arduino power circuit, hence why we’re using MOSFET transistors to switch the LEDs on and off.

The Arduino Uno will control the colour of the light by controlling three MOSFETs – one each for the red, blue and green LEDs. The limited programmability of the Arduino Uno means we can’t establish an Azure Service Bus relay connection, or use Azure Service Bus queues. Luckily Azure Mobile Services allow us to retrieve data via plain HTTP.

A Windows Phone App will control the colour of the lights by sending data to the mobile service. Subsequently the Arduino Uno can retrieve this data from the service to control the colour by using a technique called ‘pulse width modulation‘ on the red, green and blue LEDs. Pulse width modulation allows us to adjust the brightness of the LEDs by quickly turning on and off a particular LED colour, thus artificially creating a unique colour spectrum.

For the purpose of this example we won’t incorporate any authentication in our application, though you can easily enforce authentication for your Mobile Service with a Microsoft Account by following these two guides:

A diagram showing our overall implementation is shown below.

Command and Control diagram

Mobile service

We will first start by creating an Azure Mobile Service in the Azure portal and for the purpose of this demonstration we can use the service’s free tier which provides data storage up to 20MB per subscription.

Navigate to the Azure portal and create a new service:

Creating a Mobile Service 1

Next, choose a name for your Mobile Service, your database tier and geographic location. We’ll choose a Javascript backend for simplicity in this instance.

Creating a Mobile Service 2

Creating a Mobile Service 3

In this example we’ll create a table ‘sensordata’ with the following permissions:

Mobile Service Permissions

These permissions allows us to insert records from our Windows Phone app with the application key, and have the Arduino Uno retrieve data without any security. We could make the insertion of new data secure by demanding authentication from our Windows Phone device without too much effort, but for the purpose of this demo we stick to this very basic form of protection.

In the next section we’re going to create a Windows Phone application to send commands to our mobile service.

Windows Phone Application

To control the colour in a user friendly way we will use a colour picker control from the Windows Phone Toolkit, which can be installed as a NuGet package. This toolkit is not compatible with Windows Phone 8.1 yet, so we’ll create a Windows Phone Silverlight project and target the Windows Phone 8.0 platform as shown below.

Visual Studio Create Project 1

Visual Studio Create Project 2

Next, we’ll install the ‘Windows Phone Toolkit’ NuGet package as well as the mobile services NuGet package:

Install Windows Phone Toolkit Nuget

Install Mobile Services NuGet

For the purpose of this demo we won’t go through all the colour picker code in detail here. Excellent guidance on how to use the colour picker can be found at on the Microsoft Mobile Developer Wiki.

The code that sends the selected colour to our mobile service table is as follows.

The event data consists of colour data in the RGB model, separated by semicolons.

The complete working solution can be found in this Github repository. Make sure you point it to the right Azure Mobile Service and change the Application Key before you use it!

Run the application and pick a colour on the phone as shown below.

Phone ScreenShot

Now that we have a remote control that is sending out data to the Mobile Service it’s time to look at how we can use this information to control our LED strip.

Arduino sketch

In order to receive commands from the Windows Phone App we are going to use OData queries to retrieve the last inserted record from the Azure Mobile Servicewhihch exposes table data via OData out of the box. We can easily get the last inserted record in JSON format via a HTTP GET request to a URL similar to the following:

https://myiotservice.azure-mobile.net/tables/sensordata?$top=1&$orderby=__createdAt%20desc

When we send a HTTP GET request, the following HTTP body will be returned:

[
  {
    "id":"A086CE3F-5FD3-45B6-A967-E0928E3C5A96",
    "DeviceId":"PhoneEmulator",
    "SensorId":"ColorPicker",
    "EventType":"RGB",
    "EventData":"0;0;255"
  }
]

Notice how the colour is set to blue in the RGB data.

The Arduino schematics for the solution:

Arduino Command Control Schematic

For illustrative purposes I’ve drawn a single LED. In reality I’m using a LED strip that needs more voltage than the Arduino can supply, hence the 9V battery is attached and MOSFET transistors are used. Don’t attach a 9V battery to a single LED or it will have a very short life…

The complete Arduino sketch:

When we run the sketch the JSON data will be retrieved, and the colour of the LED strip set to blue:

The Working Prototype!

In this article I’ve demonstrated how to control a low end IoT device that does not have any HTTPS/TLS capabilities. This scenario is far from perfect, and ideally we want to take different security measures to prevent unauthorised access to our IoT devices and transport data. In a future article I will showcase how we can resolve these issues by using a much more powerful device than the Arduino Uno with an even smaller form factor: the Intel Edison. Stay tuned!

Microsoft Windows IoT and the Intel Galileo

You might have seen one of these headlines a while back: ‘Microsoft Windows now running on Intel Galileo development board’, ‘Microsoft giving away free Windows 8.1 for IoT developers’. Now before we all get too excited, let’s have a closer look beyond these headlines and see what we’re actually getting!

Intel Galileo

With a zillion devices being connected to the Internet by the year 2020 a lot of hardware manufacturers want to have a piece of this big pie, and Intel got into the game by releasing two different development boards / processors: the Intel Galileo and more recently the Intel Edison.

Intel Galileo

Intel Galileo

Intel Edison

Intel Edison

The Galileo is Intel’s first attempt to break into consumer prototyping, or the ‘maker scene’. The board comes in two flavours, Gen 1 and Gen 2 with the latter being a slightly upgraded model of the first release.

Like many other development platforms the board offers hardware and pin compatibility with a range of Arduino shields to catch the interest from a large number of existing DIY enthusiasts. The fundamental difference between boards like the Arduino Uno and the Intel Galileo is that Arduino devices run on a real-time microcontroller (mostly Atmel Atmega processors) whereas the Galileo runs on a System on Chip architecture (SoC). The SoC runs a standard multi-tasking operating system like Linux or Windows, which aren’t real-time.

Both Gen1 and Gen2 boards contain an Intel Quark 32-bit 400 MHz processor, which is compatible with the Intel Pentium processor instruction set. Furthermore we have a full-sized mini-PCI express slot, a 100 Mb Ethernet port, microSD slot and USB port. The Galileo is a headless device which means you can’t connect a monitor via a VGA or HDMI unlike the Raspberry Pi for example. The Galileo effectively offers Arduino compatibility through hardware pins, and software simulation within the operation system.

The microSD card slot makes it easy to run different operating systems on the device as you can simply write an operating system image on an SD card, insert it into the slot and boot the Galileo. Although Intel offers the Yocto Poky Linux environment there are some great initiatives to support other operating systems. At Build 2014 Microsoft announced the ‘Windows Developer Program for IoT’. As part of this program Microsoft offers a custom Windows image that can run on Galileo boards (there’s no official name yet, but let’s call it Windows IoT for now).

Windows on Devices / Windows Developer Program for IoT

Great, so now we can run .NET Framework application, and for example utilise the .NET Azure SDK? Well not really, yet… The Windows image is still in Alpha release stage and only runs a small subset of the .NET CLR and is not able to support larger .NET applications of any kind. Although a simple “Hello World” application will run flawlessly, applications will throw multiple Exceptions as soon as functionality beyond the System.Core.dll is called.

So how can we start building our things? You can write applications using the Wiring APIs in exactly the same way as you program your Arduino. Microsoft provides compatibility with the Arduino environment with a set of C++ libraries that are part of a new Visual Studio project type when you setup your development environment according to the instructions on http://ms-iot.github.io/content/.

We’ll start off by creating a new ‘Windows for IoT’ project in Visual Studio 2013:

New IoT VS Project

The project template will create a Visual C++ console application with a basic Arduino program that turns the built-in LED on and off in a loop:

Now let’s grab our breadboard and wire up some sensors. For the purpose of this demo I will use the built-in temperature sensor on the Galileo board. The objective will be to transmit the temperature to an Azure storage queue.

Since the Arduino Wiring API is implemented in C++ I decided to utilise some of the other Microsoft C++ libraries on offer: the Azure Storage Client Library for C++, which in return is using the C++ REST SDK. They’re hosted on Github and Codeplex respectively and can both be installed as Nuget packages. I was able to deliver messages to a storage queue with the C++ library in a standard C++ Win32 console application, so assumed this would work on the Galileo. Here’s the program listing of the ‘main.cpp’ file of the project:

The instructions mentioned earlier explain in detail how to setup your Galileo to run Windows, so I won’t repeat that here. We can deploy the Galileo console application to the development board from Visual Studio. This simply causes the compiled executable to be copied to the Galileo via a file share. Since it’s a headless device we can only connect to the Galileo via good old Telnet. Next, we launch the deployed application on the command line:

Windows IoT command line output

Although the console application is supposed to write output to the console, none of it is shown. I am wondering if there are certain Win32 features missing in this Windows on Devices release, since no debug information is outputted to the console for most commands that are executed over Telnet. When I tried to debug the application from Visual Studio I was able to extract some further diagnostics:

IoT VS Debug Output

Perhaps this is due to a missing Visual Studio C++ runtime on the Galileo board. I tried to perform an unattended installation of this runtime it did not seem to install at all, although a lack of command line output makes this guesswork.

Conclusion

Microsoft’s IoT offering is still in its very early days. That doesn’t only apply to the Windows IoT operating system, but for also to Azure platform features like Event Hubs as well. Although this is an Alpha release of Windows IoT I can’t say I’m overly impressed. The Arduino compatibility is a great feature, but a lack of easy connectivity makes it just a ‘thing’ without Internet. Although you can use the Arduino Ethernet / HTTP library, I would have liked to benefit from the available C++ libraries to securely connect to APIs over HTTPS, something which is impossible on the Arduino platform.

The Microsoft product documentation looks rather sloppy at times and is generally just lacking and I’m curious to see what the next release will bring along. According to Microsoft’s FAQ they’re focussing on supporting the universal app model. The recent announcements around open sourcing the .NET Framework will perhaps enable us to use some .NET Framework features in a Galileo Linux distribution in the not-to-distant future.

In a future blog post I will explore some other scenarios for the Intel Galileo using Intel’s IoT XDK, Node JS and look at how to connect the Galileo board to some of the Microsoft Azure platform services.

Reducing the size of an Azure Web Role deployment package

If you’ve been working with Azure Web Roles and deployed them to an Azure subscription, you likely have noticed the substantial size of a simple web role deployment package. Even with the vanilla ASP.NET sample website the deployment package seems to be quite bloated. This is not such a problem if you have decent upload bandwidth, but in Australia bandwidth is scarce like water in the desert so let’s see if we can compress this deployment package a little bit. We’ll also look at the consequences of this large package within the actual Web Role instances, and how we can reduce the footprint of a Web Role application.

To demonstrate the package size I have created a new Azure cloud service project with a standard ASP.NET web role:

1

Packaging up this Azure Cloud Service project results in a ‘CSPKG’ file and service configuration file:

2

As you can see the package size for a standard ASPX web role is around 14MB. The CSPKG is created in the ZIP format, and if we have a look inside this package we can have a closer look at what’s actually deployed to our Azure web role:

3

The ApplicationWebRole_….. file is a ZIP file itself and contains the following:

4

The approot and sitesroot folders are of significant size, and if we have a closer look they both contain the complete WebRole application including all content and DLL files! These contents are being copied to the actual local storage disk within the web role instances. When you’re dealing with large web applications this could potentially lead to issues due to the limitation of the local disk space within web role instances, which is around the 1.45 GB mark.

So why do we have these duplicate folders? The approot is used during role start up by the Windows Azure Host Bootstrapper and could contain a derived class from RoleEntryPoint. In this web role you can also include a start-up script which you can use to perform any customisations within the web role environment, like for example registering assemblies in the GAC.

The sitesroot contains the actual content that is served by IIS from within the web role instances. If you have defined multiple virtual directories or virtual applications these will also be contained in the sitesroot folder.

So is there any need for all the website content to be packaged up in the approot folder? No, absolutely not. The only reason we have this duplicate content is that the Azure SDK packages up the web role for storage and both the approot as well as sitesroot folders due to the behaviour of the Azure Web Role Bootstrapper.

The solution to this is to tailor the deployment package a little bit and get rid of the redundant web role content. Let’s create a new solution with a brand new web role:

5

This web role will hold just hold the RoleEntryPoint derived class (WebRole.cs) so we can safely remove all other content, NuGet packages and unnecessary referenced assemblies. The web role will not contain any of the web application bits that we want to host in Azure. This will result in the StartupWebRole to look like this:

6

Now we can add or include the web application that we want to publish to an Azure Web Role into the Visual Studio solution. They key point is to not include this as a role in the Azure Cloud Service project, but add it as a ‘plain web application’ to the solution. The only web role we’re publishing to Azure is the ‘StartupWebRole’, and we’re going to package up the actual web application in a slightly different way:

7

The ‘MyWebApplication’ project does not need to contain a RoleEntryPoint derived class, since this is already present on the StartupWebRole. Next, we open up the ServiceDefinition.csdef in the Cloud Service project and make some modifications in order to publish our web application along the StartupWebRole:
8

There are a few changes that need to be made:

  1. The name attribute of the Site element is set to the name of the web role containing the actual web application, which is ‘MyWebApplication’ in this instance.
  2. The physicalDirectory attribute is added and refers to the location where the ‘MyWebApplication’ will be published prior to creating the Azure package.

Although this introduces the additional step of publishing the web role to a separate physical directory, we immediately notice the reduced size of the deployment package:

9

When you’re dealing with larger web applications that contain numerous referenced assemblies the savings in size can add up quickly.

The Internet of Things with Arduino, Azure Event Hubs and the Azure Python SDK

In the emerging world of Internet of Things (IoT) we see more and more hardware manufacturers releasing development platforms to connect devices and sensors to the internet. Traditionally these kind of platforms are created around microcontrollers, and the Arduino platform can be considered as the standard in (consumer) physical computing, home automation, DIY and the ‘makers community’.

Most Arduinos come with an 8-bit AVR RISC-based microcontroller running at 16 MHz with the modest amount of 2 kilobytes of memory. These devices are perfectly capable of calling REST services with the Ethernet library and Arduino Ethernet shield. However, we do face some challenges when it comes to encrypting data, generating Azure shared access signatures and communicating over HTTPS due to a lack of processing power and memory. The Arduino platform has no SSL libraries and therefore cannot securely transmit data over HTTPS. This article shows a solution to this problem by using a secondary device as a bridge to the Internet.

Microsoft Azure allow us to store and process high volumes of sensor data through Event Hubs, currently still in preview. More information on Event Hubs, its architecture and how to publish and consume event data can be found here. In this article I focus on how to publish sensor data from these ‘things’ to an Azure Event Hub using a microcontroller with field gateway that is capable of communicating over HTTPS using the Azure Python SDK.

Azure Event Hubs

Before we start we need to create an Azure Service Bus Namespace and an Event Hub. This can be done in the Azure management portal:

Creating an Azure Event Hub

Creating an Azure Event Hub

When creating the event hub we need to specify the number of partitions. The link provided earlier will describe partitioning in detail, but in summary this will help us to spread the load of publishing devices across our event hub.

Event Hub Partitioning

Event Hub Partitioning

We can also define policies that can be used to generate a shared access signature on our devices that will be sending event data to the hub:

Event Hub Policies

Event Hub Policies

Arduino Yun

The Arduino Yun combines a microcontroller and ‘Wi-Fi System on Chip’ (WiSOC) on a single board. The microcontroller allows us to ‘sense’ the environment through its analogue input sensors, whereas the WiSOC runs a full Linux distribution with rich programming and connectivity capabilities. The WiSOC can be considered as the field gateway for the microcontroller and is able to send data to the Azure Event Hub. For other Arduino development boards that only have a microcontroller you can for example use a Raspberry Pi as the field gateway.
For the purpose of this demo we’ll keep the schematics simple and just use a simple temperature sensor and some LEDs to report back a status:

Yun temperature drawing_bb

The Arduino sketch reads the voltage signal from the temperature sensor and converts it to a temperature in Celsius degrees as our unit of measurement. The microcontroller communicates with the Yun Linux distribution via the bridge library, and blinks either the green or red LED depending on the HTTP status that is returned from the Linux distribution.
The complete Arduino sketch looks like this:

The Arduino bridge library is used to run a Python script within the Linux environment by executing a shell command to send the temperature to the Azure Event Hub. Next we’ll have a look at how this Python script actually works.

Python SDK

The Microsoft Azure Python SDK provides a set of Python packages for easy access to Azure storage services, service bus queues, topics and the service management APIs. Unfortunately there is no support for Event Hubs at this stage yet. Luckily Microsoft is embracing the open source community these days and is hosting the Python SDK on GitHub for public contribution, so hopefully this will be added soon. Details on how to install the Azure Python SDK in a Linux environment can be found on http://azure.microsoft.com/en-us/documentation/articles/python-how-to-install/. You can use a package manager like pip or easy_install to install the Python package ‘azure’.

The complete Python script to send event data to an Azure Event Hub is as follows:

The script can be called with a series of sensor values in the following format:

python script.py temperature:22,humidity:20 deviceid

Multiple key-value pairs with a sensor type and sensor value can be provided, and these will be nicely stored in the JSON message.

By using the ServiceBusSASAuthentication class from the Python SDK we can easily generate the shared access signature token that is required by the various services in the Azure ServiceBus, including Event Hubs.
Sending the actual event data is done with a simple REST call. Although Event Hubs allow any arbitrary message to be sent, we send JSON data which is easy to parse by potential consumers. Event data will be sent to a specific partition in the Event Hub. The hostname of the Arduino Yun is used as the partition key. Azure is taking care of assigning an actual Event Hub partition, but by using the hostname as the partition key it’s more likely that traffic from different devices is spread across the Event Hub for better throughput. The Python script will create the appropriate REST HTTP request according to the Azure Event Hub REST API:

When we deploy the Arduino sketch it will start sending the temperature to the Azure Event Hub continuously with one second intervals. We can confirm successful transmission by consuming the Event Hub data with Service Bus Explorer:

Service Bus Explorer

Service Bus Explorer

Conclusion

I’ve demonstrated how we can combine the Arduino microcontroller platform to read sensor data with a more powerful computing environment that runs Python. These allow our ‘things’ to leverage Azure Event Hubs for event processing with the potential to scale to millions of devices.