Publishing to Azure Event Hubs using a .NET Micro Framework Device

In previous posts, Kloudies Matt Davies and Olaf Loogman have shown how we connect Arduino based devices to the Azure platform. Preferring the .NET Micro Framework (NETMF) platform myself, I thought it time to show how we can publish senor data to Azure Event Hubs using a NETMF connected device.

.NET Micro Framework

Like Arduino, the .NET Micro Framework is an open source platform that runs on small, microcontroller based devices or “things”as we call them now in the world of the Internet-of-Things (IoT). However, unlike the Arduino platform, developers in the NETMF world use Visual Studio and C# to develop embedded applications leveraging the rich developer experience that comes with working within the Visual Studio IDE. Using the .NET Gadgeteer toolkit, we take this experience to the next level using a model driven development approach with graphical designers that abstracts much of the low level “engineering” aspects of embedded device development.

Whether we are working with earlier NETMF versions or with the Gadgeteer toolkit, we still get the rich debugging experience and deployment features from Visual Studio which is the big differentiator of the NETMF platform.

FEZ Panda II

The device I have had for a number of years is the FEZ Panda II from GHI Electronics (FEZ stands for Freak’N’Easy).

Although not a member of the newer .NET Gadgeteer family, the FEZ Panda II still provides those in the maker community a solid device platform for DIY and commercial grade applications. The FEZ Panda II sports:

• 72Mhz 32-bit processor with 512KB of FLASH and 96KB of RAM (compared to the Arduino Yun’s 16 MHz and 32KB of FLASH and  2KB of RAM)
• Micro SD socket for up to 16GB of memory
• Real Time Clock (RTC)
• Over 60 digital inputs and outputs
• TCP/IP HTTP support
• Arduino shield compatibility

Note: The FEZ Panda II does not have built in support for TLS/SSL which is required to publish data to Azure Event Hubs. This is not a problem for the newer Gadgeteer boards such as the FEZ Raptor.

Our Scenario

The scenario I will walkthrough in this post will feature our NETMF embedded device with a number of analogue sensors taking readings a couple of times per second and publishing the sensor data to an Azure Event Hub via a field gateway. A monitoring application will act as an event hub consumer to display sensor readings in near real-time.

• Sensors – Thermometer (degs Celsius) and Light (intensity of light as a % with zero being complete darkness)
• Device – FEZ Panda II connected to the internet using the Wiznet W5100 ethernet controller.
• Field Gateway – Simple IIS Application Request Routing rule in an Azure hosted Virtual Machine that routes the request as-is to a HTTPS Azure Event Hub endpoint.
• Cloud Gateway – Azure Event Hub configured as follows:
  • 8 partitions
  • 1 day message retention
  • Monitoring Consumer Group
  • Publisher access policy for our connected device
  • Consumer access policy for our monitoring application
• Monitoring Application – Silverlight (long live Ag!) application consuming events off the Monitoring Consumer Group.

Creating the Visual Studio Solution

To develop NETMF applications we must first install the .NET Micro Framework SDK and any vendor specific SDK’s. Using Visual Studio we create a NETMF project using the .NET Micro Framework Console Application template (or Gadgeteer template if you are using the newer family of devices).

For the FEZ Panda II, I need to target NETMF version 4.1.

Additionally, I also need to add assembly references to the device manufacturer’s SDK libraries, GHI Electronics in my case.

Sensor Code

Working with sensors and other modules is fairly straight forward using NETMF and the GHI libraries. To initialise an instance of my sensor class I need to know:

• The analogue pin on the device my sensor is connected to
• The interval between sensor readings
• The min/max values of the readings

public Thermometer(FEZ_Pin.AnalogIn pin, int interval, int min, int max)
{
    // Set reading parameters
    _interval = interval;
    _minScale = min;
    _maxScale = max;
            
    // Initialise thermometer sensor 
    _sensor = new AnalogIn((AnalogIn.Pin)pin);
    _sensor.SetLinearScale(_minScale, _maxScale);

    // Set sensor id
    _sensorId = "An" + pin.ToString() + _sensorType;
}

I then use a background thread to periodically take sensor readings and raise an event passing the sensor data as an event argument

void SensorThreadStart()
{
    while (true)
    {
        // Take a sensor reading
        var temp = _sensor.Read();

        // Create sensor event data 
        var eventData = new SensorEventData()
        {
            DeviceId = _deviceId,
            SensorData = new SensorData[]
                { 
                    new SensorData() { SensorId = _sensorId, SensorType = _sensorType, SensorValue = temp }
                }
        };

        // Raise sensor event
        OnSensorRead(eventData);

        // Pause
        Thread.Sleep(_interval);
    }
}

Networking

A critical attribute of any “thing” in the world of IoT is being connected. When working with resource constrained devices we quickly come to terms with having to perform many lower level functions than we may not be accustomed to in our day to day development. Initialising your network stack may be one of them…

public static void InitNetworkStack()
{
    Debug.Print("Network settings...");

    try
    {
        // Enable ethernet
        WIZnet_W5100.Enable
            (
                SPI.SPI_module.SPI1,
                (Cpu.Pin)FEZ_Pin.Digital.Di10,
                (Cpu.Pin)FEZ_Pin.Digital.Di7,
                true
            );

        // Enable DHCP
        Dhcp.EnableDhcp(new byte[] { 0x00, 0x5B, 0x1C, 0x51, 0xC6, 0xC7 }, "FEZA");
                
        Debug.Print("IP Address: " + new IPAddress(NetworkInterface.IPAddress).ToString());
        Debug.Print("Subnet Mask: " + new IPAddress(NetworkInterface.SubnetMask).ToString());
        Debug.Print("Default Gateway: " + new IPAddress(NetworkInterface.GatewayAddress).ToString());
        Debug.Print("DNS Server: " + new IPAddress(NetworkInterface.DnsServer).ToString());
    }
    catch (Exception ex)
    {
        Debug.Print("Network settings...Error: " + ex.ToString());
    }
    return;
}

Note the use of the Debug.Print statements. While in debug mode these are written to the output Window for easy troubleshooting and debugging.

Event Hub Client

As I write this, we don’t yet have a Azure SDK for NETMF (but we have been told it is on its way). Like most services in Azure, Event Hubs provides a REST based API that I can consume using plain old web http requests. To handle the access control, I assigned a pre-generated SAS token to the device during deployment. This avoids the resource constrained device having to generate a SAS token itself and use up precious memory doing so.

To construct our request to Event Hubs we need the following details:

• Service Bus Namespace
• Event Hub name
• PartitionKey (I am using a device ID)
• Authorisation token

public EventHubClient(string serviceNamespace, string eventhub, string deviceName, string accessSignature)
{
    // Assign event hub details
    _serviceNamespace = serviceNamespace;
    _hubName = eventhub;
    _deviceName = deviceName;
    _sas = accessSignature;

    // Generate the url to the event hub
    //_url = "https://" + _serviceNamespace + ".servicebus.windows.net/" + _hubName + "/Publishers/" + _deviceName;

    //  Note: As the FEZ Panda (.NET MF 4.1) does not support SSL I need to send this to the field gateway over HTTP
    _url = "http://dev-vs2014ctp-ty074cpf.cloudapp.net/" + _serviceNamespace + "/" + _hubName + "/" + _deviceName;
}

Note here I have switched my Event Hub Client to use an intermediary field gateway URL as the device does not support SSL and cannot post requests directly to Azure Event Hub endpoint.

Finally the actual payload is the sensor data that I serialise into json format. Event Hubs is payload agnostic so any stream of data may be sent through the hub. Anything from sensor data, application logging or perhaps observational data from medical devices can be published to Azure Event Hubs.

public bool SendEvent(SensorEventData sensorData)
{
    var success = false;
    try
    {
        // Format the sensor data as json
        var eventData = sensorData.ToJson();

        Debug.Print("Sending event data: " + eventData);

        // Create an HTTP Web request.
        HttpWebRequest webReq = HttpWebRequest.Create(_url) as HttpWebRequest;

        // Add required headers
        webReq.Method = "POST";
        webReq.Headers.Add("Authorization", _sas);
        webReq.ContentType = "application/atom+xml;type=entry;charset=utf-8";
        webReq.ContentLength = eventData.Length;
        webReq.KeepAlive = true;                

        using (var writer = new StreamWriter(webReq.GetRequestStream()))
        {
            writer.Write(eventData);                    
        }

        webReq.Timeout = 3000; // 3 secs
        using (var response = webReq.GetResponse() as HttpWebResponse)
        {
            Debug.Print("HttpWebResponse: " + response.StatusCode.ToString());
            // Check status code
            success = (response.StatusCode == HttpStatusCode.Created);
        }                
    }
    catch (Exception ex)
    {
        Debug.Print(ex.ToString());
    }

    return success;
}

Main

Wiring it all together is the job of our entry point Main(). Here we initialise our network stack, sensors, LEDs and of course our Azure Event Hub client. We then wire up the sensor event handlers and off we go.

public static void Main()
{            
    // Initialise device and sensors
    Init();

    // Setup Event Hub client
    client = new EventHubClient(serviceNamespace, hubName, deviceName, sas);

    Debug.Print("Device ready");

    // Start sensor monitoring
    thermo.Start();
    light.Start();
}

static void Init()
{
    // Enable ethernet
    Helpers.Network.InitNetworkStack();

    // Init LED 
    led = new LED((Cpu.Pin)FEZ_Pin.Digital.Di5, false);

    // Init thermometer sensor
    thermo = new Thermometer(FEZ_Pin.AnalogIn.An2, 500, -22, 56);
    thermo.SensorReadEvent += SensorReadEvent;

    // Init light sensor
    light = new Light(FEZ_Pin.AnalogIn.An3, 500, 0, 100);
    light.SensorReadEvent += SensorReadEvent;

    // Flash once if all is good
    led.Flash(1);
}

static void SensorReadEvent(SensorEventData data)
{
    // Send event to Event Hubs
    if (!client.SendEvent(data))
        // Flash three times if failed to send
        led.Flash(3);
    else
        // Flash once if all is good
        led.Flash(1);
}

Note the use of the LED, connected to digital pin 5, to provide runtime feedback. We flash the LED once for every successful publish of an event and three times if we have a failure. It is this kind of low level controller interaction that makes NETMF development such a satisfying, albeit geeky pastime.

Field Gateway

As mentioned above, the FEZ Panda II does not support TLS/SSL. To overcome this, I posted sensor data to a “field gateway” consisting of a simple IIS Application Request Routing rule to perform the protocol translation from HTTP to HTTPS. The ARR rule only performed a rewrite of the URL and did not need to enrich or modify the request in any other way.

Our Consumer

Azure Event Hubs provides Consumer Groups that subscribe to events published to the hub. Only one consumer can receive events from each partition at time so I have found it good practice to create at least two consumer groups so that one group can be used for monitoring as required while your downstream processing application/services consume the primary consumer group. To this end, I developed a quick Silverlight application (yes I know. long live Ag!) to act as a monitoring consumer for the event hub.

Conclusion

The .NET Micro Framework provides a great way for .NET developers to participate in the growing Internet-of-Things movement for a relatively small ( < $100 ) outlay while retaining the rich developer experience using familiar tools such as Visual Studio. Azure Event Hubs provides the platform for a cloud-based device gateway allowing the ingestion of millions of events that downstream applications and services can consume for real-time analytics and ordered processing.