Automating Azure Instrumentation and Monitoring – Part 3: Custom Metrics

One of the core data types that Azure Monitor works with is metrics – numerical pieces of data that represent the state of an Azure resource or of an application component at a specific point in time. Azure publishes built-in metrics for almost all Azure services, and these metrics are available for querying interactively as well as for use within alerts and other systems. In addition to the Azure-published metrics, we can also publish our own custom metrics. In this post we’ll discuss how to do this using both Azure Monitor’s recently announced support for custom metrics, and with Application Insights’ custom metrics features. We’ll start by looking at what metrics are and how they work.

This post is part of a series:

  • Part 1 provides an introduction to the series by describing why we should instrument our systems, outlines some of the major tools that Azure provides such as Azure Monitor, and argues why we should be adopting an ‘infrastructure as code’ mindset for our instrumentation and monitoring components.

  • Part 2 describes Azure Application Insights, including its proactive detection and alert features. It also outlines a pattern for deploying instrumentation components based on the requirements we might typically have for different environments, from short-lived development and test environments through to production.

  • Part 3 (this post) discusses how to publish custom metrics, both through Application Insights and to Azure Monitor. Custom metrics let us enrich the data that is available to our instrumentation components.

  • Part 4 covers the basics of alerts and metric alerts. Azure Monitor’s powerful alerting system is a big topic, and in this part we’ll discuss how it works overall, as well as how to get alerts for built-in and custom metrics.

  • Part 5 (coming soon) covers log alerts and resource health alerts, two other major types of alerts that Azure Monitor provides. Log alerts let us alert on information coming into Application Insights logs and Log Analytics workspaces, while resource health alerts us when Azure itself is having an issue that may result in downtime or degraded performance.

  • Part 6 (coming soon) describes dashboards. The Azure Portal has a great dashboard UI, and our instrumentation data can be made available as charts. Dashboards are also possible to automate, and I’ll show a few tips and tricks I’ve learned when doing this.

  • Part 7 (coming soon) covers availability tests, which let us proactively monitor our web applications for potential outages. We’ll discuss deploying and automating both single-step (ping) and multi-step availability tests.

  • Part 8 (coming soon) describes autoscale. While this isn’t exactly instrumentation in and of itself, autoscale is built on much of the same data used to drive alerts and dashboards, and autoscale rules can be automated as well.

  • Finally, part 9 (coming soon) covers exporting data to other systems. Azure Monitor metrics and log data can be automatically exported, as can Application Insights data, and the export rules can be exported and used from automation scripts.

What Are Metrics?

Metrics are pieces of numerical data. Each metric has both a value and a unit. Here are some example metrics:

ExampleValueUnit
12 requests per second12requests per second
54 gigabytes54gigabytes
7 queue messages7queue messages

Metrics can be captured either in their raw or aggregated form. An aggregated metric is a way of simplifying the metric across a given period of time. For example, consider a system that processes messages from a queue. We could count the number of messages processed by the system in two ways: we could adjust our count every time a message is processed, or we could check the number of messages on the queue every minute, and batch these into five-minute blocks. The latter is one example of an aggregated metric.

Because metrics are numerical in nature, they can be visualised in different ways. For example, a line chart might show the value of a metric over time.

Azure Monitor also supports adding dimensions to metrics. Dimensions are extra pieces of data that help to add context to a metric. For example, the Azure Monitor metric for the number of messages in a Service Bus namespace has the entity (queue or topic) name as a dimension. Queries and visualisations against this metric can then filter down to specific topics, can visualise each topic separately, or can roll up all topics/queues and show the total number of messages across the whole Service Bus namespace.

Azure Monitor Metrics

Azure Monitor currently has two metric systems.

  • Classic metrics were traditionally published by most Azure services. When we use the Azure Portal, the Metrics (Classic) page displays these metrics.
  • Near real-time metrics are the newer type of metrics, and Azure is moving to use this across all services. As their name suggests, these metrics get updated more frequently than classic metrics – where classic metrics might not appear for several minutes, near real-time metrics typically are available for querying within 1-2 minutes of being published, and sometimes much quicker than that. Additionally, near real-time metrics are the only metric type that supports dimensions; classic metrics do not. Custom metrics need to be published as near real-time metrics.

Over time, all Azure services will move to the near real-time metrics system. In the meantime, you can check whether a given Azure service is publishing to the classic or newer metric system by checking this page. In this post we’ll only be dealing with the newer (near real-time) metric system.

Custom Metrics

Almost all Azure services publish their own metrics in some form, although the usefulness and quality varies depending on the specific service. Core Azure services tend to have excellent support for metrics. Publishing of built-in metrics happens automatically and without any interaction on our part. The metrics are available for interactive querying through the portal and API, and for alerts and all of the other purposes we discussed in part 1 of this series.

There are some situations where the built-in metrics aren’t enough for our purposes. This commonly happens within our own applications. For example, if our application has components that process messages from a queue then it can be helpful to know how many messages are being processed per minute, how long each message takes to process, and how many messages are currently on the queues. These metrics can help us to understand the health of our system, to provision new workers to help to process messages more quickly, or to understand bottlenecks that our developers might need to investigate.

There are two ways that we can publish custom metrics into Azure Monitor.

  • Azure Monitor custom metrics, currently a preview service, provides an API for us to send metrics into Azure Monitor. We submit our metrics to an Azure resource, and the metrics are saved alongside the built-in metrics for that resource.
  • Application Insights also provides custom metrics. Our applications can create and publish metrics into Application Insights, and they are accessible by using Application Insights’ UI, and through the other places that we work with metrics. Although the core offering of publishing custom metrics into Application Insights is generally available, some specific features are in preview.

How do we choose which approach to use? Broadly speaking I’d generally suggest using Azure Monitor’s custom metrics API for publishing resource or infrastructure-level metrics – i.e. enriching the data that Azure itself publishes about a resource – and I’d suggest using Application Insights for publishing application-level metrics – i.e. metrics about our own application code.

Here’s a concrete example, again related to queue processing. If we have an application that processes queue messages, we’ll typically want instrumentation to understand how these queues and processors are behaving. If we’re using Service Bus queues or topics then we get a lot of instrumentation about our queues, including the number of messages that are currently on the queue. But if we’re using Azure Storage queues, we’re out of luck. Azure Storage queues don’t have the same metrics, and we don’t get the queue lengths from within Azure Monitor. This is an ideal use case for Azure Monitor’s custom metrics.

We may also want to understand how long it’s taking us to process each message – from the time it was submitted to the queue to the time it completed processing. This is often an important metric to ensure that our data is up-to-date and that users are having the best experience possible. Ultimately this comes down to how long our application is taking to perform its logic, and so this is an application-level concern and not an infrastructure-level concern. We’ll use Application Insights for this custom metric.

Let’s look at how we can write code to publish each of these metrics.

Publishing Custom Resource Metrics

In order to publish a custom resource metric we need to do the following:

  • Decide whether we will add dimensions.
  • Decide whether we will aggregate our metric’s value.
  • Authenticate and obtain an access token.
  • Send the metric data to the Azure Monitor metrics API.

Let’s look at each of these in turn, in the context of an example Azure Function app that we’ll use to send our custom metrics.

Adding Dimensions

As described above, dimensions let us add extra data to our metrics so that we can group and compare them. We can submit metrics to Azure Monitor with or without dimensions. If we want to include dimensions, we need to include two extra properties – dimNames specifies the names of the dimensions we want to add to the metric, and dimValues specifies the values of those dimensions. The order of the dimension names and values must match so that Azure Monitor can relate the value to its dimension name.

Aggregating Metrics

Metrics are typically queried in an aggregated form – for example, counting or averaging the values of metrics to get a picture of how things are going overall. When submitting custom metrics we can also choose to send our metric values in an aggregated form if we want. The main reasons we’d do this are:

  • To save cost. Azure Monitor custom metrics aren’t cheap when you use them at scale, and so pre-aggregating within our application means we don’t need to incur quite as high a cost since we aren’t sending as much raw data to Azure Monitor to ingest.
  • To reduce a very high volume of metrics. If we have a large number of metrics to report on, it will likely be much faster for us to send the aggregated metric to Azure Monitor rather than sending each individual metric.

However, it’s up to us – we can choose to send individual values if we want.

If we send aggregated metrics then we need to construct a JSON object to represent the metric as follows:

For example, let’s imagine we have recorded the following queue lengths (all times in UTC):

TimeLength
11:00am1087
11:01am1124
11:02am826
11:03am888
11:04am1201
11:05am1091

We might send the following pre-aggregated metrics in a single payload:

Azure Monitor would then be able to display the aggregated metrics for us when we query.

If we chose not to send the metrics in an aggregated form, we’d send the metrics across individual messages; here’s an example of the fourth message:

Security for Communicating with Azure Monitor

We need to obtain an access token when we want to communicate with Azure Monitor. When we use Azure Functions, we can make use of managed identities to simplify this process a lot. I won’t cover all the details of managed identities here, but the example ARM template for this post includes the creation and use of a managed identity. Once the function is created, it can use the following code to obtain a token that is valid for communication with Azure Monitor:

The second part of this process is authorising the function’s identity to write metrics to resources. This is done by using the standard Azure role-based access control system. The function’s identity needs to be granted the Monitoring Metrics Publisher role, which has been defined with the well-known role definition ID 3913510d-42f4-4e42-8a64-420c390055eb.

Sending Custom Metrics to Azure Monitor

Now we have our metric object and our access token ready, we can submit the metric object to Azure Monitor. The actual submission is fairly easy – we just perform a POST to a URL. However, the URL we submit to will be different depending on the resource’s location and resource ID, so we dynamically construct the URL as follows:

We might deploy into the West US 2 region, so an example URL might look like this: https://westus2.monitoring.azure.com/subscriptions/377f4fe1-340d-4c5d-86ae-ad795b5ac17d/resourceGroups/MyCustomMetrics/providers/Microsoft.Storage/storageAccounts/mystorageaccount/metrics

Currently Azure Monitor only supports a subset of Azure regions for custom metrics, but this list is likely to grow as the feature moves out of preview.

Here is the full C# Azure Function we use to send our custom metrics:

Testing our Function

I’ve provided an ARM template that you can deploy to test this:

Make sure to deploy this into a region that supports custom metrics, like West US 2.

Once you’ve deployed it, you can create some queues in the storage account (use the storage account that begins with q and not the one that begins with fn). Add some messages to the queues, and then run the function or wait for it to run automatically every five minutes.

Then you can check the metrics for the storage queue, making sure to change the metric namespace to queueprocessing:

You should see something like the following:

As of the time of writing (January 2019), there is a bug where Azure Storage custom metrics don’t display dimensions. This will hopefully be fixed soon.

Publishing Custom Application Metrics

Application Insights also allows for the publishing of custom metrics using its own SDK and APIs. These metrics can be queried through Azure Monitor in the same way as resource-level metrics. The process by which metrics are published into Application Insights is quite different to how Azure Monitor custom metrics are published, though.

The Microsoft documentation on Application Insights custom metrics is quite comprehensive, so rather than restate it here I will simply link to the important parts. I will focus on the C# SDK in this post.

To publish a custom metric to Application Insights you need an instance of the TelemetryClient class. In an Azure Functions app you can set the APPINSIGHTS_INSTRUMENTATIONKEY application setting – for example, within an ARM template – and then create an instance of TelemetryClient. The TelemetryClient will find the setting and will automatically configure itself to send telemetry to the correct place.

Once you have an instance of TelemetryClientyou can use the GetMetric().TrackValue() method to log a new metric value, which is then pre-aggregated and sent to Application Insights after a short delay. Dimensions can also be set using the same method. There are a number of overloads of this method that can be used to submit custom dimensions, too.

Note that as some features are in preview, they don’t work consistently yet – for example, at time of writing custom namespaces aren’t honoured correctly, but this should hopefully be resolved soon.

If you want to send raw metrics rather than pre-aggregated metrics, the legacy TrackMetric() method can be used, but Microsoft discourage its use and are deprecating it.

Here is some example Azure Function code that writes a random value to the My Test Metric metric:

And a full ARM template that deploys this is:

Summary

Custom metrics allow us to enrich our telemetry data with numerical values that can be aggregated and analysed, both manually through portal dashboards and APIs, and automatically using a variety of Azure Monitor features. We can publish custom metrics against any Azure resource by using the new custom metrics APIs, and we can also write application-level metrics to Application Insights.

In the next part of this series we will start to look at alerts, and will specifically look at metric alerts – one way to have Azure Monitor process the data for both built-in and custom metrics and alert us when things go awry.

Monitoring Azure Storage Queues with Application Insights and Azure Monitor

Azure Queues provides an easy queuing system for cloud-based applications. Queues allow for loose coupling between application components, and applications that use queues can take advantage of features like peek-locking and multiple retry attempts to enable application resiliency and high availability. Additionally, when Azure Queues are used with Azure Functions or Azure WebJobs, the built-in poison queue support allows for messages that repeatedly fail processing attempts to be moved to a dedicated queue for later inspection.
An important part of operating a queue-based application is monitoring the length of queues. This can tell you whether the back-end parts of the application are responding, whether they are keeping up with the amount of work they are being given, and whether there are messages that are causing problems. Most applications will have messages being added to and removed from queues as part of their regular operation. Over time, an operations team will begin to understand the normal range for each queue’s length. When a queue goes out of this range, it’s important to be alerted so that corrective action can be taken.
Azure Queues don’t have a built-in queue length monitoring system. Azure Application Insights allows for the collection of large volumes of data from an application, but it does not support monitoring queue lengths with its built-in functionality. In this post, we will create a serverless integration between Azure Queues and Application Insights using an Azure Function. This will allow us to use Application Insights to monitor queue lengths and set up Azure Monitor alert emails if the queue length exceeds a given threshold.

Solution Architecture

There are several ways that Application Insights could be integrated with Azure Queues. In this post we will use Azure Functions. Azure Functions is a serverless platform, allowing for blocks of code to be executed on demand or at regular intervals. We will write an Azure Function to poll the length of a set of queues, and publish these values to Application Insights. Then we will use Application Insights’ built-in analytics and alerting tools to monitor the queue lengths.

Base Application Setup

For this sample, we will use the Azure Portal to create the resources we need. You don’t even need Visual Studio to follow along. I will assume some basic familiarity with Azure.
First, we’ll need an Azure Storage account for our queues. In our sample application, we already have a storage account with two queues to monitor:

  • processorders: this is a queue that an API publishes to, and a back-end WebJob reads from the queue and processes its items. The queue contains orders that need to be processed.
  • processorders-poison: this is a queue that WebJobs has created automatically. Any messages that cannot be processed by the WebJob (by default after five attempts) will be moved into this queue for manual handling.

Next, we will create an Azure Functions app. When we create this through the Azure Portal, the portal helpfully asks if we want to create an Azure Storage account to store diagnostic logs and other metadata. We will choose to use our existing storage account, but if you prefer, you can have a different storage account than the one your queues are in. Additionally, the portal offers to create an Application Insights account. We will accept this, but you can create it separately later if you want.
1-FunctionsApp
Once all of these components have been deployed, we are ready to write our function.

Azure Function

Now we can write an Azure Function to connect to the queues and check their length.
Open the Azure Functions account and click the + button next to the Functions menu. Select a Timer trigger. We will use C# for this example. Click the Create this function button.
2-Function
By default, the function will run every five minutes. That might be sufficient for many applications. If you need to run the function on a different frequency, you can edit the schedule element in the function.json file and specify a cron expression.
Next, paste the following code over the top of the existing function:

This code connects to an Azure Storage account and retrieves the length of each queue specified. The key parts here are:
[sourcecode language=”csharp” wraplines=”true” collapse=”false” gutter=”false”]
var connectionString = System.Configuration.ConfigurationManager.AppSettings["AzureWebJobsStorage"];
[/sourcecode]
Azure Functions has an application setting called AzureWebJobsStorage. By default this refers to the storage account created when we provisioned the functions app. If you wanted to monitor a queue in another account, you could reference the storage account connection string here.
[sourcecode language=”csharp” wraplines=”true” collapse=”false” gutter=”false”]
var queue = queueClient.GetQueueReference(queueName);
queue.FetchAttributes();
var length = queue.ApproximateMessageCount;
[/sourcecode]
When you obtain a reference to a queue, you must explicitly fetch the queue attributes in order to read the ApproximateMessageCount. As the name suggests, this count may not be completely accurate, especially in situations where messages are being added and removed at a high rate. For our purposes, an approximate message count is enough for us to monitor.
[sourcecode language=”csharp” wraplines=”true” collapse=”false” gutter=”false”]
log.Info($"{queueName}: {length}");
[/sourcecode]
For now, this line will let us view the length of the queues within the Azure Functions log window. Later, we will switch this out to log to Application Insights instead.
Click Save and run. You should see something like the following appear in the log output window below the code editor:

2017-09-07T00:35:00.028 Function started (Id=57547b15-4c3e-42e7-a1de-1240fdf57b36)
2017-09-07T00:35:00.028 C# Timer trigger function executed at: 9/7/2017 12:35:00 AM
2017-09-07T00:35:00.028 processorders: 1
2017-09-07T00:35:00.028 processorders-poison: 0
2017-09-07T00:35:00.028 Function completed (Success, Id=57547b15-4c3e-42e7-a1de-1240fdf57b36, Duration=9ms)

Now we have our function polling the queue lengths. The next step is to publish these into Application Insights.

Integrating into Azure Functions

Azure Functions has integration with Appliation Insights for logging of each function execution. In this case, we want to save our own custom metrics, which is not currently supported by the built-in integration. Thankfully, integrating the full Application Insights SDK into our function is very easy.
First, we need to add a project.json file to our function app. To do this, click the View files tab on the right pane of the function app. Then click the + Add button, and name your new file project.json. Paste in the following:

This adds a NuGet reference to the Microsoft.ApplicationInsights package, which allows us to use the full SDK from our function.
Next, we need to update our function so that it writes the queue length to Application Insights. Click on the run.csx file on the right-hand pane, and replace the current function code with the following:

The key new parts that we have just added are:
[sourcecode language=”csharp” wraplines=”true” collapse=”false” gutter=”false”]
private static string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
private static TelemetryClient telemetry = new TelemetryClient() { InstrumentationKey = key };
[/sourcecode]
This sets up an Application Insights TelemetryClient instance that we can use to publish our metrics. Note that in order for Application Insights to route the metrics to the right place, it needs an instrumentation key. Azure Functions’ built-in integration with Application Insights means that we can simply reference the instrumentation key it has set up for us. If you did not set up Application Insights at the same time as the function app, you can configure this separately and set your instrumentation key here.
Now we can publish our custom metrics into Application Insights. Note that Application Insights has several different types of custom diagnostics that can be tracked. In this case, we use metrics since they provide the ability to track numerical values over time, and set up alerts as appropriate. We have added the following line to our foreach loop, which publishes the queue length into Application Insights.
[sourcecode language=”csharp” wraplines=”true” collapse=”false” gutter=”false”]
telemetry.TrackMetric($"Queue length – {queueName}", (double)length);
[/sourcecode]
Click Save and run. Once the function executes successfully, wait a few minutes before continuing with the next step – Application Insights takes a little time (usually less than five minutes) to ingest new data.

Exploring Metrics in Application Insights

In the Azure Portal, open your Application Insights account and click Metrics Explorer in the menu. Then click the + Add chart button, and expand the Custom metric collection. You should see the new queue length metrics listed.
4-AppInsights
Select the metrics, and as you do, notice that a new chart is added to the main panel showing the queue length count. In our case, we can see the processorders queue count fluctuate between one and five messages, while the processorders-poison queue stays empty. Set the Aggregation property to Max to better see how the queue fluctuates.
5-AppInsightsChart
You may also want to click the Time range button in the main panel, and set it to Last 30 minutes, to fully see how the queue length changes during your testing.

Setting up Alerts

Now that we can see our metrics appearing in Application Insights, we can set up alerts to notify us whenever a queue exceeds some length we specify. For our processorders queue, this might be 10 messages – we know from our operations history that if we have more than ten messages waiting to be processed, our WebJob isn’t processing them fast enough. For our processorders-poison queue, though, we never want to have any messages appear in this queue, so we can have an alert if more than zero messages are on the queue. Your thresholds may differ for your application, depending on your requirements.
Click the Alert rules button in the main panel. Azure Monitor opens. Click the + Add metric alert button. (You may need to select the Application Insights account in the Resource drop-down list if this is disabled.)
On the Add rule pane, set the values as follows:

  • Name: Use a descriptive name here, such as `QueueProcessOrdersLength`.
  • Metric: Select the appropriate `Queue length – queuename` metric.
  • Condition: Set this to the value and time period you require. In our case, I have set the rule to `Greater than 10 over the last 5 minutes`.
  • Notify via: Specify how you want to be notified of the alert. Azure Monitor can send emails, call a webhook URL, or even start a Logic App. In our case, I have opted to receive an email.

Click OK to save the rule.
6-Alert.PNG
If the queue count exceeds your specified limit, you will receive an email shortly afterwards with details:
7-Alert

Summary

Monitoring the length of your queues is a critical part of operating a modern application, and getting alerted when a queue is becoming excessively long can help to identify application failures early, and thereby avoid downtime and SLA violations. Even though Azure Storage doesn’t provide a built-in monitoring mechanism, one can easily be created using Azure Functions, Application Insights, and Azure Monitor.

Azure Functions Logging to Application Insights

We’re going to have a look at several ways to integrate Application Insights (AppInsights) with Azure Functions (Functions).
Functions supports built-in logging features using TraceWriter instance. Basic sample function might look like:

With TraceWriter, we can log information to the log console like:

However, it has the maximum limit of 1000 records. This is good for simple debugging purposes, but not for logging. Therefore, we should store logs somewhere like database or storage. Fortunately, AppInsights has recently been consolidated to Functions as a preview to overcome these limitations. Let’s have a look.

Application Insights Integration

According to the document, it’s really easy.

  1. Create an AppInsights instance. Its type MUST be General.
  2. Add a new key of APPINSIGHTS_INSTRUMENTATIONKEY to the AppSettings section of the Function instance.
  3. Give the Instrumentation Key value of the AppInsights instance to the key, APPINSIGHTS_INSTRUMENTATIONKEY.

This is it. Once it’s done, simply execute some functions and wait for the aggregated result up to 5 minutes. Then, go to the AppInsights blade and find the graph looking like:

Can’t be easier, huh?

ARM Template Setup for DevOps Engineers

We can add Instrumentation Key like above. However, this is not ideal from the CI/CD point of view. Instead, setting the key within an ARM template would be more preferred, effective and efficient. Here’s the cut-down version of ARM template sample:

As we can see above, we can directly pour the Instrumentation Key within the ARM template without knowing it. If we want to know more about ARM template, this official document would be a good starting point.

ILogger Integration

ILogger is a library of ASP.NET Core. As it supports .NET Standard 1.1, Functions has recently introduced ILogger version 1.1.1. With this, we can virtually add as many logging libraries as possible. Functions provides AppInsights logging through this interface. In other words, we can simply replace the TraceWriter instance with the ILogger one, in order to send all loggings to AppInsights.
This is also really easy. Simply replace TraceWriter with ILogger in the Function parameter and change the method name from log.Info() to log.LogInformation():

If we still want to keep the log.Info() method name, that’s fine. Simply create an extension method like:

And use the extension method like below:

Once everything is done, deploy the Function again and run Functions several times. Then check out the Function log console:

We have the exactly same experience as before. Of course, we can see additional logging details on the AppInsights blade:

As mentioned above, Functions has implemented the ILogger interface. That means, we might be able to add third-party logging libraries such as Serilog. Unfortunately, at the time of writing, we can’t use those third-party ones. But the Azure Functions Team has started looking at those implementation, according to this issue. Hope this feature is released soon.
Another known issue around the ILogger implementation on Functions is that the Azure Functions Core Tools that helps local debugging doesn’t display log to the console. So, don’t panic, even if no log is displayed on your local console. It displays as expected when you deploy your Functions to Azure. The Azure Functions Team works really hard to fix the issue sooner rather than later.
So far, we have walked through a few ways to integrate AppInsights with Azure Functions. As it’s still in preview, features around this may change over time until GA. But I’m pretty sure that this logging integration to AppInsights will be quite useful and powerful.

Monitoring Azure WebJobs Health with Application Insights

Introduction

Azure WebJobs have been available for quite some time and have become very popular for running background tasks with programs or scripts. WebJobs are deployed as part of Azure App Services (Web Apps), which include their companion site Kudu. Kudu provides a lot of features, including a REST API, which provides operations for source code management (SCM), virtual file system, deployments, accessing logs, and for WebJob management as well. The Kudu WebJobs API provides different operations including listing WebJobs, uploading a WebJob, or triggering it. One of the operations of this API allows to get the status of a specific WebJob by name.

Another quite popular Azure service is Application Insights. This provides functionality to monitor and diagnose application issues and to analyse usage and performance as well. One of these features are web tests, which provide a way to monitor the availability and health of a web site.

In this blog post I will go through the required configuration on Application Insights to monitor the health of WebJobs using Application Insights web tests calling the Kudu WebJobs API.

Calling the Kudu WebJobs API.

For this exercise, it is worth getting familiar with the WebJobs API, particularly with the endpoint to get a WebJob status. Through this post, I will be working with a triggered WebJob scheduled with a CRON expression, but you can apply the same principles for a continuous WebJob. I will be using postman to call this API.

To get a WebJob status, we need to call the corresponding Kudu WebJob API endpoint. In the case of triggered WebJobs, the endpoint looks something like:

https://{webapp-name}.scm.azurewebsites.net/api/triggeredwebjobs/{webjob-name}/

Before calling the endpoint, we need to add the Authorization header to the GET request. To create the header value, we need to use the corresponding Kudu API credentials, as explained here. Considering we want to monitor the status of a WebJob under a particular web site, I prefer to use site-level credentials (or publishing profile credentials) instead of the user-level ones.

Getting the Publishing Profile Credentials from the Azure Portal

You can get the publishing profile credentials, by downloading the publishing profile from the portal, as shown in the figure below. Once downloaded, the XML document will contain the site-level credentials.

Getting the Publishing Profile Credentials via PowerShell

We can also get the site-level credentials via PowerShell. I’ve created a PowerShell function which returns the publishing credentials of an Azure Web App or a Deployment Slot, as shown below.

Bear in mind that you need to be logged in to Azure in your PowerShell session before calling these cmdlets.

Getting the Kudu REST API Authorisation header via PowerShell

Once we have the credentials, we are able to get the Authorization header value. The instructions to construct the header are described here. I’ve created another PowerShell function, which relies on the previous one, to get the header value, as follows.

Once we have the header value, we can call the api. Let’s call it using postman.

You should be getting a response similar to the one shown below:

Note that for this triggered WebJob, there are status and duration fields.

Now that we are familiar with the response, we can start designing an App Insights web test to monitor the health of our WebJob.

Configuring an App Insights Web Test to Monitor the Health of an Azure WebJob

You can find here detailed documentation on how to create web tests to monitor availability and responsiveness of web end points. In the following sections of this post, I will cover how to create an App Insights web test to Monitor the Health of a WebJob.

As we saw above, to call the WebJobs API we need to add an Authorization Header to the GET request. And once we get the API response, to check the status of the WebJob, we would need to interpret the response in JSON format.

To create the web test on App Insights to monitor a WebJob, I will first create a simple web test via the Azure Portal, and enrich it later.

Creating a Web Test on Application Insights.

I will create a basic web test with the following configuration. You should change it to the values which suit your scenario:

  • Test type: URL ping test
  • URL: My WebJob Rest API, e.g. https://{webapp-name}.scm.azurewebsites.net/api/triggeredwebjobs/{webjob-name}/
  • Test frequency: 5 minutes
  • Test locations: SG Singapore and AU Sydney
  • Success criteria:
    • Test timeout: 120 seconds
    • HTTP Response: (checked)
    • Status code must equal: 200
    • Content match: (checked)
    • Content must contain: “status”:”success”
  • Alerts
    • Status: Enabled
    • Alert threshold location: 1
    • Alert failure time window: 5 minutes
    • Send alert emails to these email addresses: <my email address>

You could also keep email alerts disabled or configure them later.

If you enable the web test as is, you will see that it will start failing. The reason being that we are not adding the required Authorization header to the GET request.

To add headers to the test, you could record web tests on Visual Studio Enterprise or Ultimate. This is explained in details in the Azure documentation. Additionally, in these multi-steps web tests you can add more than one validation rule.

Knowing that not everybody has access to a VS Enterprise or Ultimate license, I will explain here how to create a web test using the corresponding XML format. The first step is to extract the web test XML definition from the test manually created on the portal.

Extracting the Web Test XML Definition from a Test Manually Created on the Portal.

Once we have created the web test manually on the portal, to get its XML definition, we have to open the resource explorer on https://resources.azure.com/ and navigate to subscriptions/<subscription-guid>/resourceGroups/<resourcegroup>/providers/microsoft.insights/webtests/<webtest>-<app-insight> until you are on the definition of the web test you have just created.

Once there, you need to find the member: “WebTest”, which should be something similar to:

Now, we need to extract the XML document by removing the escape characters of the double quotes, and get something like:

which is the XML definition of the web test we created manually on the portal.

Adding a Header to the Application Insights Web Test Request by updating the Web Test XML definition.

Now we should be ready to edit our web test XML definition to add the Authorization header.

To do this, we just need to add a Headers child element to the Request record, similar to the one shown below. You would need to get the Base 64 encoded Authorization header value, similarly to how we did it previously when calling the API via Postman.

Extending the Functionality of the Web Test.

When we created the web test on the portal, we said that we wanted the status to be “success”, however, we might want to add “running” as another valid value. Additionally, in my case, I wanted to check that duration is less than 10 minutes. For this I have updated the Validation Rules to use regular expressions and to have a second rule. The final web test XML definition resulted as follows:

You could play around with the web test XML definition and update or extend it according to your needs. In case you are interested on the capabilities of web tests, here the documentation.

Once our web test XML definition is ready, we save it with a “.webtest” extension.

Uploading the (Multi-Step) Web Test to Application Insights

Having the web test XML definition ready, we can update our Application Insights web test with it. For this, on the portal, we open the Edit Test blade and:

  • Change the Test Type to: Multi-step test, and
  • Upload the web test xml definition file we just saved with the “.webtest” extension.

This will update the web test, and now with the proper Authorization header and the added validation rules, we can monitor the health of our triggered WebJob.

With Application Insights web tests, we can monitor the WebJob via the dashboard as shown above, or configuring alerts to be sent via email.

Summary

Through this post I have shown how to monitor the health of an Azure WebJob using Application Insights web tests. But on the journey, I also showed some tricks which I hope can be useful in other scenarios as well, including

  1. How to call the Azure WebJobs API via Postman, including how to get the Kudu API Authorization header via PowerShell.
  2. How to manually configure App Insights web tests,
  3. How to get the XML definition of a manually created web test using the Azure Resource Explorer,
  4. How to update the web test XML definition to add a request
    header and expand the validation rules. This without requiring Visual Studio Enterprise or Ultimate, and
  5. How to update the Application Insights web test by uploading the updated multi-step web test file.

Thanks for reading, and feel free to add your comments or queries below. 🙂 

Creating Web Tests for Application Insights Programmatically

Azure SDK for .NET provides very useful features to deal with Azure services and resources including Application Insights. With this SDK, we can easily create or update resources for Application Insights and alerts. However, surprisingly this SDK doesn’t provide creating Web Tests resources to check availability. In this post, we will walk through how to create web tests programmatically using C# codes.

Web Tests in Azure Portal

Once an Application Insights instance is created in Azure Portal, we can easily create web tests by following this post, Monitor availability and responsiveness of any web site. This is pretty straightforward, so we’re not going to do here.

Web Tests in PowerShell

The problem using the approach above is that we should manually create or update web tests through Azure Portal. With PowerShell script, we can easily create or update web tests by following this post, Creating an Application Insights Web Test and Alert Programmatically. This is also quite clear to understand, so we’re not dealing it with here.

There’s a question still left. How can I create or update web tests and alerts using C# codes? This part has not been yet supported. Of course we can, but not that as easy as others. Let’s sort this out.

Code used in this post can be found at https://github.com/aliencube/Application-Insights-WebTests-SDK.

Analysing and Structuring Web Test Resources

Azure provides a website called Azure Resource Explorer. After logging in, we’re able to see a similar screen to this:

This is how a web test resource looks like. From this JSON object, we need to create strongly-typed object classes. Let’s say this is called WebTestResource that inherits GenericResource.

This is a basic structure. It needs to implement the Properties property. Currently the web test resource supports both URL ping test and multi-step test. We’re only dealing with URL ping test in this post.

Web Test Properties – URL Ping

So, we’re creating the PingWebTestProperties class inheriting the WebTestProperties class.

Note that WebTestProperties has an implicit operator that automatically convert the object to serialised JSON string. This is because GenericResource.Properties takes serialised JSON string value.

If you’re not familiar with implicit operator, this MSDN document will give you an insight.

Similarly, PingWebTestProperties has a property Configuration returning PingWebTestConfiguration. This has a WebTest property returning a serialised XML string like:

This time, we just use an extension method, ToXml(), but we can of course implement the same implicit operator for XML serialisation.

We’ve now got a strongly-typed object to create or update web test. Let’s move onto the next step.

Create or Update Web Test Resource

Azure SDK has ResourceManagementClient to handle Azure resources. With this, we can simply call the method to create a web test resource.

If the resource has been successfully created or updated, the result will return HttpStatusCode.Ok (200) or HttpStatusCode.Created (201). If you have a question around credentials, it’s worth noting the following part.

ClientCredential or UserCredential

It’s entirely up to the business decision that we should use either ClientCredential or UserCredential to access to the ResourceManagementClient object. If we need users to provide their username and password, we should use UserCredential; otherwise we should use ClientCredential. In either case, we should register an application to Azure Active Directory (AAD).

ClientCredential

Using ClientCredential requires ClientId and ClientSecret. Therefore, when we register an application to AAD, we should choose the WEB APPLICATION AND/OR WEB API option.

Then, add an application to the Permissions to other applications section like:

By doing so, we can access to Azure resources using ADAL without providing username and password like:

UserCredential

This time, let’s consider the UserCredential object. The ClientId value is only necessary for this scenario. But, if we use the existing application registered to AAD for ClientCredential, the following exception will be thrown and authentication fails:

Additional information: AADSTS90014: The request body must contain the following parameter: ‘client_secret or client_assertion’.

Hence, we need to register another application to AAD with the option of NATIVE CLIENT APPLICATION.

Then, add an application to the Permissions to other applications section like:

By doing so, we can access to Azure resoures using ADAL with username and password like:

So far, we’ve looked through how we can create a web test resource for Application Insights programmatically, using C# codes. Even though this post only dealt with a URL ping test, same approach can be used for multi-step test resources. If this is something you are interested in, this NuGet package is for you.

Azure Applications Insights for Xamarin iOS

Azure Application Insights (AI) is a great instrumentation tool that can help you learn about how your application is doing during run-time. It is currently in Preview mode, so bear that in mind when developing production ready apps. It gives you the ability to log lots of different kinds of information like tracing, page views, custom events, metrics and more.

Azure AI supports multiple platforms, but unfortunately they have not released a Xamarin package yet. There is one library which is for Xamarin.Forms since it uses the DependencyResolver. I have taken that and removed that dependency to make it compatible with Xamarin.iOS. You could do the same thing to use it for Xamarin.Android too if you like.

It’s very simple to use, all you need is your instrumentationkey which you can get from your Azure portal. Follow the steps from the MSDN tutorial as you can see here to create the Azure AI instance and get your key. Once done, you could download and reference the repository that I have created on GitHub, as you can find it here.

To start Azure AI on your Xamarin iOS app, you could do:

[code lang=”csharp”]

AzureAIManager.Setup();

AzureAIManager.Configure("my-user-or-device-name");

AzureAIManager.Start();

[/code]

The implementation of the AzureAIManager is as follows:

[code lang=”csharp”]

public static class AzureAIManager
{
public static void Setup(string appKey = "your-azure-AI-instrumentation-key")
{
AI.Xamarin.iOS.ApplicationInsights.Init();

var ai = new AI.Xamarin.iOS.ApplicationInsights ();
ApplicationInsights.Init (ai);

TelemetryManager.Init(new AI.Xamarin.iOS.TelemetryManager());
ApplicationInsights.Setup(appKey);

}

public static void Start()
{
ApplicationInsights.Start();
}

public static void Configure(string userId = "" )
{
ApplicationInsights.SetAutoPageViewTrackingDisabled(true);

if (string.IsNullOrEmpty(userId))
ApplicationInsights.SetUserId(userId);
}

public static void RenewSession()
{
ApplicationInsights.StartNewSession();
}
}

[/code]

I have not put this as a Nuget package because I am sure Microsoft will release one very soon, so until that happens, you can use this bindings to play around with Azure AI and you could even use it on your small projects.

Follow Us!

Kloud Solutions Blog - Follow Us!