Azure WebJob logs demystified

Asynchronous jobs are usually hard to troubleshoot due to the very nature of its execution. This post talks about how we can monitor and trouble shoot Azure WebJobs both during development and when it is deployed on an Azure Web App. The key is to understand the layout of the logs the WebJob runtime creates during its execution.

WebJob storage accounts

To enable logging, WebJob needs two Azure storage account connection strings to be configured:

• AzureWebJobsDashboard
• AzureWebJobsStorage

AzureWebJobsDashboard

This storage account is primarily used by Azure WebJob SDK to store logs from the WebJobs Dashboard. This connection string is optional and is required only if you are planning to use the dashboard for monitoring WebJobs.

The WebJob runtime creates two containers under this storage account with the names ‘azure-webjobs-dashboard’ and ‘azure-jobs-host-archive’. The azure-webJobs-dashboard container is used by the WebJob dashboard to store host and execution endpoint (function) details. Azure-jobs-host-archive is used as an archive for execution logs.

AzureWebJobsStorage

AzureWebJobsStorage should point to a storage account which will be primarily used for logging. WebJob runtime creates two containers in this storage account with the names ‘azure-jobs-host-output’ and ‘azure-webjobs-host’. If you point AzureWebJobsDashboard and AzureWebJobsStorage  at two different storage accounts, you will notice that these two containers are duplicated in both the storage accounts.

azure-webjobs-host container in-turn hosts three directories:

• Heartbeats – Containing 0 byte blogs for every heartbeat check performed on the service.
• Ids – Containing the directory with a single blog holding a unique identifier for this service.
• Output-logs – Hosts the output of the explicit logs for each run. Explicit logs being logs introduced by WebJob developers within the execution code.

azure-jobs-host-output container is the key for troubleshooting web jobs. This container hosts logs created by the WebJob runtime during initialization and termination of every execution.

Understanding azure-jobs-host-output (Storage Container)

To understand output logs, let’s consider the following code which defines a task to be executed by the WebJob.

When the WebJob is executed (either on a dev environment from Visual studio or by hosting within a WebApp), an output log is created in the azure-jobs-host-output container in the following format.

WebJob start log

Note: The maroon text below are comments explaining the JSON elements and are not part of the actual log file.

Once the execution is complete, a blob file in the following format is created by the WebJob runtime.

Using WebJob Dashboard

WebJob dashboard provides a detailed, user friendly interface to manage your WebJobs hosted on Microsoft Azure. The dashboard will be enabled once the Azure WebApp hosting the WebJob is deployed and will be accessible on the following path

https://<WebApp Name>.scm.azurewebsites.net/azurejobs/#/jobs

The dashboard can also be reached via Azure Portal.

On the homes screen, dashboard will list the web jobs hosted within the selected WebApp and their current execution status.

On clicking on a specific WebJob, dashboard will navigate you to a page displaying the list of executions for that WebJob along with their status.

Each execution can be further expanded to see the explicit logging.

For most cases, the logs displayed on the portal should supply sufficient information for us to troubleshoot the WebJob. Exploring the WebJob logs in Azure Storage Accounts can be used for deeper investigations.

References

Azure WebJobs SDK – https://github.com/Azure/azure-webjobs-sdk

MSDN Documentation – https://azure.microsoft.com/en-us/documentation/articles/app-service-webjobs-readme/