Asynchronous jobs are usually hard to troubleshoot due to the very nature of its execution. This post talks about how we can monitor and trouble shoot Azure WebJobs both during development and when it is deployed on an Azure Web App. The key is to understand the layout of the logs the WebJob runtime creates during its execution.

WebJob storage accounts

To enable logging, WebJob needs two Azure storage account connection strings to be configured:

  • AzureWebJobsDashboard
  • AzureWebJobsStorage

AzureWebJobsDashboard

This storage account is primarily used by Azure WebJob SDK to store logs from the WebJobs Dashboard. This connection string is optional and is required only if you are planning to use the dashboard for monitoring WebJobs.

The WebJob runtime creates two containers under this storage account with the names ‘azure-webjobs-dashboard’ and ‘azure-jobs-host-archive’. The azure-webJobs-dashboard container is used by the WebJob dashboard to store host and execution endpoint (function) details. Azure-jobs-host-archive is used as an archive for execution logs.

AzureWebJobsStorage

AzureWebJobsStorage should point to a storage account which will be primarily used for logging. WebJob runtime creates two containers in this storage account with the names ‘azure-jobs-host-output’ and ‘azure-webjobs-host’. If you point AzureWebJobsDashboard and AzureWebJobsStorage  at two different storage accounts, you will notice that these two containers are duplicated in both the storage accounts.

azure-webjobs-host container in-turn hosts three directories:

  • Heartbeats – Containing 0 byte blogs for every heartbeat check performed on the service.
  • Ids – Containing the directory with a single blog holding a unique identifier for this service.
  • Output-logs – Hosts the output of the explicit logs for each run. Explicit logs being logs introduced by WebJob developers within the execution code.

azure-jobs-host-output container is the key for troubleshooting web jobs. This container hosts logs created by the WebJob runtime during initialization and termination of every execution.

Understanding azure-jobs-host-output (Storage Container)

To understand output logs, let’s consider the following code which defines a task to be executed by the WebJob.

When the WebJob is executed (either on a dev environment from Visual studio or by hosting within a WebApp), an output log is created in the azure-jobs-host-output container in the following format.

WebJob start log

Note: The maroon text below are comments explaining the JSON elements and are not part of the actual log file.

Once the execution is complete, a blob file in the following format is created by the WebJob runtime.

Using WebJob Dashboard

WebJob dashboard provides a detailed, user friendly interface to manage your WebJobs hosted on Microsoft Azure. The dashboard will be enabled once the Azure WebApp hosting the WebJob is deployed and will be accessible on the following path

https://<WebApp Name>.scm.azurewebsites.net/azurejobs/#/jobs

The dashboard can also be reached via Azure Portal.

namit_webjoblogs1

On the homes screen, dashboard will list the web jobs hosted within the selected WebApp and their current execution status.

namit_webjoblogs1

On clicking on a specific WebJob, dashboard will navigate you to a page displaying the list of executions for that WebJob along with their status.

namit_webjoblogs2

Each execution can be further expanded to see the explicit logging.

namit_webjoblogs4

For most cases, the logs displayed on the portal should supply sufficient information for us to troubleshoot the WebJob. Exploring the WebJob logs in Azure Storage Accounts can be used for deeper investigations.

References

Azure WebJobs SDK – https://github.com/Azure/azure-webjobs-sdk

MSDN Documentation – https://azure.microsoft.com/en-us/documentation/articles/app-service-webjobs-readme/

Category:
Application Development and Integration, Azure Platform
Tags:
, ,

Join the conversation! 3 Comments

  1. You code has
    log.WriteLine(“Process Something called at : ” + DateTime.Now.ToShortDateString() + ” , ” + DateTime.Now.ToShortTimeString());
    but log files do not have the line.
    Where it suppose to be?

    Reply
  2. […] Após a criação do projeto, precisaremos configurar duas “strings de conexão”, AzureWebJobsDashboard, que será utilizada para armazenar os logs gerados pelo WebJob – sendo que esses logs são gerados de forma automática, sem a influência da codificação feita pelo usuário -, e AzureWebJobsStorage, que fará o armazenamento também dos logs, mas o que foi gerado por intervenção da codificação do usuário. Ambas as strings de conexão devem estar relacionadas ao acesso ao storage, pois todas as informações geradas pelo WebJob são armazenadas em um storage, porém nada impede que o armazenamento das informações seja feito em storages distintos. Para mais detalhes, veja este artigo no kloud.com.au. […]

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: