Azure VM troubleshooting – part 1

It’s very easy to spin up new servers in Azure but what if one of those machines starts playing up? Gone are the days of F8’s, booting into safe mode or console access. Over the last couple of weeks I lost the ability to RDP into several virtual machines. Some of these machines were still pingable but no RDP, others didn’t respond at all. From within the portal the VM would still show a status of “Running”.

On the first machine I had an issue with I followed the “Troubleshoot” option in the portal and selected “I can’t connect to my Windows VM”.

  • Resource health reported all good
  • Boot diagnostics showed a black screen on some or just a spinning wheel on others (the latter might mean it’s stuck in a reboot loop).
    Capture3

Not really helpful.

To get VMs working again I had to use 1 of the 2 methods listed below. Having said that…..I did not get a 100% score on all broken VMs.

Option 1)
Use Redeploy.
This option is available through the portal and as part of the afore mentioned portal “Troubleshoot” option.
This will migrate the virtual machine to a new Azure host. The virtual machine will be restarted and any data on the temporary drive will be lost. While the redeployment is in progress, the virtual machine will be unavailable.
Once the portal shows the VM status “Running” test to see if you now can RDP into the VM.

Option 2)
Sit down, take a deep breath and delete the VM from the portal. Don’t worry, you’re not deleting the whole machine but merely the “config” file. The VM’s storage and NIC will still be there. Wait till the portal reports that the VM (let’s call it ‘FaultyServer’)  is deleted and then perform the following steps:

  1. In the portal open the settings for a working Azure VM and attach the OS disk from ‘FaultyServer’ to this machine (Disks > Attach existing)
    This will add the OS disk from ‘FaultyServer’  as drive E: to the working VM.
  2. Now open the registry editor,  highlight “HKEY_LOCAL_MACHINE” and select “Load hive” from the menu.
  3. Browse to E:\windows\system32\config and open SYSTEM.
  4. Give the new hive a name ie “BadServer”.
  5. Next make the following changes to the registry:
    HKEY_LOCAL_MACHINE\BadServer\Select\Current Change 1 to 2
    HKEY_LOCAL_MACHINE\BadServer\Select\Default Change 1 to 2
    HKEY_LOCAL_MACHINE\BadServer\Select\Failed Change 0 to 1
    HKEY_LOCAL_MACHINE\BadServer\Select\LastKnownGood Change 2 to 3

    Capture

  6. Next make the following change to the registry:
    HKEY_LOCAL_MACHINE\BadServer\ControlSet\Control\CrashControl Change to 0

    Capture2

  7. Highlight “HKEY_LOCAL_MACHINE” and select “Unload hive” from the menu.
  8. Open the settings for the working VM and detach the ‘FaultyServer’ disk from this machine (Disks > Detach)
  9. Once the portal reports that the disk has been detached, give it a couple of minutes before continuing with the next section.

After the changes have been made to the ‘FaultyServer’s OS disk, the VM can be recreated using the following PowerShell command:

$vm = New-AzureRmVMConfig -VMName VMNAME -VMSize INSTANCE_SIZE
$nic = Get-AzureRmNetworkInterface -Name (NAME_OF_NIC) -ResourceGroupName RESOURCE_GROUP
$nicId = $nic.Id
$vm = Add-AzureRmVMNetworkInterface -VM $vm -Id $nicId
$vm = Set-AzureRmVMOSDisk -VM $vm -VhdUri "URI_OF_DISK" -name "DISKNAME" -CreateOption attach -Windows
New-AzureRmVM -ResourceGroupName RESOURCE_GROUP -Location LOCATION -VM $vm

Wait for the rebuild to finish and test connectivity again.

Add workdays and public holidays to your Azure Automation runbooks

Azure Automation runbooks are a great way to automate various aspects of your Azure environment. Probably the task it is used for the most (at least at the moment) is starting and stopping a virtual machine or a bunch of vms on a schedule. The runbook scripts take care of the “powering on” or “shutting down”. Adding a schedule to these scripts is a great way to save cost by only running virtual machines when they are required. Some might be required to run 24/7 but others might only be required to run on workdays e.g. from 7am to 7pm .

However the runbook scheduler is pretty limited in fine tuning the recurrence of a script. Currently you can choose from

  • One time
  • Daily
  • Hourly

But what if you want to run a VM only on weekdays and only on weekdays which are not public holidays. The latter was one of our customers requirement. I’m not going to add another start/stop script to this post but just the code snippets/logic you can use in your own script.

Weekdays

This is the “easy” bit.

$Weekdays = @("Monday","Tuesday","Wednesday","Thursday","Friday")

# Check if day of the week is a working day (Monday - Friday). If not, exit.
If (-Not ($Weekdays -contains $CurrentDateTimeACDT.DayOfWeek))
{
   return
}


Public Holidays

When looking at ways to check for a public holiday I had a couple of options in mind on how to do this. Create an array in PowerShell with dates? Use a CSV to read from? As long as it not only contained the national as well as the state specific (in my case ‘SA’) holidays.
When looking at the CSV file option I ended at the federal government’s website “data.gov.au” and that’s where I noticed they offer this information in datasets which can be queried using RESTAPI calls. The good thing with being able to do that is that the information is always up to date and requires no human intervention to retrieve.

I have split up the URI to the dataset in a base part ($URI) and the actual resource part ($ResourceID). Next year, when the dataset for 2017-2018 is released this is the only bit which needs updating. Or could be passed on the runbook as a parameter.

function GetPublicHolidays
{
   # Get information regarding public holidays from the DATA.GOV.AU website using RESTAPI.
   # Australian Public Holidays Dates Machine Readable Dataset
   # https://data.gov.au/dataset/b1bc6077-dadd-4f61-9f8c-002ab2cdff10
   $URI = "https://data.gov.au/datastore/odata3.0/"
   $ResourceID = "a24ecaf2-044a-4e66-989c-eacc81ded62f" # 2016/2017 dataset

   # Retrieve all national (NAT) and South Australian (SA) public holidays.
   (Invoke-RestMethod -Uri "$($URI)$($ResourceID)?`$format=json" -Method Get).Value | `
      Where-Object {($_.ApplicableTo -eq "NAT" -or $_.ApplicableTo -eq "SA")}
}

Add the above function to the runbook and then from with the runbook call it:

$PublicHolidays = GetPublicHolidays | GetPublicHolidays | Where-Object {$_.Date -eq ($CurrentDateTimeACDT.ToString("yyyyMMdd"))}
If (-Not ([string]::IsNullOrEmpty($PublicHolidays)))
{
   Write-Output "Current day is a Public Holiday: [$($PublicHolidays.HolidayName)]. Script will not continue."
   return
}

 

When using time and dates in Azure Automation it is important to keep in mind that

a) Time and date entered through the portal when creating a schedule are entered in the timezone the webbrowser runs in. This information gets stored as UTC time and date in the background.

b) The runbook scripts use UTC time and date. So when using date and time functions in a runbook which also take information from other resources make sure that whatever is compared is in the same timezone.

$CurrentDateTimeUTC = (Get-Date).ToUniversalTime()
$CurrentDateTimeACDT = (Get-Date).ToUniversalTime().AddHours(10.5)

 

 

IoT – Solar & Azure

Ever since we got our solar system installed about two years ago, I’ve been keeping track of the total power generated by the system. Every month I would write down the totals and add it to my Excel spreadsheet. Although it’s not much work, it’s still manual work… yes all 2 minutes every month.

So when the whole “Internet of Things” discussion started at our office (see Matt’s blog “Azure Mobile Services and the Internet of Things“) I thought it would be a good opportunity to look at doing this using Azure – even if it was only to prove the IoT concept. The potential solution should:

  1. Use a device which connects to the solar inverter to reads its data via RS232.
  2. This device needs to be powered by a battery as no power outlet is close to the inverter.
  3. Upload data to Azure without having to rely on a computer running 24/7 to do this.
  4. Use Azure to store and present this data.

Hardware

The device I built is based on the Arduino Uno and consists of the following components:

Arduino UNO R3
With a little bit of programming these devices are perfectly capable of retrieving data from various data sources, are small in size, expandable with various libraries, add on shields and break-out boards and can be battery powered. Having the inverter on a side of the house with no power outlet close by made this a main requirement.
MAX3232 RS232 Serial to TTL Converter module
As the Arduino Uno doesn’t come with any serial connectors this module adds a DB9 connector to the board. Now the Arduino can be connected to the inverter using a null modem cable.
Adafruit CC3000 WiFi Shield with Onboard Ceramic Antenna
Some of the existing solutions which can send inverter data to a website (e.g. PVOutput) or computer logging those details, all rely on a computer which runs 24/7 which is one of the things I definitely didn’t want to do. I ended up getting this WiFi shield which, after soldering it on top of the Arduino board, turns the Arduino into a WiFi enabled device and allows it to send data to the internet directly. After adding the required libraries and credentials to my script, having access to a wireless router already enables basic access to the internet. Even though it is sitting quite a bit away from the wireless router, connectivity is no issue.
arduinobuild inverter
The Arduino Uno unit… …connected to the inverter

Azure

To store and / or display any of the info the Arduino is collecting, an Azure subscription is required. For this project I signed up for a free trial. Once the subscription is sorted, the following Azure services have to be setup:

Azure Service Description
Cloud service Running the worker roles.
Storage Account Hosting the table storage.
Service Bus Message queue for the Arduino.
Website For displaying data in (near) realtime.

Putting it all together

So how do all these different components fit together?

The Arduino connects to the inverter via a null-modem cable. Reading data from it is achieved by adding a MODBUS library to the Arduino script. This adds additional functionality to the Arduino which is now able to read (and write) data from MODBUS (an industrial comms standard) enabled devices.
The script is set to run every 30 minutes and only after a successful connection (the inverter shuts down if there is not enough sunlight) it will set up a wireless internet connection and send the data to the TCP listener worker role in Azure.

In Azure, a service bus message queue was created to hold all incoming data packets sent from the Arduino. A storage table was also created to permantly store data received from the Arduino. The great thing with the storage table is there is no need to create a table schema before being able to use it, just creating the “placeholder” is enough!

Using Visual Studio, two worker roles were created:

  • A TCP listener which “listens” for any device sending information to the specified endpoints. If a message from the Arduino is received it will write it onto the message queue.

service bus explorer screenshot

Using Service Bus Explorer you can see the individual messages arriving in the message queue.

  • A data writer which checks the message queue for new messages. If a new message has arrived, the message will be read, its content stored in the storage table and the message deleted.

Finally, a simple ASP.Net MVC website is used to display data from the storage table in near real-time. The website displays statistics on how many KWs have been generated during that day and how a day compares to previous days.

Energy Today

Stats for current day.

solarduino-website

Website display.

Conclusion

This IoT project was a good opportunity to have a play with various Azure components through using multiple worker roles, message queues and the like. It probably sounds like overkill when just using the one device sending one message every 30 minutes, but a similar setup can be used in larger environments such factories where multiple devices send dozens of messages per minute.

Exchange 365 – Transport Rules & Distribution Groups

One of our customers is transitioning from on premise Exchange 2010 to a hybrid Exchange 365 (wave 15) environment and user management for Office 365 done through on premise Active Directory. Customer had quite a few transport rules setup up which needed to be migrated. This worked fine except for the rules using a “redirect the message to” action using a distribution group.

The error displayed in Exchange 365 generated is: The transport rule can’t be created because TR-Marketing@Company.com, the recipient to be added by a rule action, is a distribution group. Transport rules can’t add distribution groups to messages.

Checked with Microsoft Support and they confirmed that this is an issue.

Workaround

There might be better ways of doing this but I wanted to make sure that no additional Office 365 licenses were required hence the use of shared mailboxes (which are free as long as you stay within a 5Gb size limit – which shouldn’t be a problem as the mailbox is purely used for forwarding). In the example workaround below I’m using the following (account) names:

  • Marketing@Company.com as the Distribution group used in the original Exchange 2010 transport rule
  • TR-Marketing@Company.com as the account to be used in the Exchange 365 transport rule
  • Company:ENTERPRISEPACK as the Office 365 AccountSkuId
  1. Using the EMC create a normal user + mailbox: TR-Marketing@Company.com (Make sure name is meaningful and set a description for future reference.)
  2. On the properties General tab, select “Hide from Exchange address lists”
  3. Wait for DirSync to synchronise the account or manually kick off a sync.
  4. Assign an Office 365 license (this will only be temporary) to TR-Marketing@Company.com either using the EAC or the following PowerShell command:

    Set-MsolUserLicense -UserPrincipalName “TR-Marketing@Company.com” -AddLicenses “Company:ENTERPRISEPACK”

  5. From the EMC move the TR-Marketing mailbox to Exchange 365.
  6. Convert the ‘normal’ mailbox to a ‘shared’ shared mailbox using the following PowerShell command:

    Set-Mailbox -Identity “TR-Marketing@Company.com” -Type “Shared” -ProhibitSendReceiveQuota 5GB -ProhibitSendQuota 4.75GB -IssueWarningQuota 4.5GB

    As this shared mailbox is used as a forwarding address, I didn’t set any mailbox or recipient permissions.

  7. Remove the Office 365 license assigned previously assigned either using the EAC or PowerShell:

    Set-MsolUserLicense -UserPrincipalName TR-Marketing@Company.com -RemoveLicenses “Company:ENTERPRISEPACK”

  8. From either the EMC or EAC set the accounts forwarding address to Marketing@Company.com. Do NOT set Deliver message to both forwarding…..
  9. Assign TR-Marketing@Company.com to the “redirect the message to” action in the Exchange 365 transport rule.

Note: As user accounts are managed through on premise AD, do NOT delete the user created in Step 1.

As I said, there might be better or easier ways of doing this. Please comment if there are any.