A tale of two products (don’t expect Dickens)

At Re:Invent and just after, AWS released several new products. Included in those were AWS FSx Windows and AWS Backup. Both of these products had a lot of interest for me, for various reasons, so I thought I’d give them a try. None of my experience was under work conditions, but the following are my experiences. Note: Both are only in a small number of regions, currently.

AWS FSx Windows

Pros:

  • Easy setup (by itself)
  • Fully compatible Windows file server
  • DFS support
  • Has backups
  • Works as expected

Cons:

  • Requires AWS Microsoft AD in each VPC
  • Can’t change file share size
  • Some features can only be changed from CLI
  • Throughput can only be changed through restore
  • Minimum share size is 300GB

First out of the box, and released at Re:Invent is AWS FSx Windows. AWS Elastic File System has been around for a while now and works nicely for providing a managed NFS share. Great for Linux, not so good for Windows. Your Windows sharing options are now enhanced with AWS FSx Windows. This is an AWS managed Windows File Server, running on Windows Data Centre. When you go to do the setup, there are only a few options, so it should be pretty easy, right? Well, yes and no. Actually configuring FSx Windows is easy, but before you do that, you need to have an AWS Microsoft AD directory service (not just an EC2 running AD) in the VPC that you are launching FSx Windows. If you’re running Windows based workloads, you’ll likely have a Windows admin, so this shouldn’t be too hard to configure and tie into your regular AD. OK, so, I’ve got my AWS Window AD, what’s next? Well, jump into the FSx Windows console, enter the size, throughput (default 8MB/s), backup retention and window, and maintenance window. That’s it. You now have a Windows file share you can use for all your Windows file sharing goodness.

FSx Windows FS Creation

But, what’s the catch? It can’t be that easy? Well, mostly it is, but there are some gotchas. With EFS, you don’t specify a size. You just use space and Amazon bill you for what you use. If you throw a ton of data on the share, all good. It just grows. With FSx Windows, you have to specify a size, min 300GB, at creation. Right now, there is no option to grow the share. If you run out of space, you’ll need to look at DFS (fully supported). Create a new FSx Windows share and use your own DFS server to manage merging the two shares into a single namespace. While on the topic of DFS, FSx Windows is just single AZ. If you want redundancy, you’ll need to create a second share and keep the data in sync. AWS has a document on using DFS for a multi-AZ scenario: https://docs.aws.amazon.com/fsx/latest/WindowsGuide/multi-az-deployments.html

What other issues are there? Well, a semi-major one and a minor one. The major one is there is currently no way to easily change the throughput, or at least not that I’ve found. When doing a restore, you can choose the throughput, so you can restore with a new throughput, blow away the old share and then map to the new one. Not terrible, but a bit painful. For backup changes, time and retention, or scheduled maintenance window changes, those can only be done via CLI. It would be nice to have that option in the console, but not really a huge deal.

And there you have it. An AWS managed Windows File Server. Having to rely on an AWS Microsoft AD in each VPC is a bit of pain, but not terrible. Overall, I think this is a great option. Once it rolls out to GA it’s definitely worth investigating if you currently have EC2 instances running as Windows File Servers.

AWS Backup

Pros:

  • Easy EFS backup
  • RDS & Dynamo DB backup

Cons:

  • No EC2 backup (EBS isn’t the same)
  • Dashboard is fairly basic

Last year AWS launched their first foray into managed backups with Data Lifecycle Manager (DLM). This was very basic and helped with the scheduling and life cycle management of EBS snapshots. The recent announcement of AWS Backup sounded like this would be a big step forward in AWS’s backup offering. In some ways it is, but in others it is still very lacking. I’ll start with the good, because while these are great, they are relatively quick to cover. RDS and DynamoDB expands on what is already offered past the traditional 35-day mark. The big surprise, and much needed feature, was support for EFS backups. Previously, you had to roll your own, or use an AWS Answer provided by AWS to do an EFS to EFS backup. It worked, but was messy. This option makes it really easy to backup an EFS volume. Just configure it into a Backup Plan and let AWS do the work! There’s not a lot to say about this, but it’s big, and may be the main reason people include AWS Backup in their Data Protection strategy.

AWS Backup - Dashboard01

Now for the bad; there is no EC2 backup or restore. But wait! What about the EBS backup & restore option? That’s OK, but really, how many people spin up EBS volumes and only want to protect the volume? You don’t. You spin up an EC2 instance (with attached volumes) and want to protect the instance. Sadly, this is where AWS Backup falls down and products like Cloud Protection Manager from N2W Software shine. When you have an issue that requires restoring an EC2 instance from backup, it’s generally not in a calm situation with plenty of time to stop and think. There’s a disaster of some form and you need that server back now! You don’t want to have to restore each EBS volume, launch an instance from the root volume, make sure you know what volumes were attached in what order and with what device name and have all the details of security groups, IPs, etc. You just want to hit that one button that says “restore my instance” and be done. That’s what tools like CPM give you and what is sorely lacking in this offering from AWS Backup.

Summary

What’s my wrap up from my “Tale of two products”? For FSx Windows, if you have a need for a Windows File Server, wait until it comes to your region and go for it. With AWS Backup, it has a place, especially if you have important data in EFS, but it’s no replacement for a proper backup solution. Most likely this will be implemented in a hybrid arrangement with another backup product.

Note: AWS Backup also manages backups of AWS Storage Gateway. I didn’t have one to test, so I won’t comment.

Backups? Doesn’t Amazon handle that?

For many, the cloud is a magical place where servers just appear and your cloud provider looks after everything, or, if they at least have a concept of the servers, they just assume that the provider will also back them up. Lots of people never bothered to think about protection in a VMware environment, so why start now?

Unfortunately, while your cloud provider probably supplies the tools, you still need to do the configuration and management. In this blog, I won’t talk in specifics about tools, but I’ll start with discussing the concepts around backups. I’ll be discussing these concepts in relation to AWS, but the IaaS elements are roughly the same in Azure too.

Oh crap! I didn’t mean to blow away that server!
This is the time when you really need to find out if your backups are configured correctly, and by that, I don’t mean “are they working”, but are they backing up the right servers, at the right times, with the right frequency. There are two big buzz terms when designing backup policies: Recover Time Objective (RTO) and Recover Point Objective (RPO).

RTO is the “how quick can I get my data back” question. This is the question that governs how you back up your data. AWS natively has snapshots for volumes, plus backup options for RDS, DynamoDB, Aurora, and Redshift. They also have Lifecycle Management (DLM) for volume snapshots. There are all sorts of options in this space, from using the native tools; roll your own using Lambda, Step Functions, SSM, etc.; or backup products from vendors using their own methods or better frontends around AWS APIs. While tools like DLM & lambda functions are cheap and may backup well, the ease, and hence speed, with which restores are done may not be the fastest.

RPO is the “how much data can I afford to lose” question. For things like app servers, or even file servers, your standard nightly backup is often sufficient. For infrastructure like database servers, some logging servers, time sensitive data, then more frequent backups are often required. Often times, it’s a mix of the two, e.g. full nightly backup of the database and hourly archive log backups, or even writing the archive logs to a secondary location, S3, upon completion.

RTO & RPO are vital to your DR, or operational recovery. They answer the questions of how fast will you get your infrastructure back and at what point your data will be. The next set of questions are for historical data recovery. This is a mixture of questions around regulations, business processes, and cost. If your industry has certain legal requirements, those need to be allowed for. The longer you keep data, the more it’s going to cost. Also, how you keep your data will have an impact on cost. Is it snapshots, is it EBS, is in in some virtualised dedupe device, or S3/Glacier?

In summary, backups of data is still the customer’s responsibility, even S3. If you are going to backup your data, it is worth taking the time to plan it properly and regularly review that plan. Think of it as business insurance, along the lines of, if I lost this data, can the business survive?

Automate the nightly backup of your Development FIM/MIM Sync and Portal Servers Configuration

Last week in a customer development environment I had one of those oh shit moments where I thought I’d lost a couple of weeks of work. A couple of weeks of development around multiple Management Agents, MV Schema changes etc. Luckily for me I was just connecting to an older VM Image, but it got me thinking. It would be nice to have an automated process that each night would;

  • Export each Management Agent on a FIM/MIM Sync Server
  • Export the FIM/MIM Synchronisation Server Configuration
  • Take a copy of the Extensions Folder (where I keep my PowerShell Management Agents scripts)
  • Export the FIM/MIM Service Server Configuration

And that is what this post covers.
https://dl.dropboxusercontent.com/u/76015/BlogImages/MIMBackupFunction/output.PNG

Overview

My automated process performs the following;

  1. An Azure PowerShell Timer Function WebApp is triggered at 2330 each night
  2. The Azure Function App initiates a Remote PowerShell session to my Dev MIM Sync Server (which is also a MIM Service Server)
  3. In the Remote PowerShell session the script;
    1. Creates a new subfolder under c:\backup with the current date and time (dd-MM-yyyy-hh-mm)

https://dl.dropboxusercontent.com/u/76015/BlogImages/MIMBackupFunction/Exports.PNG

  1. Creates further subfolders for each of the backup elements
    1. MAExports
    2. ServerExport
    3. MAExtensions
    4. PortalExport

https://dl.dropboxusercontent.com/u/76015/BlogImages/MIMBackupFunction/Folders.PNG

    1. Utilizes the Lithnet MIIS Automation PowerShell Module to;
      1. Enumerate each of the Management Agents on the FIM/MIM Sync Server and export each Management Agent to the MAExports Folder
      2. Export the FIM/MIM Sync Server Configuration to the ServerExport Folder
    2. Copies the Extensions folder and subfolder contexts to the MAExtensions Folder
    3. Utilizes the FIM/MIM Export-FIMConfig cmdlet to export the FIM Server Configuration to the PortalExport Folder

Implementing the FIM/MIM Backup Process

The majority of the setup to get this to work I’ve covered in other posts, particularly around Azure PowerShell Function Apps and Remote PowerShell into a FIM/MIM Sync Server.

Pre-requisites

  • I created a C:\Backup Folder on my FIM/MIM Server. This is where the backups will be placed (you can change the path in the script).
  • I installed the Lithnet MIIS Automation PowerShell Module on my MIM Sync Server
  • I configured my MIM Sync Server to accept Remote PowerShell Sessions. That involved enabling WinRM, creating a certificate, creating the listener, opening the firewall port and enabling the incoming port on the NSG . You can easily do all that by following my instructions here. From the same post I setup up the encrypted password file and uploaded it to my Function App and set the Function App Application Settings for MIMSyncCredUser and MIMSyncCredPassword.
  • I created an Azure PowerShell Timer Function App. Pretty much the same as I show in this post, except choose Timer.
    • I configured my Schedule for 2330 every night using the following CRON configuration

0 30 23 * * *

  • I set the Azure Function App Timezone to my timezone so that the nightly backup happened at the correct time relative to my timezone. I got my timezone index from here. I set the  following variable in my Azure Function Application Settings to my timezone name AUS Eastern Standard Time.

    WEBSITE_TIME_ZONE

The Function App Script

With all the pre-requisites met, the only thing left is the Function App script itself. Here it is. Update lines 2, 3 & 6 if your variables and password key file are different. The path to your password keyfile will be different on line 6 anyway.

Update line 25 if you want the backups to go somewhere else (maybe a DFS Share).
If your MIM Service Server is not on the same host as your MIM Sync Server change line 59 for the hostname. You’ll need to get the FIM/MIM Automation PS Modules onto your MIM Sync Server too. Details on how to achieve that are here.

Running the Function App I have limited output but enough to see it run. The first part of the script runs very quick. The Export-FIMConfig is what takes the majority of the time. That said less than a minute to get a nice point in time backup that is auto-magically executed nightly. Sorted.
https://dl.dropboxusercontent.com/u/76015/BlogImages/MIMBackupFunction/FunctionOutput.PNG
 

Summary

The script itself can be run standalone and you could implement it as a Scheduled Task on your FIM/MIM Server. However I’m using Azure Functions for a number of things and having something that is easily portable and repeatable and centralised with other functions (pun not intended) keeps things organised.
I now have a daily backup of the configurations associated with my development environment. I’m sure this will save me some time in the near future.
Follow Darren on Twitter @darrenjrobinson
 
 
 

Leveraging Cloud Storage for the Enterprise: Microsoft StorSimple – Part 1

Originally posted on Bobbie’s blog @ www.thecloudguy.info

It’s no secret that one of the biggest pain points for enterprises today is the rapid growth of unstructured data. The ability to manage, protect and archive an organisation’s most valuable assets is arguably one of the biggest strains on IT department budgets.

The advent of cloud technology has many organisations looking for a way to leverage Pay-as-You-Go cloud storage offerings to assist in the data life-cycle process. The difficulty with these offerings is that data is stored as objects rather than on file systems such as NFS and CIFS, meaning integration with existing business processes and solutions isn’t straight forward.

Cloud Storage Gateways

Cloud Storage Gateways resolve integration issues by bridging the gap between commonly used storage protocols such as iSCSI/CIFS/NFS and cloud-based object storage. Storage Gateways take the form of a network appliance, server or virtual machine that resides on-premises and typically provide storage capacity for frequently used data.

As data ages and is accessed less frequently, it is moved into cloud storage which can cost considerably less than traditional on-premises storage offerings. Additional features are integrated into cloud Storage gateways such as backup technology to protect large volumes that can no longer be protected using traditional means.

Microsoft StorSimple

Microsoft have a competitive hybrid cloud storage offering called Microsoft StorSimple that takes into consideration a wide range of existing business pain points such as backup and Disaster Recovery.

Microsoft StorSimple is a physical on-premises storage system that uses three tiers of storage: SSD, SAS, and cloud storage. A number of models are offered based on storage and performance requirements, however StorSimple’s ability to leverage cloud storage as a cold tier significantly reduces its on-premises footprint compared to other storage offerings.

Some of the main features of StorSimple include:

  • Storage tiering – StorSimple dynamically moves data between the tiers of disk based on how active data is, providing efficient data storage and maximum performance for heavily used data.
  • iSCSI protocol support – StorSimple volumes are provisioned as block storage, allowing them to be used for file, database, or hypervisor attached storage.
  • Encryption in-flight and at rest in the cloud.
  • High capacity – The largest StorSimple appliance can currently store up to 500TB of deduplicated data (including cloud storage) in eight rack units.
  • Snapshot backups – Traditionally, snapshots were not considered a reliable form of backup due to their reliance on the source physical storage appliance, however StorSimple snapshots are stored in geographically redundant storage accounts in Microsoft Azure, meaning six copies of data are stored across two geographically separate regions.
  • Single pane management – All StorSimple devices in an organisation, regardless of location, can be managed from the same interface in the Azure portal.
  • Near instant DR – In the event of a disaster, a backup can be cloned and mounted on a virtual or physical StorSimple device and brought online. Only a fraction of the volume needs to reside on the target StorSimple for the volume to come online.
  • Virtual StorSimple – Virtual StorSimple devices can be provisioned in Azure to provide DR and test capabilities for volumes that were previously, or are currently, hosted on-premises.
  • Deduplication and compression – Microsoft StorSimple is able to minimize disk footprint by using global deduplication and compression across on premise and cloud storage.
  • Highly available physical storage architecture with dual components, to prevent single point of failure.

StorSimple in Action

Azure Portal Dashboard and Device Management

All StorSimple devices are managed from the familiar Azure Portal. The sample below shows five StorSimple devices with four being virtual and residing in Azure, and one a physical StorSimple 8100 running on-premises.

StorSimple Manager in Azure Portal

Each device can be selected and configured via the Portal and shown below where I am managing the network interface configuration for a physical StorSimple 8100 device.

Firstly we have the configuration of the WAN interface which is used to communicate with the cloud storage in Azure:

Managing WAN interface for StorSimple

Secondly I can manage the iSCSI interface used to connect storage to local servers (note that the StorSimple 8000 series offers multiple redundant 10GbE iSCSI interfaces, however the lab used for this blog post only has 1GbE switching infrastructure)

Managing iSCSI interface for StorSimple

Storage Provisioning

Compared to traditional storage systems, provisioning storage is incredibly simple (no pun intended!) Once a device is selected, the administrative user navigates to the Volume Containers menu and selects to add a Container (shown below).

Provisioning Storage on StorSimple

A Volume Container is used to group Volumes that share a storage account, bandwidth, and encryption key.

Best practice suggests a geographically redundant storage account is used in Azure to ensure data is highly available in the event of a regional disaster. Bandwidth can be throttled to ensure the WAN link is not saturated when tiering to cloud storage. If cloud storage encryption is used, an encryption key must be specified.

Once confirmed, the Volume Container is created and appears in the list of Containers for the device as shown below.

Creating a Volume Container on a StorSimple

A Volume can then be added to the Container:

Adding a Volume to a Container on a StorSimple

Notice that a usage type of “Tiered Volume” or “Archive Volume” can be selected which allows the StorSimple appliance to better judge where to initially place the data that is being moved to the Volume.

This can be handy for organisations that are looking to migrate stale data they are required to keep for compliance purposes to the cloud. Also note the capacity specified is always thin provisioned.

After confirming the basic settings, iSCSI initiators (servers) are specified that are allowed to access the volume. Once this is completed, the volume appears in the Volume Container page.

New Volume in Container on StorSimple

The Volume can now be attached to a host via the iSCSI protocol. Assuming that iSCSI connectivity is already configured, we log onto the server and perform a scan for new disks, which discovers the Volume recently provisioned as highlighted below.

StorSimple Volume available to Windows Host

This Volume can now be brought online, initialised, be assigned a drive letter, and then function as a drive on the server as shown below. Pretty simple stuff!

StorSimple Volume mounted as drive on server

One of the biggest benefits of StorSimple is its ability to provision traditional block storage which can be leveraged by familiar operating systems. Many other platforms offer storage in NAS form, requiring administrators to learn and manage another platform.

Data Protection and Backup

Now that we’ve provisioned a file system, how do we protect that data? Snapshots are used to protect data on a StorSimple in an efficient and reliable manner. Due to global deduplication, snapshots consume minimal storage in the cloud while still providing reliable protection due to Azure’s geographically redundant storage.

StorSimple backup policies and data protection is configured in the Azure Portal. Administrators navigate to the backup policies section of the device and add a new policy. Multiple Volumes can be grouped together within a policy to take crash-consistent snapshots across the multiple volumes.

Defining a backup policy on a StorSimple

A schedule is then defined. A policy can only have one schedule, however multiple policies can be defined for a Volume.

For example: a daily backup policy can be used perform daily snapshots of a Volume and retain them for short periods of time, while a monthly backup policy can take a snapshot of the same Volume once a month and retain snapshots long term for compliance purposes.

Additionally, a snapshot can be stored on either local storage for rapid restores or cloud storage for resiliency.

Defining backup schedule on a StorSimple

Once the schedule is defined, it appears in the backup policies tab.

Backup policies tab for a StorSimple

Although the first cloud snapshot can take some time, as all Volume data that resides on-premises needs to be copied to the cloud, all subsequent snapshots are quick, as only the changed data is deduped and copied to cloud storage.

Below is a view from the backup catalog, after a backup is complete.

StorSimple backup catalog view

Well, that’s it from me for now! Stay tuned for part two, where I will dive deeper into disaster recovery scenarios, data mobility and performance monitoring.

Originally posted on Bobbie’s blog @ www.thecloudguy.info

Follow Us!

Kloud Solutions Blog - Follow Us!