Office 365 Import Service via PowerShell
Rate this post

UPDATE 10/02/2017

Ok, so sorry everyone, I’ve been a bit slack with this one and Microsoft have made some significant changes in this space since I blogged on it. I thought it best to get this page updated so anyone who googled it would have current info!

Firstly, Microsoft have changed the BLOB they give you for the ingestion service to write once. This of course means if you don’t place things in the right location (folder for example) it’s not going anywhere! While a little inconvenient, I think it’s still thoroughly acceptable, they are still giving you unlimited BLOB storage at no cost for you to upload and ingest your PST’s. While I have read other blogs which have talked about signing up for your own blob (which will work really well with my original blog listing) I’m not sure it’s really necessary for the majority of customers as it will incur cost.

Also, I found that the old ‘Azure Storage Explorer 6’ wasn’t able to interact with the new BLOB so you couldn’t visualise the uploads with a GUI. Well they have released a new version in preview with the 6 dropped that can be downloaded here instead, oh and it’s really straight forward to use!

Finally Microsoft now recommend using the GUI and I must admit it, overall, it works pretty well. I used it for a recent project and I found it far superior to what it was originally. So I found I automated the AzCopy commands and the CSV generation but then I loaded the CSV and ran this import via the GUI. If anyone wants to see some examples, leave some comments below and I’ll write a new blog on it in the coming weeks! Original blog below…


If you’ve been involved in an Exchange Migration to Office 365 of late, I’m sure you’ll be well aware of the new Microsoft Office 365 Import Service.  Simply put, give them your PST’s via hard disk or over the network, and they’ll happily ingest them into the mailbox of your choice – what’s not to like!  If you’ve been hiding under a rock, the details are available here!

Anyway, rather than marvelling at the new Microsoft offering, I thought I’d share my recent experience with the network route option.  When I first started playing [read testing] I soon noticed that the Portal GUI was, well, struggling to keep up with the requests.  So instead, I reached out to MS and found that this is all possible programmatically using PowerShell!

So first of all, you want to get the files into an Azure Blob.  That’s achieved via AzCopy which is available on this link.  Then you’ll want to grab the Azure Storage Explorer, granted, you could probably do this all via PowerShell, but when there is a good GUI sometimes it’s just easier, it’s available here.

When you fire up the Storage Explorer, you’re looking for the Storage Key.  This is available via the Portal | Admin | Import.  From there, you’ll see a key icon which is referred to as ‘Copy Office 365 Storage Key and URL‘.  Once you’re connected you should see some default folders which I have screenshot below…

Blob Storage

Blob Storage

Select one and click Security, select the Shared Access Key tab and generate Signature.  So now we’ve got the necessary items, we can plumb them into variable as I’ve demonstrated below…

cd "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy"

$Source = "\\Server\Share"
$Destination = "https://55f6dd****49d9a5bcf36.blob.core.windows.net/ingestiondata/"
$StorageKey = "9XAsHc/OmkZgdrSXkE+***+njq9qQWvt*******rXysz//CvB2j6mlb6XSJgi4bfh+Br6Onn*****AEVtA=="
$LogFile = "C:\Users\dyoung\Documents\Uploads.log"

& .\AzCopy.exe /Source:$Source /Dest:$Destination /DestKey:$StorageKey /S /V:$LogFile

This will grab all the PST’s from the \ \Server\Share UNC path and look to upload them to the ingestiondata directory. There is a log file which is pretty useful and the PowerShell window will let you know once the upload is complete.

Now we’ve got our PST’s up in Azure, but we need to let MS know where to ingest them. For this, we need to generate a CSV. When I was creating the PST names I found it good to name them ADDisplayName.pst, simply so we can automate the CSV creation.  I also added incremental numbers at the end of each file, as it’s possible to insert multiple PST’s to a single mailbox…

e.g. ‘Dave Young.pst‘, ‘Dave Young1.pst‘ and ‘Dave Young2.pst‘ would all be imported into Dave Young’s mailbox.

I also decided to import to the users archive mailbox (as I’ve already given the users an archive) and I wanted the PST’s in the root directory…these options are all covered in the TechNet article here

$FilePath = "/"

$AllPSTs = Get-ChildItem '\\Server\Share' -filter "*.pst"

$CSVTable= @()
for($Cycle=0; $Cycle -lt $AllPSTs.name.Count; $Cycle++)
 {
 $DisplayName = [io.path]::GetFileNameWithoutExtension($AllPSTs.name[$Cycle])
 if ($DisplayName -match "[1-9]")
 {
 $DisplayName = $DisplayName -replace ".$"
 }
 $SMTPAddress = $null
 $Mailbox = Get-Mailbox $DisplayName
 $SMTPAddress = $Mailbox.PrimarySmtpAddress.ToString()
 $SMTPAddress = $SMTPAddress.ToLower()
 $AzureBSAccountUri = 'https://55f6ddf********a5bcf36.blob.core.windows.net/ingestiondata/'+$AllPSTs[$Cycle].Name
 $CSVColumn = New-Object PSObject
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'Workload' -Value 'Exchange'
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'FilePath' -Value $FilePath
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'Name' -Value $AllPSTs.name[$Cycle]
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'Mailbox' -Value $Mailbox.UserPrincipalName
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'AzureSASToken' -Value '?sv=2014-02-14&sr=c&sig=2dSSbF9CsQ******HpwHOJy4pmGFvsniv46hZh%2BA%3D&st=2016-02-01T13%3A30%3A00Z&se=2016-02-09T13%3A30%3A00Z&sp=r'
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'AzureBlobStorageAccountUri' -Value $AzureBSAccountUri
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'IsArchive' -Value 'TRUE'
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'TargetRootFolder' -Value '/'
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'SPFileContainer' -Value ''
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'SPManifestContainer' -Value ''
 Add-Member -InputObject $CSVColumn -MemberType NoteProperty -Name 'SPSiteUrl' -Value ''
 $CSVTable += $CSVColumn
 }

$CSVTable | Export-Csv "PstImportMappingFile-$((Get-Date).ToString("hh-mm-dd-MM-yy")).csv" -NoTypeInformation

Hopefully the output of this script is a pretty decent csv.  Now it’s time to create the import jobs.  Now you might be wondering, there is no ‘New-o365MailboxImportRequest‘ cmdlet and you’d be right.  I’ve used a suffix so I could load both on-premises and cloud cmdlets into the same PowerShell session, it’s achieved by ‘Import-PSSession $Session -Prefix o365′ which is described in point 2 here

Add-Type -AssemblyName System.Windows.Forms
$FileBrowser = New-Object System.Windows.Forms.OpenFileDialog -Property @{
InitialDirectory = [Environment]::GetFolderPath('MyDocuments')
Filter = 'CSV (*.csv)|*.csv'
}
[void]$FileBrowser.ShowDialog()
$FileBrowser.FileNames

Import-Csv $($FileBrowser.SafeFileName) | foreach {New-o365MailboxImportRequest -Mailbox $_.Mailbox -AzureBlobStorageAccountUri $_.AzureBlobStorageAccountUri -BadItemLimit unlimited -AcceptLargeDataLoss -AzureSharedAccessSignatureToken $_.AzureSASToken -TargetRootFolder $_.TargetRootFolder -IsArchive}

These jobs will be in PowerShell and they can of course be checked using the cmdlets ‘Get-MailboxImportRequest’ & ‘Get-MailboxImportRequestStatistics’.

And that’s pretty much it, yes it was quick but I reckon it’s pretty self explanatory and it’s considerably more efficient than doing it via the GUI – Enjoy!

Cheers,
Dave

Category:
Exchange, Office 365, Uncategorized

Leave a Reply

  Subscribe  
newest oldest most voted
Notify of
Tony
Guest
Tony

Thanks a ton for sharing this–this it the only thing I’ve found about how to automate O365 email ingestion. We have several thousand mailboxes, so getting the new-mailboximportrequest to work will save a lot of busy work. I am getting a (403) Forbidden error from the New-MailboxImportRequest command. My SASToken works in a brower (I’ve tried a toke both for the directory and for a specific file). Let me know if you have any ideas about what may be causing it.

Nicke Källén (@znackattack)
Guest

Hello, I have imported a session for O365 and can access quite a few new cmdlets, such as New-O365Mailbox, however the New-o365MailBoxImportRequest doesn’t seem to be available. As I am not part of the team that normally manages the Exchange / O365 tenant I potentially see this as a case of my account missing access rights, or in the other part of my mind it may be because the tenant doesn’t have this functionality yet. Apart from directly engaging Microsoft support, are there any gotchas that potentially could have simply gotten the best of me? I did receive the following… Read more »

Diane
Guest
Diane

In order to see those cmdlets, your account needs to be a member of organization management as well as having the ‘mailbox import/export’ and ‘mailbox-search’ roles added to your account/group.

Nicke Källén (@znackattack)
Guest

Yeah, I noticed that. Back on track again. A bit surprised that the O365 Dev support was so strongly saying that this cmdlet wasn’t available and was caught off guard by that

Ben
Guest
Ben

This has been an awesome resource so far. Thank you! I have been able to follow all of the steps and have reached the last portion of the blog, but am running into some issues. New-MailboxImportRequest doesn’t seem to like the URI I have supplied. I have constructed the AzureBlobStorageAccountUri like such: PowerShell: ($AzureBlobStorageAccountUri + $row.FilePath + “/” + $row.Name) Resulting String: https://XXXX.blob.core.windows.net/ingestiondata/username/psts/Archive001.pst and all of the other relevant cmdlet parameters have also been supplied. When I actually kick off the cmdlet though I get 404 Not Found. If I connect to the ingestiondata storage container via PowerShell, I can… Read more »

Trevor
Guest
Trevor

Hi, I’m trying to use this procedure, but the SAS key provided by Microsoft doesn’t have read access to the Azure storage container. I can upload and enumerate files, but I can’t download files and I get the following error when I run New-MailboxImportRequest: “Unable to open PST file ‘https://xxxx.blob.core.windows.net/ingestiondata/xxxx.pst’. Error details: The remote server returned an error: (404) Not Found.” I’m able to use New-MailboxImportRequest with my own Azure storage account where I have full access, but not with the ingestiondata container and SAS key provided by Microsoft. Has Microsoft changed the permissions on the SAS key they give… Read more »

Markus
Guest
Markus

I noticed the same thing as Trevor.
Tested with explorer and I can’t even download the files. So the token appears to be write only…

Maybe csv is the only way for now…

Kevin
Guest
Kevin

Dear Dave,

I’m trying to find out how to delete the PST files now that I’ve completed the import process. Not sure if its just me but I feel the need to do a bit of post-import cleanup and I’d rather not leave PST files sitting there. Tried deleting via the Azure Storage Explorer GUI but it keeps saying “Resource not found. Do you have the required permissions?” If the SAS key is for write only then that explains the error message. But that still leaves me with trying to figure out how to delete those PST files.

Jeffrey Mueller
Guest
Jeffrey Mueller

The pst files will delete after 60 days. You can’t delete them manually.

Oliver Weber
Guest
Oliver Weber

If you are concerned about privacy, you can overwrite your existing psts in ingestiondata with empty files, though…

Nick
Guest
Nick

We are having the same issue. I have a ticket open for the past 3 weeks with MS.

Nick
Guest
Nick

MS has stated that this is a bug and there currently is no way to download or delete the files.

vic4real
Guest

Please I would like to know how to, if anyone succeeds in getting past the ..”Resource not found. Do you have the required permissions?” message
Thanks!

Marcel
Guest
Marcel

Hello,
I cannot find any details on Microsoft sites about the bug. I’m having the same errors.
For the time being I did overwrite the files with empty text files.
Marcel

Alex Carlock
Guest
Alex Carlock

Marcel, Thanks for the suggestion about overwriting with empty text files. I just did that and feel more comfortable. I’m getting the same error when trying to delete the files. This is a decent workaround.

Glen V
Guest
Glen V

You can try using Clumsy Leaf’s Cloud Xplorer. This has enabled me to delete the uploaded PSTs.
http://clumsyleaf.com/products/cloudxplorer

Nick
Guest
Nick

Whats not to like?
That you cant delete, download, or manage what ever you upload, confirmed bug by MS with not eta to be fixed.

Chabango
Guest
Chabango

Hi Dave,
Can you confirm that that using the new-mailboximportrequest is still valid with O365? I have tried every combination but am still getting the (404) Not found. I have verifies the case, as well as the path in Azure Explorer. Any ideas?

Thanks

James
Guest
James

Microsoft had a ticket open with me and have comment that they officialy do not support the new-mailboximport request command in office 365. This is what the told me specifically.
“This command is unequivocally not supported. If you are using the PST Import service to upload PSTs, you need to use the UI to manage the imports.”
It would have been easier if they just removed the command from the snap-in.

James

Follow Us!

Kloud Solutions Blog - Follow Us!