Introduction
In the last 12 months I’ve lost count of the number of PowerShell Management Agents I’ve written to integrate Microsoft Identity Manager with a plethora of environments. The majority though have not been of huge scale (<50k objects) and the import of the managed entities into the Connector Space/Metaverse runs through pretty timely.
However this week I’ve been working on a AzureAD Groups PS MA for an environment with 40k+ groups. That in itself isn’t that large, but when you start processing Group Memberships as well, the Import process can take an hour for a Full Sync. During this time before the results are passed to the Sync Engine you don’t have any visual of where the Import is up to (other than debug logging). And the ability to stop the MA requires a restart of the Sync Engine Server.
I’ve wanted to mess with Paging the Imports for sometime, but it hadn’t been a necessity. Now it is, so I looked to working out how to achieve it. The background information on Paged Imports is available at the bottom of the PSMA documentation page here. However there are no working examples. I contacted Soren and he had misplaced his demo scripts for the time being. With some time at hand (in between coats of paint on the long weekend renovation) I therefore worked it out for myself. I detail how to implement Paged Imports in this blogpost.
This post uses an almost identical Management Agent to the one described in this post. Review that post to get an understanding of the AzureAD Differential Queries. I’m not going to cover those elements in this post or setting up the MA at all.
Getting Started
There are two things you need to do in preparation for enabling Paged Imports on your PowerShell Management Agent;
- Enable Paged Imports (if your Import.ps1 is checking for this setting)
- Configure Page Size on your Import Run Profiles
The first is as simple as clicking the checkbox on the Global Parameters tab on your PS MA as shown below.
https://dl.dropboxusercontent.com/u/76015/BlogImages/PSMA%20PagedImport/PageImports.PNG
The 2nd is in your Run Profile. By default this will be 100. For my “let’s figure this out” process I dropped my Run Profiles to 50 on one Run Profile and 10 on another.
https://dl.dropboxusercontent.com/u/76015/BlogImages/PSMA%20PagedImport/PageSize50.PNG
Import Script
With Paged Imports setup on the MA the rest of the logic goes into your Import Script. In your param section at the start of the script $usepagedimport and $pagesize are the variables that reflect the items from the two enablement components you did above.
$usepageimport is either True or False. Your Import.ps1 script can check to see if it is set and process accordingly. In this example I’m not even checking if it is set and doing Paged Imports anyway. For completeness in a production example you should check to see what the intention of the MA is.
$pagesize is the pagesize from the Run Profile (100 by default, or whatever you changed your’s too).
param (
$Username,
$Password,
$Credentials,
$OperationType,
[bool] $usepagedimport,
$pagesize
)
An important consideration to keep in mind is that the Import.ps1 will be called multiple times (ie. #of_times = #ofObjects / pagesize).
So anything that you would normally expect in any other MA to only process once when the Import.ps1 runs you need to limit to only running once.
Essentially the way I’ve approached it is, retrieve all the objects that will be processed and put them in a Global variable. If the variable does not have any values/data then it is the first run, so go and get our source data. If the Global variable has values/data in it then we must be on a subsequent loop so no need to go process that part, just page through our import.
In my example below this appears as;
if (!$global:tenantObjects) {
# Authenticate
# Search and get the users
# Do some rationalisation on the results (if required)
# setup some global variables so we know where we are with processing the data
} # Finish the one time tasks
As you’ll see in the full import.ps1 script below there are more lines that could be added into this section so they don’t get processed each time. In a production implementation I would.
For the rest of the Import.ps1 script we are expecting it to run multiple times. This is where we do our logic and process our objects to send through to the Sync Engine/Connector Space. We need to keep track of where we are up to processing the dataset and continue on from where we left off. We also need to know how many objects we have processed in relation to the ‘pagesize’ we get from the Run Profile so we know when we’ve finished.
When we reach the pagesize but know we have more objects to process we set the $global:MoreToImport to $true and break out of the foreach loop.
When we have processed all our objects we set $global:MoreToImport = $false and break out of the foreach loop to finish.
With that explanation out of the way here is a working example. I’ve left in debugging output to a log file so you can see what is going on.
You can get the associated relevant Schema.ps1 from the Management Agent described in this post. You’ll need to update your Tenant name on line 29, your directory paths on lines 10 and 47. If you are using a different version of the AzureADPreview PowerShell Module you’ll need to change line 26 as well.
Everything else is in the comments within the example script below and should make sense.
Summary
For managing a large number of objects on a PS MA we can now see progress as the import processes the objects, and we can now stop an MA if required.
I’m sure this will help someone else. Enjoy.
Follow Darren on Twitter @darrenjrobinson