Multi-Threading Granfeldt PowerShell Management Agent Imports

As I’m sure you are familiar (with my many posts on the topic), the Granfeldt PowerShell Management Agent is extremely flexible. When used to integrate Microsoft Identity Manager with modern REST API’s it is easy to retrieve pages of results from a REST API and process the objects through the Management Agent. However sometimes you need to integrate Microsoft Identity Manager with an API (e.g. a SOAP WebService) that doesn’t provide functionality to page results.

Workday is an example of this (but the concepts and approach detailed in this blog post are valid to similar API integration scenarios). As I’ve posted about previously using the WorkdayAPI PowerShell Module in conjunction with the Granfeldt PowerShell Management Agent (PSMA) we can easily get the base records of Workday Users. However, to get the full Workday Record for each user using the WorkdayAPI PowerShell Module, requires a SOAP WebService call (facilitated by the PS Module) for each user. If you have a large Workday Tenant that can be a large number of calls and very time consuming (to be executed serially) for a Full Import into MIM. To achieve this as efficiently as possible, whilst still using the Granfeldt PSMA, a solution is to multi-thread the API calls to retrieve Workday Users’ full records.

Multi-Threading Granfeldt PowerShell Management Agent Imports

Here is the approach to implementing Multi-Threaded Granfeldt PowerShell Management Agent Imports using Workday as an example;

• Retrieve the base records from Workday
  • using the WorkdayAPI PowerShell Module
• Split the records into batches based off your PSMA RunProfile PageSize
  • I suggest 150 for use with Workday
• Process each batch through the PSMA as a Page
  • multi-thread / parallel process the requests for each of the full records to retrieve
  • repeat until all processed

Retrieve the base records from Workday

Retrieving the base records for all Workday User Records is the same as I detailed in this post here. Using the -Force switch will also return in-active records, so only do that if you really need to.

Split the records into batches

My method for splitting up the records into batches is a slight variation on the method I detailed in my PowerShell Snippets Volume 1 due to the way the collection is returned.

The excerpt below shows the Import Script taking the Run Profile PageSize and splitting the records into batches of that size. My recommendation (for Workday) is to configure a PageSize of 150 so that each page is returned and processed in ~3-6 minutes (dependant on your implementation of Workday and individual users record size).

Note: Make sure you configure your PSMA for Paging so that the PageSize is passed to the Import Script.

# Split the Workday Records into Batches based on PageSize from the Run Profile
# Suggest setting the Run Profile for 150 which will allow importing from Workday in digestable pages
$Global:groups = @()
$parts = [math]::Ceiling($WDayRecordsBase.Length / $pagesize)

for ($i = 0; $i -le $parts; $i++) { 
   $start = $i * $pagesize 
   $end = (($i + 1) * $pagesize) - 1 
   $Global:groups += , @($WDayRecordsBase[$start..$end]) 
}

Process each batch

Processing each batch is effectively passing each Batch to a PowerShell ScriptBlock that executes a specified number of the individual requests in parallel.

As I’ve done previously for processing parallel functions in PowerShell I’m using the Invoke-Parallel function from Warren Frame. It will take a little effort for each different API you are talking to, in order to determine the optimal number of requests to run in parallel. This is done with the -throttle parameter. For Workday the optimal is ~15. Going much higher will trigger the Workday API to reset your connection. And in my experience the connection reset could be triggered quickly or after a few hours of operation.

The following snippet shows defining a batch to process (batch of records to retrieve based off Run Profile PageSize) and multi-threading the calls (e.g. 15 maximum parallel requests) to retrieve the full records.

$global:wdBatch = $Global:groups[$global:groupsProcessed + 1] 
....
$wddetails = Invoke-Parallel -InputObject $global:wdBatch -throttle 15 -runspaceTimeout 60 -NoCloseOnTimeout -ImportVariables -ImportModules -ScriptBlock {...}

Process the results through the MA as you normally would then trigger the MA to reload for the next page (see full examp.

Example Multi-threaded Granfeldt PowerShell Management Agent Import Script

Here is an example Import Script implementing the multi-threaded approach detailed in this post (from Workday). You can find the previous example Workday PSMA I wrote here.

Summary

Multi-Threading Granfeldt PowerShell Management Agent Imports is achievable by leveraging PowerShell Runspaces (via the Invoke-Parallel function) and a little manipulation of a collection using PowerShell.

Another thing to keep in mind is that coming in PowerShell v7 is ForEach-Object -Parallel functionality. This will likely negate the need for the Invoke-Parallel function.