Deploy and Add SPFx webparts to Modern Pages using OfficeDevPnP CSOM library

In the previous blog here, we looked at how to install apps on a SharePoint site. With SharePoint and Office Dev PnP CSOM, we could also add web parts to Modern Pages, both out of the box (OOB) web parts and custom web parts. For out of box web parts, refer to Chris O’Brien article here , where he has provided steps and also the web part IDs for the OOB webparts which is really helpful.

In this blog, we will look at steps to add a custom web part and set it properties.

For the below code, we will use OfficeDevPnP CSOM library. The high level steps for implementation are below:

1. Find the page where to add the web part. For creating a modern page programmatically, refer to this link here.

2. Find if the web part is added to the page, if not then add it using web part ID. We will read the web part ID from a JSON class where we have kept the details of the custom web part

3. After the web part is added, then set the web part properties using JSON schema of the properties using NewtonSoft.Json.

Note: If the web part app is just installed, then it might take time for 
the web part to be ready for use. 
Hence put a wait state till the web part is loaded for use.

The below code has the steps for adding the web part programmatically using CSOM

Conclusion

The above process could be used to add web part programmatically onto a page and set it’s properties.

Happy Coding !!

Programmatically deploy and add SharePoint Framework Extensions using SharePoint CSOM and PowerShell

In the previous blog here, we looked at how to deploy and install SharePoint Apps. Now let’s look at installing SharePoint Framework extensions – Listview command sets programmatically.

SharePoint CSOM

SharePoint Framework has three type of extensions that could be created – Application customiser, Listview command sets and Field customisers. In this blog, we will look at adding list view command sets programmatically.

Listview command extensions are actually custom actions installed in a library or list. Hence to activate it we will go to the library/list, find the installed custom actions, if not installed we will install the new custom action. Below is the code for that.

PowerShell

We could also use PnP PowerShell to add the Library extension onto a page using the code below

Hence, above we saw how we could add extensions onto a library or list using CSOM or PowerShell

Happy Coding!!

Deploy and Install SharePoint Apps using SharePoint CSOM and PnP PowerShell

In this blog, we will look at steps to install and deploy SharePoint apps to Modern Sites using SharePoint ALM CSOM and PnP PowerShell. Using the below steps, it is possible to programmatically deploy and install custom SharePoint Framework apps using an Azure Function or a Local PowerShell script.

Installing SharePoint Apps

SharePoint Apps can be deployed on a site using ALM (Application Lifecycle Management) APIs. After the app is installed in the App catalog, we could add it to a SharePoint site.

SharePoint CSOM

The steps are simple. The below snippet has the code for deploying and installing apps.

1. Get the App Id
2. Create an App Manager Object
3. Deploy the App
4. After deploy the app, install the app.
5. Use Try – catch to handle if the installation has already done

PnP PowerShell

First, lets’ get a list of apps in the App catalog.

Note: There are two values that is supported by scope paramaters for Apps – Tenant and Site.

Get-PnPApp -Scope Tenant
or
Get-PnPApp -Scope Site

If the App is not installed, then we will add the app to the App catalog

Add-PnPApp -Path "<.sppkg file location>" -Scope Site

Then, publish the App

Publish-PnPApp -Identity  -Scope Tenant -SkipFeatureDeployment

-SkipFeatureDeployment is helpful to deploy Apps across the SharePoint tenancy if the app is developed for tenant wide deployments

After the above, we will install the App

Install-PnPApp -Identity  -Scope Site

After the app is installed, it is ready to be added or used at the site.

In the upcoming blog, we will see how to add SharePoint Framework extensions and web parts programmatically.

Happy Coding!!

SharePoint Integration for Health Care eLearning – Moving LMS to the Cloud

Health care systems often face challenges in the way of being unkept and unmaintained or managed by too many without consistency in content and harbouring outdated resources. A lot of these legacy training and development systems also wear the pain of constant record churning without a supportable record management system. With the accrual of these records over time forming a ‘Big Data concern’, modernising these eLearning platforms may be the right call to action for medical professionals and researchers. Gone should be the days of manually updating Web Vista on regular basis.

Cloud solutions for Health Care and Research should be well on its way, but the better utilisation of these new technologies will play a key factor in how confidence is invested by health professionals in IT providing a means for departmental education and career development moving forward.

Why SharePoint Makes Sense (versus further developing Legacy Systems)

Every day, each document, slide image and scan matters when the paying customer’s dollar is placed on your proficiency to solve pressing health care challenges. Compliance and availability of resources aren’t enough – streamlined and collaborative processes, from quality control to customer relationship management, module testing and internal review are all minimum requirements for building a modern, online eLearning centre i.e. a ‘Learning Management System’.

ELearningIndustry.com has broken down ten key components that a Learning Management System (LMS) requires in order to be effective. From previous cases, working in developing an LMS, or OLC (Online Learning Centre) Site using SharePoint, these ten components can indeed be designed within the online platform:

  1. Strong Analytics and Report Generation – for the purposes of eLearning, e.g. dashboards which contain progress reports, exam scores and other learner data, SharePoint workflows allows for progress tracking of training and user’s engagement with content and course materials while versioning ensures that learning managers, content builders (subject matter experts) and the learners themselves are on the same page (literally).
  1. Course Authoring Capability – SharePoint access and user permissions are directly linked to your Active Directory. Access to content can be managed, both from a hierarchical standpoint or role-based if we’re talking content authors. Furthermore, learners can have access to specific ‘modules’ allocated to them based on department, vocation, etc.
  1. Scalable Content Hosting – flexibility of content or workflows, or plug-ins (using ‘app parts’) to adapt functionality to welcome new learners where learning requirements may shift to align with organisational needs.
  1. Certifications – due to the availability and popularity of SharePoint online in large/global Enterprises, certifications for anywhere from smart to super users is available from Microsoft affiliated authorities or verified third-parties.
  1. Integrations (with other SaaS software, communication tools, etc.) – allow for exchange of information through API’s for content feeds and record management e.g. with virtual classrooms, HR systems, Google Analytics.
  1. Community and Collaboration – added benefit of integrated and packaged Microsoft apps, to create channels for live group study, or learner feedback, for instance (Skype for Business, Yammer, Microsoft Teams).
  1. White Labelling vs. Branding – UI friendly, fully customisable appearance. Modern layout is design flexible to allow for the institutes branding to be proliferated throughout the tenant’s SharePoint sites.
  1. Mobile Capability – SharePoint has both a mobile app and can be designed to be responsive to multiple mobile device types
  1. Customer Support and Success – as it is a common enterprise tool, support by local IT should be feasible with any critical product support inquiries routed to Microsoft
  1. Support of the Institutes Mission and Culture – in Health Care Services, where the churn of information and data pushes for an innovative, rapid response, SharePoint can be designed to meet these needs where, as an LMS, it can adapt to continuously represent the expertise and knowledge of Health Professionals.

Outside of the above, the major advantage for health services to make the transition to the cloud is the improved information security experience. There are still plenty of cases today where patients are at risk of medical and financial identity fraud due to inadequate information security and manual (very implicitly hands-on) records management processes. Single platform databasing, as well as the from-anywhere accessibility of SharePoint as a Cloud platform meets the challenge of maintaining networks, PCs, servers and databases, which can be fairly extensive due to many health care institutions existing beyond hospitals, branching off into neighbourhood clinics, home health providers and off-site services.

Provisioning complex Modern Sites with Azure Functions and Flow – Part 2 – Create and Apply Template

In the previous blog here, we got an overview of the high level Architecture of a Complex Modern team site provisioning process. In this blog, we will look at the step 1 of the process – Create and Apply template process, in detail.

Before that, below are few links to earlier blogs, as a refresher, to prerequisties for the blog.

  1. Set up a Graph App to call Graph Service using App ID and Secret – link
  2. Sequencing HTTP Trigger Azure Functions for simultaneous calls – link
  3. Adding and Updating owners using Microsoft Graph Async calls – link

Overview

The Create and Apply Template process aims at the following

1. Create a blank modern team site using Groups Template (Group#0 Site template)

2. Apply the provisioning template on the created site.

Step 1 : Create a blank Modern team site

For creating a modern team site using CSOM we will use the TeamSiteCollectionCreationInformation class of OfficeDevPnP.  Before we create the site, we will make sure the site doesn’t already exist.

Note: There is an issue with the Site Assets library not getting intialized 
when the site is created using the below code. 
Hence, calling the EnsureSiteAssets library is necessary.

Step 2:  Apply the Provisioning Template

Note: The Apply template process is a long running process and takes from 60-90 min to complete 
for a complex provisioning template with many site columns, content types and libraries. 
In order to prevent the Azure function from timing out, it is required to host the Azure Function 
using a App Service Plan instead of a Consumption plan so the Azure function 
is not affected by the 10 min time out. 

For the Apply Provisioning Template process, use the below steps.

1. Reading the Template

It is important to note that the XMLPnPSchemaFormatter version (in the code below) must match the PnP version used to generate the PnP template. If the version is older, then set the XMLPnPSchemaFormatter to read from the older version. In order to find the version of the PnP Template, open the xml and look at the start of the file

PnPTemplateVersion

2. Apply the Template

For applying the template, we will use the ProvisioningTemplateApplyingInformation class of the OfficeDevPnP module. The ProvisioningTemplateApplyingInformation also has a property called HandlerToProcess which could be used the invoke the particular handler in the provisioning template process. Below is the code for the same.

After the apply template process is complete, since the flow will have timed out, we will invoke another flow to do the post process by updating a list item in the SharePoint list.

Conclusion

In this blog, we saw how we could create a modern team site and apply the template on it. The next blog we will finalize the process by doing site specfic changes after applying the template.

Provisioning complex Modern Sites with Azure Functions and Microsoft Flow – Part 1 – Architecture

In one of my previous blog here,  I have discussed about creating Office 365 groups using Azure Function and Flow. The same process could be used also to provision Modern Team sites in SharePoint Online because Modern Team Sites are Office 365 groups too. However, if you are creating a Complex Modern Team Site with lots of Libraries, Content types, Termstore associated columns etc. it will challenging to do it with a single Azure Function.

Thus, in this blog (part 1), we will look at the Architecture of a Solution to provision a complex Modern Team Site using multiple Azure Function and Flows. This is an approach that went through four months of validation and testing. There might be other options but this one worked for the complex team site which takes around 45-90 mins to provision.

Solution Design

To start with lets’ look at the solution design. The solution consists of two major components

1. Template Creation – Create a SharePoint Modern Team site to be used as a template and generate a Provisioning template from it

2. Provisioning Process – Create a SharePoint Inventory List to run the Flow and Azure Function. There will be three Azure Functions that will run three separate parts of the provisioning lifecycle. More details about the Azure Functions will in upcoming blog.

Get the Provisioning Template

The first step in the process is to  create a clean site that will be used as a reference template site for the Provisioning template. In this site, create all the lists, libraries, site columns, content type and set other necessary site settings.

In order to make sure that the generated template doesn’t have any elements which are not needed for provisioning, use the following PnP PowerShell cmdlet. The below cmdlet removes any content type hub association, ALM api handles and site security for provisioning requirements.

Get-PnPProvisioningTemplate -Out "" -ExcludeHandlers ApplicationLifecycleManagement, SiteSecurity -ExcludeContentTypesFromSyndication

The output of the above cmdlet is ProvisioningTemplate.xml file which could be applied to new sites for setting up the same SharePoint elements. To know more about the provisioning template file, schema and allowed tags, check the link here.

ModernSitesProvisioningFlow_GetTemplate

Team Site Provsioning Process

The second step in the process would be to create and apply the template to a Modern SharePoint Team site using Flow and Azure Function. The detail steps would be as follows:

1. Create an Inventory list to capture all the requirements for Site Creation

2. Create two flows

a) Create and Apply Template flow, and

b) Post Provisioning Flow

3. Create three Azure Functions –

a) Create a blank Modern Team Site

b) Apply Provisioning Template on the above site. This is a long running process and can take about 45-90 min for applying a complex template with about 20 libraries, 20-30 site columns and 10-15 content types

Note: Azure Functions on Consumption plan have a timeout of 10 min. Host the Azure function on an App Service Plan for the above to work without issues

c) Post Provisioning to apply changes that are not supported by Provisioning Template such as Creating default folders etc.

Below is the process flow for the provisioning process. It has steps from 1 – 11 which goes from creating the site to applying it. The brief list of the steps are as follows

  1. Call the Create Site flow to start the Provisioning Process
  2. Call the Create Site Azure Function
  3. Create the Modern Team Site in Azure Function and set any dependencies required for the Apply template such as Navigation items, pages etc, and then return to flow
  4. Call the Apply Template Azure Function.
  5. Get the previously generated ProvisioningTemplate.xml file from a shared location
  6. Apply the Template onto the newly created Modern site. Note: The flow call times out because it cannot wait for such a long running process
  7. Update the status column in the Site Directory for the post provisioning flow to start
  8. Call the Post provisioning flow to run the Post provisioning azure function
  9. The Post provisioning azure function will complete the remaining SharePoint changes which were not completed by the apply template such as, set field default values, create folders in any libraries, associate default values to taxonomy fields etc.

ModernSitesProvisioningFlow_ProvisioningProcess

Conclusion:

Hence in the above blog, we saw how to create a provisioning process to handle complex modern team site creation at a high architectural level. Next, we will deep dive into the Azure functions to create, apply template and post process in the next upcoming blogs.

Happy Coding!!!

Planning Site structure and Navigation in SharePoint Modern Experience Communication and Team sites

If you are planning to implement or implementing Modern team sites or Communication sites, there is change in best practices for planning and managing the Sites structure, Site Hierarchy and Navigation. This is a very common question during my presentations – how do we manage site structures, navigation and content in Modern experiences.

So, in this blog, we will look at few strategies for planning Site structure and Navigation in Modern Experience sites.

1. First and foremost, get rid of nested subsites and Site hierarchy navigation. Recently Microsoft has been pushing for Site Collections flat structure with Modern Team and Communication sites, which adds a lot of benefit for managing isolation and content. So, the new approach – Flat Site Collections and no Subsites. (There are various advantages of flat structure site collections which will be listed in another upcoming blog)

2. Secondly, to achieve a hierarchy relationship among sites such as Navigation, news items, search etc, use Hub Sites. Hub sites are the new way of connecting SharePoint site collections together. Besides, they have added advantage of aggregating information such as News and Search results from related hub associated sites. So, create a Hub site for Navigation across related sites.HubSiteAssociatedTeam

3. A best candidate for Hub sites, in my opinion, is Communication sites. Communication sites have a top navigation that can be easily extended for Hub sites. They are perfect for publishing information and showing aggregrated content. However, it also depends on if the Hub site is meant for a team and business unit or company as a whole. So, use Communication as a Hub site if targeting all company or a major group.QuickLaunchNestedCommunicationSite

4. One Navigation structure – Quick launch (Left hand) is Top Navigation for Communication sites. So no need to maintain two navigations. If you ask me, this a big win and removes a lot of confusion for end users.QuickLaunchEdit_CommSite

5. Quick launch for Modern Team and Communication Sites allows three level sub hierarchy which allows to define a nested custom hierarchy structure for Navigation which could be different from the content structure and site structure.

6. Focus on Content, not on Navigation or location of Content, through new Modern web parts such as Highlighted content, Quick links etc. which allow you to find content anywhere easily.HighlightedContent

7. Finally, few limitations of Modern Site Structure and Navigation (as of June 2018) for reference. Hopefully, this will be changed soon.

    • Permissions management still needs to be managed at each Site Collection, no nested structure there yet. Yet it is possible to use AD groups for consistent permissions set up
    • Office 365 Unified Security Groups cannot have AD or other Office 365 groups nested for Modern Team sites. But SharePoint site permissions could be managed through AD groups
    • Contextual Navigation bolding is missing in Hub sites i.e. if you click on the link to move to a child site then navigation is not automatically bolded, but this might be coming soon.
    • Navigation headers in Modern sites cannot be blank and needs to be set to a link

Conclusion:

Hence in this blog, we looked at an approach for Modern site structures, hierarchy and navigation.

Automate network share migrations to Sharepoint Online using ShareGate PowerShell

Sharegate supports PowerShell scripting which can be used to automate and schedule migrations. In this post, I am going to demonstrate an example of end to end automation to migrate network Shares to SharePoint Online. The process effectively reduces the task of executing migrations to “just flicking a switch”.

Pre-Migration

The following pre-migration activities were conducted before the actual migration:

  1. Analysis of Network Shares
  2. Discussions with stakeholders from different business units to identify content needs
  3. Pilot migrations to identify average throughput capability of migration environment
  4. Identification of acceptable data filtration criteria, and prepare Sharegate migration template files based on business requirements
  5. Derive a migration plan from above steps

Migration Automation flow

The diagram represents a high-level flow of the process:

 

The migration automation was implemented to execute the following steps:

  1. Migration team indicates that migration(s) are ready to be initiated by updating the list item(s) in the SharePoint list
  2. Updated item(s) are detected by a PowerShell script polling the SharePoint list
  3. The list item data is downloaded as a CSV file. It is one CSV file per list item. The list item status is updated to “started”, so that it would not be read again
  4. The CSV file(s) are picked up by another migration PowerShell script to initiate migration using Sharegate
  5. The required migration template is selected based on the options specified in the migration list item / csv to create a migration package
  6. The prepared migration task is queued for migration with Sharegate, and migration is executed
  7. Information mails are “queued” to be dispatched to migration team
  8. Emails are sent out to the recipients
  9. The migration reports are extracted out as CSV and stored at a network location.

Environment Setup

Software Components

The following software components were utilized for the implementing the automation:

  1. SharePoint Online Management shell
  2. SharePoint PnP PowerShell
  3. Sharegate migration Tool

Environment considerations

Master and Migration terminals hosted as Virtual machines – Each terminal is a windows 10 virtual machine. The use of virtual machines provides following advantages over using desktops:

  • VMs are generally deployed directly in datacenters, hence, near the data source.
  • Are available all the time and are not affected by power outages or manual shutdowns
  • Can be easily scaled up or down based on the project requirements
  • Benefit from having better internet connectivity and separate internet routing can be drawn up based on requirements

Single Master Terminal – A single master terminal is useful to centrally manage all other migration terminals. Using a single master terminal offers following advantages:

  • Single point of entry to migration process
  • Acts as central store for scripts, templates, aggregated reports
  • Acts as a single agent to execute non-sequential tasks such as sending out communication mails

Multiple Migration terminals – It would be optimal to initiate parallel migrations and use multiple machines (terminals) to expedite overall throughput by parallel runs in an available migration window (generally non-business hours).  Sharegate has option to use either 1 or 5 licenses at once during migration. We utilized 5 ShareGate licenses on 5 separate migration terminals.

PowerShell Remoting – Using PowerShell remoting allows opening remote PowerShell sessions to other windows machines. This will allow the migration team to control and manage migrations using just one terminal (Master Terminal) and simplify monitoring of simultaneous migration tasks. More information about PowerShell remoting can be found here.

PowerShell execution policy – The scripts running on migration terminals will be stored at a network location in Master Terminal. This will allow changing / updating scripts on the fly without copying the script over to other migration terminals. The script execution policy of the PowerShell window will need to be set as “Bypass” to allow execution of scripts stored in network location (for quick reference, the command is “Set-ExecutionPolicy -ExecutionPolicy Bypass”.

Windows Scheduled Tasks – The PowerShell scripts are scheduled as tasks through Windows Task schedulers and these tasks could be managed remotely using scripts running on the migration terminals. The scripts are stored at a network location in master terminal.

Basic task in a windows scheduler

PowerShell script file configured to run as a Task

Hardware specifications

Master terminal (Manage migrations)

  • 2 cores, 4 GB RAM, 100 GB HDD
  • Used for managing scripts execution tasks on other terminals (start, stop, disable, enable)
  • Used for centrally storing all scripts and ShareGate property mapping and migration templates
  • Used for Initiating mails (configured as basic tasks in task scheduler)
  • Used for detecting and downloading migration configuration of tasks ready to be initiated (configured as basic tasks in task scheduler)
  • Windows 10 virtual machine installed with the required software.
  • Script execution policy set as “Bypass”

Migration terminals (Execute migrations)

  • 8 cores, 16 GB RAM, 100 GB HDD
  • Used for processing migration tasks (configured as basic tasks in windows task scheduler)
  • Multiple migration terminals may be set up based on the available Sharegate licenses
  • Windows 10 Virtual machines each installed with the required software.
  • Activated Sharegate license on each of the migration terminals
  • PowerShell remoting needs to be enabled
  • Script execution policy set as “Bypass”

Migration Process

Initiate queueing of Migrations

Before migration, migration team must perform manual pre – migration tasks (if any as defined by the migration process as defined and agreed with stake holders). Some of the pre-migration tasks / checks may be:

  • Inform other teams about a possible network surge
  • Confirming if another activity is consuming bandwith (scheduled updates)
  • Inform the impacted business users about the migration – this would be generally set up as the communication plan
  • Freezing the source data store as “read-only”

A list was created on a SharePoint online site to enable users to indicate that the migration is ready to be processed. The updates in this list shall trigger the actual migration downstream. The migration plan is pre-populated in this list as a part of migration planning phase. The migration team can then update one of the fields (ReadyToMigrate in this case) to initiate the migration. Migration status is also updated back to this list by the automation process or skip a planned migration (if so desired).

The list provides as a single point of entry to initiate and monitor migrations. In other words, we are abstracting out migration processing with this list and can be an effective tool for migration and communication teams.

The list was created with the following columns:

  • Source => Network Share root path
  • Destination site => https://yourteanant.sharepoint.com/sites/<Sitename>
  • Destination Library => Destination library on the site
  • Ready to migrate => Indicates that the migration is ready to be triggered
  • Migrate all data => Indicate if all data from the source is to be migrated (default is No). Only filtered data based on the predefined options will be migrated. (more on filtered options can be found here)
  • Started => updated by automation when the migration package has been downloaded
  • Migrated => updated by automation after migration completion
  • Terminal Name => updated by automation specifying the terminal being used to migrate the task

 

Migration configuration list

 

After the migration team is ready to initiate the migration, the field “ReadyToMigrate” for the migration item in the SharePoint list is updated to “Yes”.


“Flicking the switch”

 

Script to create the migration configuration list

The script below creres the source list in sharepoint online.

Script to store credentials

The file stores the credentials and can be used subsequent scripts.

Queuing the migration tasks

A PowerShell script is executed to poll the migration configuration list in SharePoint at regular intervals to determine if a migration task is ready to be initiated. The available migration configurations are then downloaded as CSV files, one item / file and stored in a migration packages folder on the master terminal. Each CSV file maps to one migration task to be executed by a migration terminal and ensures that the same migration task is not executed by more than one terminal. It is important that this script runs on a single terminal to ensure only one migration is executed for one source item.

 

 

Execute Migration

The downloaded migration configuration CSV files are detected by migration script tasks executing on each of the migration terminals. Based on the specified source, destination and migration options the following tasks are executed:

  1. Reads the item from configuration list to retrieve updated data based on item ID
  2. Verify Source. Additionally, sends a failure mail if source is invalid or not available
  3. Revalidates if a migration is already initiated by another terminal
  4. Updates the “TerminalName” field in the SharePoint list to indicate an initiated migration
  5. Checks if the destination site is created. Creates if not already available
  6. Checks if the destination library is created. Creates if not already available
  7. Triggers an information mail informing migration start
  8. Loads the required configurations based on the required migration outcome. The migration configurations specify migration options such as cut over dates, source data filters, security and metadata. More about this can be found here.
  9. Initiates the migration task
  10. Extracts the migration report and stores as CSV
  11. Extracts the secondary migration report as CSV to derive paths of all files successfully migrated. These CSV can be read by an optional downstream process.
  12. Triggers an information mail informing migration is completed
  13. Checks for another queued migration to repeat the procedure.

The automatioin script is given below –

 


Additional Scripts

Send Mails

The script triggers emails to required recipients.

This script polls a folder:   ‘\masterterminal\c$\AutomatedMigrationData\mails\input’ to check any files to be send out as emails. The csv files sepcify subject and body to be send out as emails to recipients configured in the script. Processed CSV files are moved to final folder.

 

Manage migration tasks (scheduled tasks)

The PowerShell script utilizes PowerShell remoting to manage windows Task scheduler tasks configured on other terminals.

 


Conclusion

The migration automation process as described above helps in automating the migration project and reduces manual overhead during the migration. Since the scripts utilize pre-configured migration options / templates, the outcome is consistent with the plan. Controlling and monitoring migration tasks utilizing a SharePoint list introduces transparency in the system and abstracts the migration complexity. Business stakeholders can review migration status easily from the SharePoint list and this ensures an effective communication channel. Automated mails informing about migration status provide additional information about the migration. The migration tasks are executed in parallel across multiple migration machines which aids in a better utilization of available migration window.

 

 

Utilizing Sharegate migration Templates for Network share migrations

Sharegate supports PowerShell based scripting which can be used to automate and schedule migrations. The purpose of this post is to demonstrate the use of pre-created migration templates to initiate migration tasks in Sharegate using PowerShell scripts. In one of my previous project, we were migrating network shares to SharePoint Online using Sharegate as the migration tool of choice.

Based on our discussions with business divisions and IT department, the following requirements were identified for most of the divisions:

  1. Office documents, PDFs, Image files will be migrated
  2. Include only documents modified after a date for e.g. January 1, 2016
  3. Permissions should be preserved
  4. There were some exceptions where the identified divisions requested all available data to be migrated.

To address the above business requirements, we created custom migration templates and these templates were used to trigger the migration tasks. This allowed us to do use pre-created configurations for each migration and eliminate of manual overhead and risk of errors. Sharegate migration templates (.sgt files) are xml based configuration documents and encapsulate the configuration options such as:

  • Copy options
  • Content type mappings
  • Allowed extensions
  • Cut-off dates

Migration Templates

I’ll present two (2) migration templates, that were prepared to cover off the business requirements:

Template 1 – All data This migration template configures Sharegate to migrate all data to the destination library. The configuration values to be considered are:

  • CopyPermissions Specifies that all permissions on the files / folders be over [configured as “true”]
  • KeepAuthorsAndTimestamps – Specifies that the file / folder metadata such as Authors, associated time stamps (created date, modified date) be copied over [configured as “true”]
  • ContentFileExtensionFilterType – specifies file extension type filter to copy data [configured as “AllExtensions”]

 

Template 2 – Filtered Data This migration template configures Sharegate to migrate all the documents that was modified / created after 1st January, 2016 and matches one of the configured extensions to the destination library. The configuration values to be considered are:

  • CopyPermissions – Specifies that all permissions on the files / folders be over [configured as “true”]
  • ContentFrom – specifies the date cutover anything modified after this date shall be copied [configured as “01/01/2016 10:00:00”]
  • KeepAuthorsAndTimestamps – Specifies that the file / folder metadata such as Authors, associated time stamps (created date, modified date) be copied over [configured as “true”]
  • ContentFileExtension – specifies that all allowed file type to be copied over [configured as “pdf; xlsx; xls; doc; pptx; docx; ppt; xlsm; rtf; vsd; pub; docm; xlsb; mpp; one; pps; dotx; dot; pot; xlk; vdx; pptm; xlt; xlam; potx; odt; xla; wk4; dic; dotm; xltm; xltx; onetoc2; vss; thmx; vsdx; xps; msg; eml; jpg; cr2; png; tif; psd; bmp; eps; ai; jpeg; gif; indd; dwg; svg; djvu; ico; wmf; dcx; emf; xpm; pdn; cam; ”]
  • ContentFileExtensionFilterType – specifies file extension type filter to copy data [configured as “LimitTo”]

 

Migration templates in Sharegate user interface

The migration templates presented as XML files before may be imported into Sharegate user interface to inspect and test. Migration options are available while configuring a migration as demonstrated in the screen shot provided below:

 

The screen shots below represent the migration options as visible in Sharegate user interface, when the templates are imported into the tool.

Copy Options Screen – Template 1

Content type mapping screen – Template 1

Filter options screen – Template 1

 

Copy Options Screen – Template 2

Content type mapping screen – Template 2

Filter options screen – Template 2

 

PowerShell scripts to initiate migrations

Migrations can be initiated from PowerShell console by running the following script

 

The coded migration templates as demonstrated above has merits in terms of eliminating the need to manually configure options for each migration using Sharegate user interface.

To extend this to a whole new level, the overall migration project can further be split into multiple migration tasks and scheduled as sequential migrations across multiple machines utilizing common migration templates and scripts stored in a network location, but that is something I’ll cover off as a part of separate blog post.

Happy Migrating…

Automation and Creation of Office 365 groups using Flow, Microsoft Graph and Azure Function – Part 2

In the Part 1 blog here, we discussed an approach for the Group creation process and important considerations for provisioning groups. In this blog, we will look at getting a Graph App ID and App secret for invoking the graph service and then implementation of the group provisioning process.

MS Graph App Set up

Before we start creating groups we will need to set up a Graph App that will be used to create the group in the Office 365 tenancy. The details are in this blog here on how to create a Microsoft Graph app.

Regarding the permissions, below are the settings that are necessary to allow creating groups through the graph service.

GroupApp_Rights

Creating a Group

As discussed in Part 1 here, below are the broad level steps for automating group creation using a SharePoint inventory list, Microsoft Flow and Azure Function

1. Create a SharePoint list, with the metadata necessary for Group and SharePoint assets provisioning

We can use a SharePoint list to act as a trigger to create groups with the custom metadata necessary for provisioning the groups such as Owners and metadata necessary for creating site assets for SharePoint sites. As a best practice, I recommend you create multiple master lists to manage the details separately if there are too many to manage. In our case, we have created three separate lists for managing the Group details.

1. Group details and metadata
2. Owners and Team Members List
3. Site Assets configuration list

2. Create a Microsoft flow. The flow will validate a new or existing group and pick the unique Group Alias from the list which will allow us to find the group if it exists.

The flow will act as a trigger to start the provisioning process and call the Azure function passing the appropriate metadata as shown below. The flow also allows error handling scenarios as described in the Part 1 blog here

Note: The GroupAlias is the unique name of the Group and is not necessarily the SharePoint URL. For example, in the case where a group was created and subsequently deleted, the unique alias could be used again but the Site URL will be different (unless cleared from the SharePoint recycle bin).

Group_FlowAzureFunctionCall

3. Create the Group in an Azure Function using SharePoint Online CSOM

In order to create the group, we will need to authenticate to the Graph service using the Graph App created earlier. For authenticating the app through Azure AD, please install the NuGet Package for Microsoft.IdentityModel.Clients.ActiveDirectory.

After authenticating, we will create the group using the UnifiedGroup Utility provided through the SharePoint Online CSOM.

Below is a quick snapshot of the code. Note the inclusion of Graph module of the OfficeDevPnP class.

Note: One important bit to note is that, in the above code owners and members email array is the same. If the owners and members email array differ, then the group provisioning delays significantly. Also, it is important to keep the other parameters same as during creation in the below method because it might reset the other properties to default otherwise. For eg. if isPrivate is not set, then the group becomes public.

4. After the group is created, we can fetch the details of the group as below.

5. The group provisioning generally takes about 2-3 mins to provision. However, if there are multiple hits, then one request might override the second request causing the group creation to fail. In such cases, we can sequence the Azure Functions to run by modifying the host.json file. A quick blog covering this can be found here.

Provisioning SharePoint Assets in Azure Function after Group Creation

1. For provisioning of the SharePoint assets, we might have to wait for the Office 365 AD sync to finish granting access to the Admin account.

Sometimes, the AD sync process takes much longer, so to grant direct access to the SharePoint Site Collection using tenant admin, we could use the below code. Recommendation: Only proceed with the below code approach if the access fails for more than few mins.

Note: As a best practice, I would recommend using a Service Account when working on the SharePoint Site. We could also use an App as suggested in the Site Scripting blog here.

2. Once you have access, you can use the normal SharePoint CSOM to do the activities that are pertaining to SharePoint asset provisioning such as Libraries, Site Pages content, Lists, etc.

3. After you’re done, you can return the success from the Azure function as below.

Note: Use HttpStatusCode.Accepted instead of HttpStatusCode.Error in case there is error handling in the Flow or else Flow will trigger another instance of the flow when the Azure Function fails

Conclusion:

Above we saw how we can have a SharePoint Inventory list and create groups using Flow and Azure Functions. For a quick reference, below are the links to the other related blogs.

Part 1 – Automation and Creation of Office 365 groups approach

How to create a Microsoft Graph App

Sequencing calls in Azure Functions