Automation and Creation of Office 365 groups using Flow, Microsoft Graph and Azure Function – Part 2

In the Part 1 blog here, we discussed an approach for the Group creation process and important considerations for provisioning groups. In this blog, we will look at getting a Graph App ID and App secret for invoking the graph service and then implementation of the group provisioning process.

MS Graph App Set up

Before we start creating groups we will need to set up a Graph App that will be used to create the group in the Office 365 tenancy. The details are in this blog here on how to create a Microsoft Graph app.

Regarding the permissions, below are the settings that are necessary to allow creating groups through the graph service.

GroupApp_Rights

Creating a Group

As discussed in Part 1 here, below are the broad level steps for automating group creation using a SharePoint inventory list, Microsoft Flow and Azure Function

1. Create a SharePoint list, with the metadata necessary for Group and SharePoint assets provisioning

We can use a SharePoint list to act as a trigger to create groups with the custom metadata necessary for provisioning the groups such as Owners and metadata necessary for creating site assets for SharePoint sites. As a best practice, I recommend you create multiple master lists to manage the details separately if there are too many to manage. In our case, we have created three separate lists for managing the Group details.

1. Group details and metadata
2. Owners and Team Members List
3. Site Assets configuration list

2. Create a Microsoft flow. The flow will validate a new or existing group and pick the unique Group Alias from the list which will allow us to find the group if it exists.

The flow will act as a trigger to start the provisioning process and call the Azure function passing the appropriate metadata as shown below. The flow also allows error handling scenarios as described in the Part 1 blog here

Note: The GroupAlias is the unique name of the Group and is not necessarily the SharePoint URL. For example, in the case where a group was created and subsequently deleted, the unique alias could be used again but the Site URL will be different (unless cleared from the SharePoint recycle bin).

Group_FlowAzureFunctionCall

3. Create the Group in an Azure Function using SharePoint Online CSOM

In order to create the group, we will need to authenticate to the Graph service using the Graph App created earlier. For authenticating the app through Azure AD, please install the NuGet Package for Microsoft.IdentityModel.Clients.ActiveDirectory.

After authenticating, we will create the group using the UnifiedGroup Utility provided through the SharePoint Online CSOM.

Below is a quick snapshot of the code. Note the inclusion of Graph module of the OfficeDevPnP class.

Note: One important bit to note is that, in the above code owners and members email array is the same. If the owners and members email array differ, then the group provisioning delays significantly. Also, it is important to keep the other parameters same as during creation in the below method because it might reset the other properties to default otherwise. For eg. if isPrivate is not set, then the group becomes public.

4. After the group is created, we can fetch the details of the group as below.

5. The group provisioning generally takes about 2-3 mins to provision. However, if there are multiple hits, then one request might override the second request causing the group creation to fail. In such cases, we can sequence the Azure Functions to run by modifying the host.json file. A quick blog covering this can be found here.

Provisioning SharePoint Assets in Azure Function after Group Creation

1. For provisioning of the SharePoint assets, we might have to wait for the Office 365 AD sync to finish granting access to the Admin account.

Sometimes, the AD sync process takes much longer, so to grant direct access to the SharePoint Site Collection using tenant admin, we could use the below code. Recommendation: Only proceed with the below code approach if the access fails for more than few mins.

Note: As a best practice, I would recommend using a Service Account when working on the SharePoint Site. We could also use an App as suggested in the Site Scripting blog here.

2. Once you have access, you can use the normal SharePoint CSOM to do the activities that are pertaining to SharePoint asset provisioning such as Libraries, Site Pages content, Lists, etc.

3. After you’re done, you can return the success from the Azure function as below.

Note: Use HttpStatusCode.Accepted instead of HttpStatusCode.Error in case there is error handling in the Flow or else Flow will trigger another instance of the flow when the Azure Function fails

Conclusion:

Above we saw how we can have a SharePoint Inventory list and create groups using Flow and Azure Functions. For a quick reference, below are the links to the other related blogs.

Part 1 – Automation and Creation of Office 365 groups approach

How to create a Microsoft Graph App

Sequencing calls in Azure Functions

Latest updates to Modern Libraries experience in SharePoint Communication sites (Apr 2018)

Modern Libraries in Communication Sites have got some welcome facelift during the last few months (Apr 2018) and there have been many great changes. I am going to list of few of these updates here.

Note: Some of these updates might be limited to Targeted release (or First release) versions only. In case these changes are not available then they might be not in Standard release ( or GA release) yet.

1. Full page view of SharePoint libraries

The SharePoint libraries now have a full page view, which provides it to use the full home page layout of Communication sites. It looks great 🙂

ModernLibExperienceSitePages

2. Custom Metadata support for Site Pages.

Now it is possible for newly created Communication sites (after Mar 2018) to have custom metadata updates with Site Pages content type.

Modern_Site_Pages_with_custom_content_type

Few catches in this scenario are:

1. It is still not possible to create a page by selecting a Child Site Page content type unless the child content type is set to default. When a page is created, it is set to default content type of the Site Pages library

2. Any communication sites, created prior to March 2018 mayn’t get this update. For associating metadata to site pages prior to Mar 2018, please check this blog for a custom approach to associate custom metadata to Site Pages. This will require custom code build for the same.

3. Support for more columns types through Modern UI Panel

Now we can create columns of additional metadata types such as Date, Choice and Picture with Modern Libraries, so don’t have to go to classic experience which is great from a UX and usability prespective.

ModernLibExperience_O3651

4. New command bar on Modern Libs

The Library command bar now provides a seemless experience of search and command items at the same level. Though small this is a great change because it would drive user to search content prior to creating new one.

NewCommandBar_SitePages

Conclusion

The above are some great updates for SharePoint modern libraries.

Set up a Microsoft Graph App for Office 365 and SharePoint Online management to use in Azure Functions, Flow, .Net solutions and much more

Microsoft Graph API can be used to connect and manage the Office 365 SaaS platforms such as SharePoint Online, Office 365 Groups, One Drive, OneNote, Azure AD, Teams (in beta) and much more.

A Graph app is an Azure AD app that has privileges (with provided permissions) to authenticate and then execute operations when using PowerShell, Azure Functions, Flow, Office Online CSOM, SharePoint Online and many other tools.

It is quite easy to set up a graph app, below is a brief preview of the process.

1. Go to the following link in your tenancy – https://apps.dev.microsoft.com/ and create an App. Below a brief screenshot of the App registration page.

GraphAppRegistrationScreen1

  1.  Then, first create a password and make sure to copy the password because it will be shown that time only. Also copy the App ID later for any future use.

GraphAppRegistrationScreen2

3. Then select the platforms that will be used to call the Graph app. For web calls use Web and for PowerShell scripts use Native as platform. You can leave the fields in the apps blank unless there is a specific endpoint that you would like to refer to. More information is provided at this link – https://developer.microsoft.com/en-us/graph/docs/concepts/auth_register_app_v2

Note: Turn off “Allow Implicit Flow” for web calls

4. Add the owners that could manage the App in Azure AD.
5. Next, select the proper application permissions that the App will need to run the actions. These settings are very important for your app to do the right calls, so try to set the appropriate settings. In some cases, it might be necessary to trial various app permission levels till you get it correct.

Note: For admin programs or scripts, it will be necessary to get Admin consent to the url below

https://login.microsoftonline.com//adminconsent?client_id=[clientid]&state=[something]

GraphAppRegistrationScreen7

6. Leave the other fields as is and create the App. We can turn off the ‘Live SDK support’ if not needed

GraphAppRegistrationScreen6

7. The App will show up in the Apps home page once created

GraphAppMyApplications

After the Graph app is created, we can use to perform various operations on Office 365 platforms. More details of the various operations are detailed here – https://developer.microsoft.com/en-us/graph/docs/concepts/overview.

There is also the beta release (https://graph.microsoft.com/beta/) which has more features upcoming in Graph Api.

Conclusion:

In the above blog, how we can create an Graph App that will allow us to connect to Graph Api and do operations with it.

Automation and Creation of Office 365 groups using Flow, Microsoft Graph and Azure Function – Part 1

Automating the creation of Office 365 groups via an event triggered process can help business teams use a consistent template across all their groups, especially if there are numerous groups provisioned throughout the year.

For example, in our case we have 500+ custom Office 365 groups that are created and maintained each year. Hence in this case it became obvious we wanted to spin up an Office 365 group from a SharePoint list where users could make the request on their own. Below is a quick overview of the approach and some of the learnings from it. Hopefully this helps.

In this blog – Part 1, we will discuss the approach to the Group creation process and important considerations for provisioning groups. In the upcoming blog – Part 2, we will look at using the Graph App for invoking the graph service and then implementation of the group provisioning process.

Solution overview

The solution uses the below applications.

1. SharePoint List – Data Entry for Groups requests
2. Microsoft Flow – Handles events on the list item create and update
3. Azure AD Apps – Generate App ID and App secret for Graph API calls
4. Azure Function – Automation Code for applying additional rules on the SharePoint sites after the Group is created

High level business logic

The high-level logic flow of the process is as follows. First users make an entry into a SharePoint list. Next step, is to invoke a flow from the SharePoint list item. The flow is based on the defined rules and calls the Azure Function with appropriate parameters. The Azure function then creates the group using the Graph service and assigns owners and members to the group. After the group is created, the Azure Function also applies the later changes to the Group using the SharePoint CSOM.

After the Azure Function has created the group and provisioned the SharePoint artefacts, it will send the success response as the result. In the case of an error, it will reply with an error response.Office365GroupsProvisioningFlow

Important Considerations

For the above process we must make a few important considerations before implementation. There may be slight modifications based on these.

Azure Function Call Sequencing

The most important part of the above process is how we handle parallelism. The Microsoft graph service is not capable of too many subsequent requests from a single tenant, so to make it more robust we can sequence the Azure Function to handle only one request at one time. This can be done by modifying the host.json of the Azure Function as below.

{
  "http": {
    "routePrefix": "api",
    "maxOutstandingRequests": -1,
    "maxConcurrentRequests": 1,
    "dynamicThrottlesEnabled": false
  }
}

Group and SharePoint Site Permission Sync – A LONG wait…

Group creation is managed through an integrated provisioning process handled by Microsoft, so not all artefacts of an Office 365 group are created at the same time.  Some components are created instantaneously, such as the Exchange mailbox and SharePoint Site, but others take time to provision such as the AD Office Group security sync.  In some cases, the AD Security sync can take as long as 15 min to finish.  As such, we must wait to obtain owner level access.

For provisioning the SharePoint artefacts, you can use this blog to set the SharePoint site to allow site scripts.

Error Handling

Since the Group creation process is a long running process, it will be important to make sure that any failure is handled properly, and the process is re-run later.

In our case, we will use the Flow to handle any error case until we have a success as shown below. If you’re using an Azure Queue Storage trigger for the Azure Function, then we can use the Poison queue for handle error runs.

Office365GroupsProvisioningErrorHandlingInFlow

Conclusion

In this blog post we covered the broad level process of the Group Provisioning process, and some important considerations associated with it. Next, we will see how we can implement the remaining processes in the follow up blog.

How to make Property Bag Values indexed and searchable in SharePoint Online

In an earlier post here we have seen how we can set Property Bag values in Modern SharePoint sites. One of the major reasons for setting Property Bag values in SharePoint sites is to make the SharePoint sites searchable based on custom metadata.

However, property bag values are not crawled by the SharePoint Online Search index directly. To make a property bag value searchable, we must explicit set the property bag values to be indexed by the Search crawler. To do this, we simply set the Property bag values to indexed using the SharePoint PnP PowerShell.

The “-Indexed” attribute of the Set-PnPPropertyBagValue cmdlet makes an entry in a hidden system property bag value vti_indexedpropertykeys which then makes these property bag values searchable. A screenshot of the result is below.

SetPnPPropertyBagIndexed.png

After performing the above, we flag the site to be reindexed so that the SharePoint search crawler will pick up the new Property Bag values in the next crawl. This can be done from “Site Settings -> Search and offline availability” (in the Search group), then click the button “Reindex site” (screenshot below).

Note: Since SharePoint Online is a massive SaaS system, it can take up to 24 hours for the crawler to pick up this property.  Unfortunately, there’s no workaround for this delay, you simply must be patient.

Search and Offline Availability

Conclusion

In the above post we saw how we can enable Property Bag values to be searchable.

 

Report of All Taxonomy Fields containing a term in SharePoint Tenancy

Recently we had a request to find fields/columns in all lists across the tenancy which have a specific Taxonomy term because we needed to report on field usage across all site collections. However, we found that getting a report of all Taxonomy fields in your SharePoint tenancy that is linked to a specific Term Set can get quite daunting because there is no direct SharePoint Query to fetch the associations.

The technical challenge is that using PnP PowerShell, the Taxonomy fields are returned as a generic SP.Field type and not of type SP.TaxonomyField. Hence the Taxonomy field metadata values such as Group ID and Termset ID are absent. To resolve the above limitation, we used the Field $field.SchemaXml to find the specified values.

Note: Querying the Term store while searching for a specific termset by using Get-PnPTerm can add a lot of latency time. Hence to decrease the additional time we could download the entire term store to an Excel file and use that Excel file as the master data for matching. Below is the command to get an export of all Taxonomy values as a CSV file.

Export-PnPTaxonomy -Path "[path]\taxonomyreport.csv" -Delimiter "," -IncludeID

Steps:

The steps to retrieve and check for taxonomy fields can be found below.

1. Get all lists in a web site
2. Get the Taxonomy fields for a List
3. Read the schema.xml and search for a TermsetID and AnchorID (Thanks to @Colin Philips (http://itgroove.net/mmman/) for finding the correct xpath for parsing xml)
4. Match the data with the above Taxonomy report for Group ID, Termset ID, and Anchor ID with Term ID that is column is linked to
5. In case of a match, save the values into the report.

The below code uses PnP PowerShell. For a quick set up of PnP PowerShell, please refer to this blog.

Conclusion:

Hence with the above approach, we can retrieve the taxonomy fields for all site collections.  Be aware that the above process can take about half a day or more to run depending on the number of site collections and taxonomy fields in your tenancy.  Be sure to give it enough time to run before prematurely cancelling it.

Happy Coding!! 😊

Promoting and Demoting Site pages to News in Modern SharePoint Sites using SPFx extension and Azure Function

The requirement that I will be addressing in this blog is how to Promote and Demote site pages to news articles in Modern SharePoint sites. This approach allows us to promote any site page to News, add approval steps and demote news articles to site pages if the news need to be updated. The news also shows in the modern news web part when the site page is promoted.

Solution Approach:

To start with, create a site page. For creating a Modern page using Azure Function, please refer to this blog. After the site page is created, we will be able to use a status column to track the news status and promote a site page to news status. The status column could have three values – draft, pending approval and published.

We will use a SPFx extension to set the values of the status column and call the Azure Function to promote the site page to news page using SharePoint Online CSOM.

Promoting a site page to news page

Below are the attributes that need to be set for site pages to promote as news article.

1. Promoted State Column set to 2 – set through SPFx extension
2. First Published date value set to published date – set through SPFx extension
3. Promoted state tag in the news site page to be set to value 2 – done in Azure Function
4. Site page needs to be published – done in Azure Function

For a detailed walkthrough on how to create a custom site page with metadata values, please refer to this blog. In order to set the values of ‘Promoted State’ and ‘First Published Date’ metadata values, use the below code after the page is created.

For calling Azure Function from SPFx extension, which will promote the site page to news, can be done using the below method.

Inside the Azure Function, use the below to promote a site page to news article.

Demoting a news to site page

Below are the attributes that needs to be set for demoting a news article to site page

1. Promoted State Column set to 0 – set through SPFx extension
2. First Published date value set to blank – set through SPFx extension
3. Promoted state tag in the news site page to be set to value 0 – done in Azure Function
4. Site page needs to be published – done in Azure Function

For setting the metadata values, the method calls as done above during promotion of site page, can be used. Next in Azure Function, use the below to demote a site page.

Conclusion:

Hence above we saw how we can use SPFx extension and Azure Function to promote and demote site pages to news articles in Modern SharePoint sites.

Creating SharePoint Modern Team sites using Site Scripts, Flow and Azure Function

With Site Scripts and Site design, it is possible to invoke custom PnP Provisioning for Modern Team Sites from a Site Script. In the previous blog, we saw how we can provision Simple modern sites using Site Scripts JSON. However, there are some scenarios where we would need a custom provisioning template or process such as listed below:

  • Auto deploy custom web components such as SPFx extension apps
  • Complex Site Templates which couldn’t be configured
  • Complex Document libs, content types that are provided by JSON schema. For an idea of support items using the OOB schema, please check here.

Hence, in this blog, we will see how we can use Flow and Azure Functions to apply more complex templates and customization on SharePoint Modern Sites.

Software Prerequisites:

  • Azure Subscription
  • Office 365 subscription or MS Flow subscription
  • PowerShell 3.0 or above
  • SharePoint Online Management Shell
  • PnP PowerShell
  • Azure Storage Emulator*
  • Postman*

* Optional, helpful for Dev and Testing

High Level Overview Steps:

1. Create an Azure Queue Storage Container
2. Create a Microsoft Flow with Request Trigger
3. Put an item into Azure Queue from Flow
4. Create an Azure Function to trigger from the Queue
5. Use the Azure Function to apply the PnP Provisioning template

Detailed Steps:

This can get quite elaborate, so hold on!!

Azure

1. Create an Azure Queue Store.

Note: For dev and testing, you can use the Azure Storage Emulator to emulate the queue requirements. For more details to configure Azure Emulator on your system, please check here.

Microsoft Flow

2. Create a Microsoft Flow with Request trigger and then add the below JSON.

Note: If you have an Office 365 enterprise E3 license, you get a Flow Free Subscription or else you can also register for a trial for this here.

3. Enter a message into the Queue in the Flow using the “Add message to Azure Queue” action.

FlowSiteDesignAzureQueue

Note: The flow trigger URL has an access key which allows it to be called from any tenant. For security reasons, please don’t share it with any third parties unless needed.

Custom SharePoint Site Template (PowerShell)

4. Next create a template site for provisioning. Make all the configurations that you will need for the initial implementation. Then create the template using PnPPowerShell, use the PnP Provisioning Command as shown below.

Get-PnPProvisioningTemplate -Out .\TestCustomTeamTemplate.xml -ExcludeHandlers Navigation, ApplicationLifecycleManagement -IncludeNativePublishingFiles

Note: The ExclueHandlers option depends on your requirement, but the configuration in the above command will save a lot of issues which you could potentially encounter while applying the template later. So, use the above as a starting template.

Note: Another quick tip, if you have any custom theme applied on the template site, then the provisioning template doesn’t carry it over. You might have to apply the theme again!

5. Export and save the PowerShell PnP Module to a local drive location. We will use it later in the Azure Function.

powershell Save-Module -Name SharePointPnPPowershellOnline -Path “[Location on your system or Shared drive]”

SharePoint
6. Register an App key and App Secret in https://yourtenant.sharepoint.com/_layouts/appregnew.aspx and provide the below settings.
7. Copy the App Id and Secret which we will use later for Step 9 and 10. Below is a screenshot of the App registration page.
8. Trust the app at https://yourtenant-admin.sharepoint.com/_layouts/appinv.aspx by providing the below xml. Fill in the App Id to get the details of the App.

Azure Function

9. Create a Queue Trigger PowerShell Azure function
10. After the function is created, go to Advance Editor (kudu) and then create a sub folder “SharePointPnPPowerShellOnline” in site -> wwwroot -> [function_name] -> modules. Upload all the files from the saved PowerShell folder in the Step above into this folder.
11. Add the below PowerShell to the Azure Function

12. Test the Function by the below input in PowerShell

$uri = "[the URI you copied in step 14 when creating the flow]"
$body = "{webUrl:'somesiteurl'}"
Invoke-RestMethod -Uri $uri -Method Post -ContentType "application/json" -Body $body

PowerShell and JSON

13. Create a Site Script with the below JSON and add it to a Site Design. For more details, please check the link here for more detailed steps.

14. After the above, you are finally ready to run the provisioning process. Yay!!

But before we finish off, one quick tip is that when you click manual refresh, the changes are not immediately updated on the site. It may take a while, but it will apply.

Conclusion:

In the above blog we saw how we can create Site templates using custom provisioning template by Flow and Azure Function using SharePoint site scripts and design.

Create Modern Pages and update metadata using SPFx Extensions, SP PnP JS and Azure Functions

Modern Site Pages (Site Page content type) have a constraint to associate custom metadata with it. In other words, the “Site Page” content type cannot have other site columns added to it as can be seen below.

SitePageContentTypeMissing

On another note, even though we can create a child content types from Site Page content type, the New Site page creation (screenshot below) process doesn’t associate the new content type when the Page is created. So, the fields from the child content type couldn’t be associated.

For eg. In the below screenshot, we have created a new site page – test.aspx using “Intranet Site Page Content Type” which is a child of “Site Page” content type. After the page is created, it gets associated to Site Page Content type instead of Intranet Site Page Content type. We can edit it again to get it associated to Intranet Page content type but that adds another step for end users to do and added training effort.

 

 

 

Solution Approach:

To overcome the above constraints, we implemented a solution to associate custom metadata into Modern Site Pages creation using SharePoint Framework (SPFx) List View Command Set extension and Azure Function. In this blog, I am going to briefly talk about the approach so it could be useful for anyone trying to do the same.

1. Create a List View Command Item for creating site pages, editing properties of site pages and promoting site pages to news
2. Create an Azure function that will create the Page using SharePoint Online CSOM
3. Call the Azure Function from the SPFx command.

A brief screenshot of the resulting SPFx extension dialog is below.

NewSitePage

Steps:

To override the process for modern page creation, we will use an Azure Function with SharePoint Online PnP core CSOM. Below is an extract of the code for the same. On a broad level, the Azure Function basically does the following

1. Get the value of the Site Url and Page name from the Query parameters
2. Check if the Site page is absent
3. Create the page if absent
4. Save the page

Note: The below code also includes the code to check if the page exists.

Next, create a SPFx extension list view command and SP dialog component that will allow us to call the Azure Function from Site Pages Library to create pages. The code uses ‘fetch api’ to call the Azure Function and pass the parameters for the Site Url and page name required for the Azure Function to create the page. After the page is created, the Azure function will respond with a success status, which can be used to confirm the page creation.

Note: Make sure that the dialog is locked while this operation is working. So, implement the code to stop closing or resubmitting the form.

After the pages are created, lets update the properties of the item using PnP JS library using the below code.

Conclusion:

As we can see above, we have overridden the Page Creation process using our own Azure Function using SPFx List View command and PnP JS. I will be detailing the SP dialog for SPFx extension in another upcoming blog, so keep an eye for it.

There are still some limitations of the above approach as below. You might have to get business approval for the same.

1. Cannot hide the out of the box ‘New Page’ option from inside the extension.
2. Cannot rearrange order of the Command control and it will be displayed at last to the order of SharePoint Out of the box elements.

Update Managed Metadata and Hyperlink column in a SharePoint list using PnP PowerShell

If you are trying to update a Managed Metadata and Hyperlink column in SharePoint online using PnP PowerShell and it is not getting updated, then this blog might be of help.

I had been working on a quick requirement for updating Managed Metadata columns using a PowerShell script for a bulk update requirement and it was not working as planned. The managed metadata column we were updating didn’t have the right format of Term store hierarchy and this was the cause of the issue.

Managed Metadata

The Managed Metadata termstore format below, will not work if anything is missed in the hierarchy or symbols.

$val = "Group|TermSet|Term1|Term2..."

So for eg. Your group is “Test Group” with Termset “Test Termset1” with Level 1 Term “Parent” and Level 2 Term “Child”, then the value is:

$val = "Test Group|Test Termset1|Parent|Child"

The update command in PnP Powershell will be:

Set-PnPListItem -List "[Lib Name]" -Identity [ItemID] -Values  @{"Managed Metadata Column" = $val}

Hyperlink

This one was amusing because I didn’t see the mistake till I realised that strings are not comma delimited, but space delimited! 🙂

HyperlinkImage

The format for updating a hyperlink is below. Notice the space after the comma (,) it is very important to get this right.

Set-PnPListItem -List "[Lib Name]" -Identity [ItemID] -Values  @{"Hyperlink Column" = $val}

Conclusion

So above we saw how we can update Managed Metadata and Hyperlink columns using PnP PowerShell.