Connected data source error code: 8344: insufficient access rights to perform the operation.

Originally blogged @ Lucian.Blog. Follow Lucian on Twitter @LucianFrango.

---

I’m in the final stages of a long running Exchange migration from two on-premises ADDS forests and Exchange organisations to Exchange Online. The infrastructure foundations were laid out by some Kloudie colleagues some time ago. The environment has been running great for a while now, however, recently when trying to do some remote move migration batches to Exchange Online, I’ve been running into failures.

A few months ago I had the same issue and at that time I quickly found it to be related to DirSync. This project has an older deployment of DirSync with some customisation specific to this environment. That time I managed to find some duplicate attributes between the DirSync metaverse and Active Directory on-premises for the problematic users, which for the most part, was the cause of the problems. Fast forward and this week and I’m again running into some migration failures.

The problem

This deployment of DirSync has been in production for a year now. There are close to 40,000 objects synced to Azure AD and there has already been 10,000 mailboxes migrated to Exchange Online. What that word picture paints is a running live/production environment.

The DirSync server when running sync tasks for the internal Active Directory schedules, under the Export task, started to get errors. Allot of them. The errors were related to “permission-issue”.

I spent a bit of time investigating this. I looked at what process it was doing and why it needed permissions. Down the internet rabbit hole I ended up on TechNet and looking through various articles around DirSync and permissions until I came across a line of thinking around the MSOL_AD_Sync_RichCoexistence account.

Then it stuck me- that satisfying feeling of the eureka moment!

#EurekaMoment is one of the best feelings when, late in the day, things start going your way.. pic.twitter.com/utlWZXucgC

— Lucian Franghiu (@lucianfrango) November 30, 2015

The solution

The solution that worked in this instance came from a TechNet with a powershell script to re-apply some permissions in ADDS. The page on TechNet has the following description for the scripts purpose:

Looks at the permissions currently assigned to “MSOL_AD_Sync_RichCoexistence” at the root and applies them to the OU or object you specify. You’ll want to run it from your Exchange 2010 or later EMS with an account that has permissions to modify the ACL on the target object. In the script you’ll specify the DN for the domain where the “MSOL_AD_Sync_RichCoexistence” group resides and the DN for the object you want to add the permissions to.

If while running the script, you receive an error stating “INSUFF_ACCESS_RIGHTS”, you may need to execute the steps in Microsoft KB2983209 before running the script.

The script itself looks like as follows:

I used Josephs script and it did indeed work. Here’s the process that I used to get through the problem and back to mailbox migrations:

• From a server with Powershell v2.0 and modules for Active Directory and Exchange 2010, I was able to specify the domain distinguished name and the user distinguished name to run the script
  • NOTE: As well as running the permissions change/update on a per user basis, it can also be run on an OU in ADDS
• In the script I also specific the MSOL_[XXXXXXXXXXXX] account that the DirSync service was using
• This MSOL_[XXXXXXXXXXXX] account requires the correct permissions for DirSync to write back to ADDS objects
• Once the script was run, I ran a repadmin /syncall to force ADDS replication
• After about 15min when ADDS should have replicated, I ran the DirSync manual sync tasks
• I ran both the Delta Import/Sync + Export tasks for both the on-prem ADDS connections, as well as the WAAD connector
• Once DirSync had finalised I then created new migration batches in Exchange Online and successfully SWRC’ed 30 mailboxes, –woohoo!

This all sounds like happy days. However, for those 30 mailboxes, as an initial pilot, I ran the script 30 times. Each time a new mailbox DistinguishedName was added. That process took all of 10min to do, but, I had another 700 mailboxes to run the same task against. That would have definitely not made my afternoon.

Big credit to Joseph (@joepalarchio) for setting the foundations for the solution with the above script, but, I needed to modify it to allow me to quickly and effortlessly run the task against many more accounts / mailboxes. The following is the end result which uses CSV files with users DistinguishedName attributes imported and powershell running “foreach” tasks against:

To use the above script: create a simple, single column CSV file with the header of “DistringuishedName”, then add in a single column all the users distinguished names. Put the CSV file in the desired folder and update the path in the script above. Finally, change the your domain details and the MSOL_ user account DirSync uses and run the script. Happy days!

Interesting…

With this updated script I breezed through the mailboxes I needed to run the fix against. In DirSync I could see that after I ran import tasks that the metaverse was updated and the errors for the users disappeared as intended. I left this as is for sometime and kicked off a few successful mailbox SWRC batches. Out of curiosity though I went back to DirSync to review the metaverse for those users again. I wanted to see if indeed the errors and problems were cured once and for all. In this instance unfortunately theres something else underling. While the mailboxes and user objects are syncing with Exchange Online / Office 365 just fine, DirSync still has the same errors. I’m not out of the woods yet, but, I did see some significant progress so therefore I’m out of tough spot for the time being.

Final words

I can’t say for certain that this process in this post is a complete fix or solution, but, for the scenario I’m in it’s helped get a whole bunch of mailboxes up to Office 365 / Exchange Online. I wanted to share this incase others run into the same issue and the process can at least get, for the most part, back on track with migrations.

I’ve had further discussions around the issue and we’ve found some ADDS related permissions are also at play here. As I mentioned earlier that the permissions script can be run against OU’s in ADDS, it’s likely that the main problem here in this deployment is around that, rather than permissions on individual user objects. A process to complete that is going to be put forward and I’ll post feedback and updates to this blog if that resolves the issue entirely.

Thanks for reading,

-Lucian

---

Originally blogged @ Lucian.Blog. Follow Lucian on Twitter @LucianFrango.