18th of July, 2018 / / No Comments

A couple months ago, I had the opportunity to help a client to patch their Sharepoint on-premise 2013, which was last patched up to 2014. It was a challenging but interesting experience. We decided to use the Nth-1 patch, which was March 2018 at that time. It was a 5-weeks engagements, where we had to break down the steps and processes from back-up preparations, roll back strategies, on-patching strategies to post-patching testings. Lots of inputs and discussions among DBAs, system engineers, IT manager and Sharepoint engineers. After all, it was a worthwhile experience as we all learnt a lot from it. Here’ i am breaking down the summary in what we have done.
First of all, we need to lay out the path from early stage to implementation and completion. The high level steps are as:
Inventory check on Sharepoint sites
It is very important to do an inventory check through the existing sites. Do a breakdown list of major sites and their owners, find out which are the most used sites, find out about the amount of storage being used, find out how many custom farm solutions there are.
Test through all aspects of Sharepoint
Application level and backend (central admin) level – We need to be confident with the existing functionalities in Sharepoint, whether they are out of box or custom. Testings need to be done before anything, this is to ensure what you test after the patching still work the same way as before patching.
I would also recommend backing up the existing search service application, just in case it gone corrupted after patching. Remember, anything can happen with the patching, regardless of whether it worked on DEV or TEST farms as each farm configurations can have different configurations and settings.
I would also suggest taking lots of screenshots in Central Admin so that you have a record of existing settings and configurations to compare at later stage.
Backup content databases
Liaise with DBAs to perform backups for content databases. This will likely to take overnight depending on the sizes of your farm and tool being used. Should it have completed successfully, we need to liaise with the system engineer to perform snapshots on all VMs. This should take less time than the content databases backup, however all VMs need to be shutdown prior.
The important thing to note is, we are trying to capture the current state of the farm, so any mis-sync between the VMs or databases would result in a corrupted farm.
Stopping all Sharepoint services
It is important to stop these critical Sharepoint services: IISAdmin, SPTimerV4 and Search. There is Powershell script that can do this but you could also stop them in local server’s Services window.
Applying CU patches
Always login using the farm account on each server.
The order of applying Sharepoint patch is critical. The order we chose to apply was as follow:

  • Front end servers
  • Backend servers
  • Search servers

You can run the patch asynchronously on all server as it doesn’t interfere with each other. The patch only updates the local file system, it doesn’t update the database just yet. Each patch should take no longer than half an hour to complete. Upon completion, restart the server.
Run PSConfig.exe
This must be run in order for the patching process to complete, or else the server status page in Central Admin will not show the updates being applied! I recommend using the command line version rather than the GUI version as we found the GUI version likes to hide error messages and just shows a completion message.
The command we used was this:
PSConfig.exe -cmd upgrade -inplace b2b -wait -cmd applicationcontent -install -cmd installfeatures -cmd secureresources -cmd services -install
The order of running this is also critical. We ran it on App servers first, then front end servers and lastly search servers. Always check on the results, any error or issue would likely be reported here. You need to fix all issues and re-run the command until a success message is shown.
If the patch has been applied successfully, Central Admin -> Upgrade and Migration -> Check product and patch installation status page would immediately reflect the update. Also, check the database configuration version reflects the right new version number.
Note, one issue we had to fix was the removal of orphan features found in various sites. There were a number of these orphan features which was reported by the tool and we used Powershell script to manually remove one by one.
Post-patching testing
Once all patches have been applied, ensure all servers are back up running. Check all the major services are running (i.e. Central Admin, IIS, Search, etc). This is when snapshots from your pre-testing come handy if you can’t remember what services should be running on what server. There are also ULS logs which you could check and scan through to see if there’s any critical issues found.
Then we started with post-patching testings. This should be the same test plans as the pre-testing and the result should match as well.
Some useful links for reference:

 
 
 

Category:
SharePoint
Tags: