File Element OverwriteIfAlreadyExists=”TRUE”

I have been implementing workarounds since SharePoint 2007 to overwrite an unghosted (or customized) file that already exists at a specified URL for a File element within a Module element of a Feature element manifest (or site definition which I stopped using extensively in SharePoint 2010 preferring web templates).

Updating existing files when upgrading our SharePoint custom Features can be accomplished by provisioning ghosted files using the Type attribute set to Ghostable or GhostableInLibrary. For unghosted (or customized) files a common trap is setting IgnoreIfAlreadyExists=”TRUE” expecting existing files to be overwritten. The MSDN Library documentation description for the File element IgnoreIfAlreadyExists attribute contributes to the confusion because it does not behave as expected:

Optional Boolean. TRUE to provision the view even if the file already exists at the specified URL; otherwise FALSE.

To overcome the limitation of not being able to update unghosted files using the IgnoreIfAlreadyExists attribute which is a very common scenario for SharePoint Online (where we do not have a reference to ghosted files on the file system) read on…

There are two possible approaches depending on whether your unghosted files exist in SharePoint 2010 or SharePoint 2013. For SharePoint 2013 you can now use the ReplaceContent attribute (which is not yet documented in MSDN Library) in the file element like this:

<File Path=”Branding.css” Url=”Style Library/Branding.css” ReplaceContent=”TRUE” />

One limitation I noticed is ReplaceContent will not overwrite files that are checked-out to a different user than the person provisioning the file. To overcome this scenario in SharePoint 2013 and update unghosted files in SharePoint 2010 we need to fall-back to custom code in a feature receiver.

We can use the GetElementDefinitions method of the SPFeatureDefinition returned by SPFeatureReceiverProperties to identify Feature element files to overwrite in full trust solutions only. Identifying the Feature element files in SharePoint Online is a little bit more challenging as both the GetElementDefinitions method and RootDirectory property are not available in Sandbox solutions.

In order to read the element manifest and overwrite already existing unghosted files in SharePoint Online we need to provision the element manifest file and additional copies of the element files (remember IgnoreIfAlreadyExists=”TRUE” does not replace existing files).

<File Path=”Branding.css” Url=”Style Library/Branding.css” IgnoreIfAlreadyExist=”TRUE” />
<File Path=”Elements.xml” Url=”Branding_$SharePoint.Feature.Id$.xml” Type=”Ghostable” />
<File Path=”Branding.css” Url=”Style Library/Branding.css.copy” IgnoreIfAlreadyExist=”TRUE” />

The element manifest file is provisioned to the root folder of the parent web as {Prefix Name}_{Parent Feature ID}. $SharePoint.Feature.Id$ replaceable token is used to provide the parent feature id. If the Feature definition has multiple element manifest files that require element files to be overwritten we must provide a unique {Prefix Name} for each element manifest.

A copy of each element file to be overwritten in the element manifest is provisioned with a .copy file extension. This allows the feature receiver custom code to parse the provisioned element manifest file to identify which element files to overwrite and more importantly retrieve .copy element files to replace existing unghosted files with.

We created a helper function to process the provisioned element manifest and .copy element files to overwrite existing unghosted files:

public static void CopyFiles(SPWeb web, Guid featureID)
{
// create a regular expression pattern for the feature element manifest files
string pattern = string.Format(@”^.+_{0}.xml$”, featureID);
Regex fileNameRE = new Regex(pattern, RegexOptions.Compiled | RegexOptions.IgnoreCase);
// get the feature element manifest files from the root folder of the site
SPFile[] manifestFiles = web.RootFolder.Files.Cast<SPFile>().Where(f => fileNameRE.IsMatch(f.Name)).ToArray();
try
{
// iterate the feature element manifest files
foreach (SPFile manifestFile in manifestFiles)
{
// load the contents of the element manifest file in an XDocument
MemoryStream mStream = new MemoryStream(manifestFile.OpenBinary());
StreamReader reader = new StreamReader(mStream, true);
XDocument manifestDoc = XDocument.Load(reader, LoadOptions.None);
/ / iterate over the ‘Module’ and ‘File’ elements in the XDocument, concatenating their Url attributes in a smart way so that we grab the site relative file Url-s
string[] fileUrls = manifestDoc.Root.Elements(WS + “Module”).SelectMany(me => me.Elements(WS + “File”), (me, fe) => string.Join(“/”, new XAttribute[] { me.Attribute(“Url”), fe.Attribute(“Url”) }.Select(attr => attr != null ? attr.Value : null).Where(val => !string.IsNullOrEmpty(val)).ToArray())).Where(x => x.EndsWith(“.copy”)).ToArray();
bool copyFilesCheckedOut = false;
// Check for checked-out .copy files
foreach (string fileUrl in fileUrls)
{
string copyFileUrl = fileUrl.Substring(0, fileUrl.Length – “.copy”.Length);
SPFile file = web.GetFile(copyFileUrl);
if (file.Exists && file.CheckOutType != SPFile.SPCheckOutType.None)
{
copyFilesCheckedOut = true;
break;
}
}
// Overwrite existing unghosted files
foreach (string fileUrl in fileUrls)
{
// get the .copy file
SPFile file = web.GetFile(fileUrl);
if (!copyFilesCheckedOut)
{
// get the existing unghosted file
string copyFileUrl = fileUrl.Substring(0, fileUrl.Length – “.copy”.Length);
// SPFile.CopyTo() does not preserve file version history;
// file.CopyTo(copyFileUrl, true);
// overwrite existing unghosted file preserving version history
SPFile copyFile = web.GetFile(copyFileUrl);
copyFile.CheckOut();
copyFile.SaveBinary(file.OpenBinary());
copyFile.CheckIn(string.Empty, SPCheckinType.MinorCheckIn);
// depending on the settings of the parent document library we may need to check in and/or (publish or approve) the file
if (copyFile.Level == SPFileLevel.Draft)
{
if (copyFile.DocumentLibrary.EnableModeration)
{
copyFile.Approve(string.Empty);
}
else
{
copyFile.Publish(string.Empty);
}
}
}
// remove the .copy file so we can provision newer versions
file.Delete();
}
}
}
finally
{
// remove feature element manifest files from the site root folder so we can provision newer versions
foreach (SPFile manifestFile in manifestFiles)
{
manifestFile.Delete();
}
}
}

Finally we add the feature receiver custom code to call the helper function:

public override void FeatureActivated(SPFeatureReceiverProperties properties)
{
SPWeb web = null;
if (properties.Feature.Parent is SPWeb)
{
web = (SPWeb)properties.Feature.Parent;
}
else if (properties.Feature.Parent is SPSite)
{
web = ((SPSite)properties.Feature.Parent).RootWeb;
}
if (web!=null)
{
CopyFiles(web, properties.Feature.DefinitionId);
}
}

The feature receiver custom code approach for SharePoint 2013 allows us to validate all the element files to be overwritten are checked-in before replacing them. This is particularly important for client-side Apps where failure to overwrite all App .html, .js and .css files can lead to unpredictable behaviour. The CopyFiles function can also be modified to override checked out files and force overwrites.

A note of caution about using the feature receiver custom code in SharePoint Online. Sandboxed code execution during a single request cannot exceed 30 seconds or the user code service will recycle the application domain and the request will return an error. This may require splitting large numbers of element files that need to be overwritten into multiple Features and element manifests.

If you need to overwrite element files in SharePoint 2013 start using the new ReplaceContent attribute unless you are experiencing issues with overwriting checked-out files. If you need to overwrite element files in SharePoint 2010 I hope you found this blog post useful.

SharePoint Code Quality

During a recent technical interview I asked the candidate which tools they used to review code quality for their SharePoint custom solutions.
This blog post describes the process and tools we use to check SharePoint code quality for a SharePoint Online 100K user deployment I have been working on.

Our project team uses Team Foundation Service (TFS) Cloud for source code management and continuous integration.  We perform code quality checks as part of TFS Team Build and Release Management processes.

SPCAF and MSOCAF code quality checkpoints

Figure 1: SPCAF and MSOCAF code quality checkpoints

The release management process for our project implements separate SharePoint Online tenants for Development (Integration), Staging (UAT), and Production environments. SPCAF provides code quality checks in the Development environment.  MSOCAF is used to provide consistency in packaging of releases to the Staging and Production environments.

Our SharePoint custom code must pass 400+ SPCAF code checking rules and MSOCAF test cases defined by the SharePoint Online engineering team before it can be approved for deployment to the Staging and Production environments.

Both tools provide extensive code checks to validate custom code against SharePoint development best practices and guidance.  SPDisposeCheck, FxCop, CATNET, and FxCop Metrics analysis tools are integrated into SPCAF and MSOCAF code review.

SharePoint Code Analysis Framework

SPCAF provides build activities that can be integrated into Team Build workflow.  For projects that do not use TFS or environments where Visual Studio is not available, the SPCAF client application can be used to perform analysis of WSP files or apps.

SPCAF client application

Figure 2: SPCAF client application

SPCAF provides rule check for WSP package and SharePoint 2013 apps to identify possible problems or coding errors in assemblies, XML files, ASPX files, CSS files etc. The rules check the artefacts and in case of violations they return the result as CriticalError, Error, CriticalWarning or Warning.  The Ruleset Editor can be used to disable rules or rule categories or to change the severity of a rule.  SPCAF includes a SDK to create custom rules, metrics or dependency checks.

SPCAF ruleset editor

Figure 3: SPCAF ruleset editor

SPCAF generates rule violations, metrics, dependencies, and inventory reports included as part of each release for our SharePoint Online application.

SPCAF code analysis results

Figure 4: SPCAF code analysis results

The SharePoint Rule Violations Report includes all rule checks that have failed.  A nice feature enhancement would be to provide the ability to override the rule check and provide a comment (similar to MSOCAF tool).
Some of our rule violations may be false positives depending on the deployment scenario . We include the HTML report together with a document providing reasons for any rule violations as part of our release management process.

SPCAF rule violations reporting

Figure 5: SPCAF rule violations reporting

The SharePoint Metrics Report provides a breakdown of the artifacts in our SharePoint custom application.  We use the metrics report to monitor the how our SharePoint code base is evolving across different project releases.

SPCAF code metrics reporting

Figure 6: SPCAF code metrics reporting

The SharePoint Dependency Report provides detailed information about dependencies in our SharePoint custom application.  I particularly like the DGML diagram for visualizing application dependencies but have not found a DGML viewer other than Visual Studio.

SPCAF code dependencies reporting

Figure 7: SPCAF code dependencies reporting

The SharePoint Application Inventory Report provides detailed documentation about the contents of our SharePoint custom application.  Both Dependency and Inventory reports save hours of effort documenting our SharePoint custom application for each release.

SPCAF code inventory reporting

Figure 8: SPCAF code inventory reporting

Microsoft SharePoint Online Code Analysis Framework

The MSOCAF tool ensures each release of our SharePoint custom application is packaged using a standard folder structure.

MSOCAF folder structure

Figure 9: MSOCAF folder structure

MSOCAF test cases are created by the Microsoft SharePoint Online engineering team and used to review code quality for each release of our SharePoint custom application before it is deployed to the Staging (UAT) environment.  The MSOCAF tool is built using an extensible framework so that the SharePoint Online engineering team can add new rules and/or plug-ins in the future. Code analysis within MSOCAF focuses on areas like memory management, security vulnerabilities, exception management, object model usage, quality gates for unsupported features and reporting.

MSOCAF test cases

Figure 10: MSOCAF test cases for custom code

Using SPCAF and MSOCAF as part of our build and release management process has been extremely helpful in improving the overall code quality and release procedures for our SharePoint projects at Kloud Solutions.