Content Deployment, CQWP, Cross List Queries, Object Cache, and SharePoint 2007 Performance

Thursday, 10 April 2008 21:17 by RanjanBanerji

Oh what tangled webs we weave and in this case without an intent to deceive.

In the past few months I have been introduced to SharePoint 2007 and WSS 3.0 and as my blog shows I have had my fair share of frustrations.  It started with performance on the Content Query Web Part, led to my learning about Cross List Queries and the Object Cache.  I then switched to Content deployment and all its complexities.

Little did I know that the two will team up and raise new levels of hell for me.  Rather dramatic isn't it?

So we know that the CQWP uses cross list queries to get data.  Cross List Queries in turn utilize the object cache.  For best CQWP performance you want to make sure that your object cache is optimally configured.   I talked about this issue here.

In another galaxy far far away completely unrelated from CQWPs and Cross List Queries is the world of Content Deployment.  I have written several posts on Content Deployment.  One of them focuses on a Content Deployment approach that recommends creating several smaller jobs in order to get content from a large database across to another farm.

So what on Earth, I mean what on the galaxy far far away is the relationship between content deployment and content query web parts (CQWP) and Cross list queries?

Well it all started long long ago in a galaxy....  well you get the point.  I was working on an application that was deploying content using 8 small jobs every 30 minutes.  The 8 small jobs were created to break the deployment tasks of the whole application into manageable chunks of data as the network we were dealing with was full of problems.  The jobs would kick off on the hour and 30 past the hour.  During development everything went fine.  However, Once the application went live with hundreds of users hitting it the story changed.  The performance of the application was horrible.  Well not exactly.  It was horrible for about 5 minutes every 30 minutes on the hour and 30 past.

Coincidence?  At first I thought so.  But the correlation was too high.  So I turned off content deployment and performance improved.  Everything was back to normal.  But what was it about content deployment that made the application perform so poorly?  When I mean poorly I mean a page that loads in 1 second was taking 45 seconds to a few minutes.  So something was killing the application.

Well not really.  You see its not the entire application that was slow.  Only a few pages that loaded so slow.  The rest of the site acted absolutely normal.  So what was special about the slow pages?  Well they all had one or more content query web parts on them.  Hmmmm! so why were pages with CQWPs performing so poorly when a content deployment took place. 

My first step was to eliminate any kind of database transactions that may be occurring during content deployment.  So I ran content deployment with zero objects to deploy.  Zero exported and zero imported.  Yet, just as content deployment completed, performance of pages with a CQWP on it tanked.

Then it hit me.  I recalled my own post about optimizing content query web part performance and the importance of the object cache.  So I did the following to watch what happens when content deployment occurs/completes.

  • I turned on perfmon and watched the SharePoint cache hit ratio when the site was being hit by many users.  The hit ratio was consistently above 99%.  This was good.
  • I turned on SQL profiler during the same time and watched no queries being executed.  Well none that were being issued by the CQWP/Cross List Queries.
  • I kept watching as a content deployment job started and then completed. Ahaaa! SQL Server got slammed with 100s of queries all coming from CQWPs.

So content deployment wiped my object cache?  Why? Why? Why? 

So on the one hand we know how important the object cache is and on the other we have in the past talked about how you can break content deployment into several smaller jobs.  Unfortunately the two strategies do not work well together.  Apparently Microsoft has a patch out for fixing this problem or is about to release one.  Until then be prepared to lose performance if you have an application that has frequent content deployments. 

Steps you can take to mitigate the problem are:

  • Reduce the number of jobs running at any given point in time.  Imagine if you have 5 jobs running every hour on the hour.  Not all 5 will complete at the same time.  Lets say they finish between 5 to 8 minutes after they start, i.e., the hour.  So in 5 to 8 minutes past the hour your object cache will be wiped clean 5 times.  You really do not want that happening.
  • Don't run jobs too frequently.
  • Wait for Microsoft to get their patch out.

 

Tags:  
Categories:   SharePoint
Actions:   E-mail | Permalink | Comments (1) | Comment RSSRSS comment feed

SharePoint Content Deployment - Identifying Object Causing Error

Monday, 7 April 2008 22:13 by RanjanBanerji

Yet another problem with SharePoint 2007 content deployment is that its errors are not very informative.  The fact that content deployment is a delicate feature that tends to break with the slightest hiccup one would think that Microsoft should at least log out a sensible message.

Here is an error I got a few days ago:

The given key was not present in the dictionary. at System.ThrowHelper.ThrowKeyNotFoundException() at System.Collections.Generic.Dictionary`2.get_Item(TKey key) at Microsoft.SharePoint.Deployment.ListItemSerializer.GetFileUrl(SerializationInfoHelper infoHelper, SPWeb web) at Microsoft.SharePoint.Deployment.ListItemSerializer.CreateListItem(SerializationInfoHelper infoHelper, Int32 listItemId, SPWeb web, SPList list, Boolean isIdTaken, String& listItemServerRelativeUrl) at Microsoft.SharePoint.Deployment.ListItemSerializer.SetObjectData(Object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector) at Microsoft.SharePoint.Deployment.XmlFormatter.ParseObject(Type objectType, Boolean isChildObject) at Microsoft.SharePoint.Deployment.XmlFormatter.DeserializeObject(Type objectType, Boolean isChildObject, DeploymentObject envelope) at Microsoft.SharePoint.Deployment.XmlFormatter.Deserialize(Stream serializationStream) at Microsoft.SharePoint.Deployment.ObjectSerializer.Deserialize(Stream serializationStream) at Microsoft.SharePoint.Deployment.ImportObjectManager.ProcessObject(XmlReader xmlReader) at Microsoft.SharePoint.Deployment.SPImport.DeserializeObjects() at Microsoft.SharePoint.Deployment.SPImport.Run()

 

Great! and just what am I supposed to do about this?  After speaking to our Microsoft tech support folks I found out that this error can occur if a page has been deleted then recovered from the recycle bin.  Apparently Content Deployment does not like that.  Fair enough, but wait, the job I was running deployed 830 items.  The application is used by thousands of users.  So how am I to find out which of the 830 items was the culprit, as in was deleted and then recovered.  The error message does not tell you the name or ID of the object in question.  Thanks so much Microsoft and don't tell users that this information is hard to get at.  If you are dealing with an object you better know its name, ID or some usable identified before reporting the error.  Else why even report the error?

Now Microsoft and its MVPs have spent a lot of time calling SharePoint a development platform rather than a poorly stitched up hack. Let me prove to you why they are all correct.  You see, if you write a little code using the SharePoint object model you will be able to get some additional information that Microsoft decided not to share.  It's not a bug, its a feature.  This is why one can claim its a development platform.

OK, enough cynicism.  I have many other gems to share on that front in upcoming blog posts.  For now let's try to find a way to get some help.

You can create a console application (or wait forever for me to complete my SharePoint Helper application) to make once change on how Content Deployment works.

In the application get the Content Deployment Path object and set KeepTemporary to true.

Microsoft.SharePoint.Publishing.Administration.ContentDeploymentPath.GetInstance( pathName ).KeepTemporaryFiles

Now run Content Deployment.  When complete as in when you get an error do the following:

  • Look at the Content Deployment Job's details.  It will say some thing like 800 objects exported, 250 imported and that it failed.  Keep teh 250 number in mind.
  • Collect the error message and hope someone has an explanation as to what that error implies.
  • On the source farm's Central Admin computer go to the TEMP folder where SharePoint will now leave the files used to export content.  You can find this by reading http://techblog.ranjanbanerji.com/post/2008/02/13/SharePoint-2007-Content-Deployment-Error-Out-of-disk-space.aspx
  • In the temp folder you will see a folder named ContentDeployment with sub folders named with GUIDs.  One of the GUID named folders is your job that failed.  Check the timestamp of the files in the folders, not the folder itself and find  which is the one you need.
  • In the folder you will find several CAB files.  Find the latest one and open it.  In it you will find several files including the manifest.xml file.
  • Open the manifest.xml file and find the 251st SPObject element (because your job reports said failure after importing 250 objects)
  • If you are lucky this is the object that failed.  Get it's details and start researching what went wrong.

 

If you get an error while exporting you can sift through the ULS logs or create a console application with a few lines of code as such:

using Microsoft.SharePoint.Publishing.Administration;

namespace TestRun {
    class Program {
        static void Main( string[] args ) {
            string jobName = args[ 0 ];
            ContentDeploymentJob job = Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.GetInstance( jobName );
            job.Run();
            job.Dispose();
        }
    }
}

jobName is the name of a job that you have already created using the Central Administration.  When you run your console application you will see that SharePoint Content Deployment code will automatically spit out every object being exported to the console.  If it fails on any object, well the object name and error will be right there in front of you.  What is also interesting is that when you run a job from the console you can go to the Central Administration Content Deployment Page and see that the job is running and monitor its progress from there too.

So, now you have two code based approaches to help you debug your Content deployment problems.  Simple isn't it? ;-)

 

Tags:  
Categories:   SharePoint
Actions:   E-mail | Permalink | Comments (4) | Comment RSSRSS comment feed