Content deployment gotchas

When planning for content deployment there is one thing that is important to watch out for: Hard drive space. Specifically, the C: drive. When you deploy content, it is downloaded, compressed, copied, stored, uncompressed and loaded into sharepoint. Sometimes, this process fails and you're left with data lying around. The biggest problem is space. You need at least twice the size of your deployment as free space on your drive. When setting up deployment, you are given the option to setup the location where files are stored temporarily. So you think to yourself, great! I'll set this up on my SAN and have plenty of space. Sadly, this doens't work. This free space is only used to store the CAB files that are moved. It is not used to store the files when they are uncompressed. The cab files are stored in the folder you define(C:\programdata\contentdeployment is the default i believe), but then they are uncompressed and stored in the temporary files folder for the user that runs the process. And to make it worse, ALL of the files are uncompressed before the process begins. This means if you are migrating 10GB of data, you'll need 10GB of free space in the user's home directory, and 10GB of free space in the content deployment folder! So pushing 10GB of data, requires 20GB of space plus the amount of space needed for the manifest XML files which can be quite large. The uncompressed location of the files is: c:\users\\AppData\Local\Temp\ This progess happens quickly and it will clean up the files after it fails. You may look at your computer and see many GB of free space. What you didn't see was the drive filling up and then failing, then deleting the files. So, when setting up content deployment. Figure out how much data you will be pushing with a full content deployment (and plan for future expansion), then add a few more GB for manfest XML(this is probably overkill), then double it. It's not always easy to expand the C: drive later, and moving the temp directory isn't fun. Just make your C: drive REALLY big. Don't forget to check back in every now and then and clean up the files that are sometimes left from a failed deployment. I seen to remember a way to set the max CAB size in 2007, but I don't see it in 2010. Keep in mind that it's actually the minimum cab size, not max. It will keep adding files until the cab reaches the max. If it's one byte below the max, and the next file is 200MB, it will end up with a cab that is 200MB+max-1 byte. This can be confusing when IIS rejects the file because it is too large.

Fixing missing Hidden and Sealed attributes in Site columns

After a recent upgrade to sharepoint 2010, we were no longer able to create new site templates. Creating a template would fail with an error:

Unexpected System.InvalidOperationException: Error copying temporary solution file to solutions gallery: _catalogs/solutions/TestProduction.wsp at Microsoft.SharePoint.SPSolutionExporter.ExportWebToGallery

Digging a little deeper into the ULS logs, I was able to find a better explaination of the error:

SharePoint Foundation General 9fjj Monitorable SPSolutionExporter: Microsoft.SharePoint.SPException: Feature definition with Id 10a563f6-0afb-4d38-9ddc-241c61694ac6 failed validation,
file 'TestProductionListInstances\ElementsFields.xml', line 39, character 167:
The 'Hidden' attribute is invalid - The value '' is invalid according to its datatype 'http://schemas.microsoft.com/sharepoint/:TRUEFALSE' - The Enumeration constraint failed. at Microsoft.SharePoint.Administration.SPSolutionPackage.SolutionFile.FeatureXmlValidationCallBack

Something was wrong with our columns. The "Hidden" parameter was blank instead of true or false. Looking into the database confirmed the error. The xml schema of the column definition had : Hidden="" instead of Hidden="FALSE". I also noticed that "Sealed" was also blank, and would cause the same error when trying to save the site as a template. The error describes the elementfields.xml file. This file is part of the site template WSP file. To view the file, go to the solutions gallery and download the Site template WSP that gets created. Change the extension to a CAB and open it. You will find the ElementFields.xml file and can use the line and character from the ULS log error to find the field that causes the problem. In my case, it was about 40 differnet fields.

These were errors on the site columns, many but not all of them being custom columns we created. Sealed and Hidden are not things that are easily changed through the UI, so I had to use Powershell to update the columns. I probably could have used SP Manager too, but since there were 40+ columns duplicated over a few different site collections, it was easier to use powershell.

Updating the Sealed column is fairly easiy since it is a property on a column that can be set directly. When trying to change the 'Hidden' property, I would get the following error: PS C:\> $f = $web.Fields["Meeting Location"]
PS C:\> $f.Hidden = $false
Exception setting "Hidden": "Cannot change Hidden attribute for this field"
At line:1 char:4
+ $f. <<<< Hidden = $false
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : PropertyAssignmentException


So, if I can't change the field, how do I change it?. The answer is to change the xml schema. sharepoint will let you change the schema directly and skip around whatever check it has in place.


$field = $Web.fields[$i]
[XML]$schema = $field.schemaxml
$field.schemaxml = $schema.Innerxml
$field.update()


The next problem, how do you know what fields need to be changed? If you just check for $field.hidden -eq "" or something like that it will not work. It will tell you that the field is "FALSE" even when it is blank! So, we have to look at the xml:
[XML]$schema = $field.schemaxml
if ( $schema.InnerXML.Contains("Hidden=""""")){
do something}

All together the script looks like this:

$site = Get-SPSite "http://YourSiteCollectionURL"
$Web =$site.RootWeb

for($i = $Web.Fields.count -1; $i -ge 0; $i--)
{
$field = $Web.fields[$i]
[XML]$schema = $field.schemaxml
if ( $schema.InnerXML.Contains("Hidden=""""")){
$schema.field.Hidden = "FALSE"
$field.schemaxml = $schema.Innerxml
Write-Host $field "HIDDEN: " $schema.field.Hidden
$field.update()
}
if ( $schema.InnerXML.Contains("Sealed=""""")){
Write-Host $field "Sealed: " $schema.field.Sealed
$field.Sealed = "FALSE"
$field.update()
}
if ( $schema.InnerXML.Contains("Required=""""")){
Write-Host $field "Required: " $schema.field.Sealed
$field.Required = "FALSE"
$field.update()
}
}




This script will set all the hidden and sealed fields to FALSE for the site columns that have the field as blank. Another way to do this is to modifiy the database directly, but that isn't supported and you can easily break things.

I hope this can help someone else who runs into the same problem. I wasn't able to find a good solution on the web. MSFT support(wipro) did help, but it took them almost 2 months, and 3 different support techs to come up with a script that I then had to extensivly modify to get to actually solve the problem.

Sum Grouped Calculated Columns in SharePoint

After finally migrating to sharepoint 2010 from sharepoint 2007, i realized some of our custom Data view webparts were no longer working. They had been customized to add up the total hours and display it for each group.
The view had items grouped by user and a total number of hours for that user. This the hours was computed by a check in time and check out time. Sharepoint does not provide a way to calculate a total for calculated fields.
Most solutions tell you to use a dataview webpart to do this. Our old solution worked that way, but it was a hassle to add new views and re-use the code in other projects.

My solution was to throw out the old code and re-build it using out of the box, easy to create views and some javascript/jQuery magic.

The following code will find items by group, and compute a total for that group.
The javascript has two variables:
COLUMNTITLE: The name of the column  you would like to find the total of all the rows
TEXTHEADER: The text you would like to show in front of the total. This cannot be blank, and should be some unique text. It is also used to tell if the total has already been calculated and displayed.

This code will automatically display totals for groups that have been expanded. If the group has not been expanded, it cannot show the total because SharePoint loads the data using ajax. When the group is expanded, the total will be calculated and displayed.

This works on a SharePoint 2007 site upgraded to 2010, but still using the 2007 UI. I haven't tested this in other scenarios. The idea should be the same, some of the css tags might change. YMMV.


var TEXTHEADER = "Total Hours: ";// text to put in front of the total
var COLUMNTITLE = "TotalBillableTime";// which column of data do you want to sum up?
var COLUMNNUMBER =0; // placeholder variable.

//alert("found:"+FindColumnNumber(COLUMNTITLE));

$(function() {
    // attach to clicks to load the data when the user clicks to expand any closed sections
    COLUMNNUMBER = FindColumnNumber(COLUMNTITLE);
    AttachClicksToComputeTotalTime();
    // find the loaded blocks of data and populate the header
    // this can take some time, so we're calling it after a delay.
    // better method would be to attach to the ajax complete call of sharepoint if there is one.

    setTimeout(function(){FindLoadedBlocks();},1000);
   
});

function     FindLoadedBlocks(){
        // search for all tbody's that have tbod in the id and have isLoaded = true
        // compute the totals for these
         $('tbody[id^="tbod"][isLoaded="true"]').each(function($this){
            ComputeBlock(this.id);
          });   
}

function AttachClicksToComputeTotalTime(){
    // find all the header links that the user clicks to expand the data.
    // add an onClick event to compute and display the totals
    $("td.ms-gb2 a").click(function($this){
        FindBlock(this);
    });
}

function FindBlock(Headerblock){
    // given a header, find it's child tbody and wait until it is fully loaded to compute its data
        var parentSectionID = Headerblock.parentNode.parentNode.parentNode.id;
        var childSection = parentSectionID.replace("titl","tbod")+"_";
        var kidSection = $("#"+childSection);
        if($(kidSection).text().length<100 br="">            setTimeout(function(){FindBlock(Headerblock);},50);
            return;
       
        }    else{
        // it's done, let's load it.
            ComputeBlock(childSection);           
        }

}


function ComputeBlock(blockName){
    // for a block of data rows, compute the total and insert it into the header
        var parentSectionID = blockName.replace("tbod","titl").replace("__","_");       

    // when groups are expanded by default, a second tobdy is created
    // check if this tobody has data, if not, go to the next
         if($('#'+blockName).html().length<100 br="">             var block = $('#'+blockName).next();

        }else{
            var block = $('#'+blockName);
        }
       
        var rowSum = 0;
        childRows = $(block).find("tr").each(function(){ 

            // find the 6th column
            var x = $(this).find(":nth-child("+COLUMNNUMBER +")");
            var rowVal = parseFloat($(x).text().trim());
            if(!isNaN(rowVal)){
                rowSum+= rowVal;
            }

        });
        rowSum = rowSum.toFixed(2); // trim it to 2 decimal places

        // find the parent section
        var ParentSection =$("#"+parentSectionID+" tr td");
    // check that the parent section doesnt' already have the totals   
        if($(ParentSection).text().indexOf(TEXTHEADER)>0){

        }
        else{

            $(ParentSection).append("   "+TEXTHEADER +"   "+rowSum+"");
        }

}

function FindColumnNumber(ColumnTitle){
// find the column by name and figure out where it is.
    var pos = 0;
    var foundVal = 0;

    $(".ms-vh2").each(function(){
         pos++;
         if($(this).text() == ColumnTitle){
                 foundVal = pos;
         }
    });
    return foundVal;
}





each group section will look something like this: