Dianoga 3.0 is released

I’m happy to announce the release of Dianoga 3.0. This release brings significant improvements compared to previous releases, and is recommended for existing users.

What’s Dianoga?

Dianoga is a tool that optimizes images uploaded to the Sitecore media library to save bandwidth and improve page loading times as a result.

It's also the monster that lives in the Star Wars trash compactor. What?

Dianoga ensures that your site is always serving fully optimised media library images even if you are using Sitecore’s dynamic resizing features (for example with Adaptive Images). Even if you have already optimized every image uploaded into the media library, after Sitecore performs image processing the result is not optimized (an unfortunate limitation of other image optimization libraries for Sitecore is that they only apply to the original image).

Dianoga is also great for situations where content editors may not be image editing experts and upload images that contain gobs of EXIF data and other nonessential metadata - these are removed automatically before being shown to visitors. All of the optimizations done are lossless (no quantizing, etc) so you lose no image quality.

What’s new in 3.0?

SVG support

With the rise of SVG as a common format for media library media, it became apparent Dianoga needed to support optimizing SVG. Dianoga 3 includes SVG optimization!

SVG media is automatically:

  • Optimized for size (the SVG is processed using SVGO)
  • Gzipped before going in the media cache, and served using the cached gzipped version (reduces size over the wire)
  • Apropos configuration to enable SVG support in media library on Sitecore < 8.1 is enabled by default (thanks to Kamruz Jaman and Anders Laub for the blogs e.g. this and this)

Optimization strategies

Dianoga 1.x used a synchronous optimization technique that resulted in a slower initial image load time, but always served optimized images. Dianoga 2.x instead relied on an asynchronous technique where the first media served would be unoptimized and subsequent hits were optimized, but the response time was never impacted.

It became apparent that both of these strategies have their place, for example you need to be synchronous when uploading to a CDN, and so Dianoga 3 supports optimization strategies that let you choose when to optimize. If you don’t like any of the strategies or want to optimize only when sending media to a CDN programmatically, you can also invoke the optimization pipeline directly to optimize precisely when you need to (the MediaOptimizer class is what you’re after here).

Pipeline-based optimization

Dianoga is now powered by Sitecore pipelines, providing simple and flexible extension options. The root <dianogaOptimize> pipeline runs for all optimizations, and spins off into <dianogaOptimizeJpeg> and other similar sub-pipelines for individual file types.

Optimizer chaining

With Dianoga 2, you could not simply apply more than one optimizer to a file format. For example, you might wish to quantize a PNG (which is lossy), and then also optimize its encoding after quantization to further reduce file size. This is very simple with Dianoga 3 - now each optimizer is just a step in a pipeline. Add or remove optimizers as you wish.

Media path exclusion

Got a folder of huge photos you don’t want optimized because they should be kept pristine for downloads? Have another reason you want to not optimize specific parts of the media library? Great - Dianoga 3 now supports that.

Because optimization is pipeline based, this just takes the form of a processor that aborts optimization when the input is to be ignored. See Dianoga.ExcludePaths.config.example which ships with the NuGet package for how to use this.

Framework version requirements

Dianoga 3 requires .NET 4.5.2 or later to be the target framework of the project it is referenced in. This provides compatibility with Sitecore 8.2. Dianoga 3 requires Sitecore 7.x or later.

Bugs and fixes

  • Asynchronous optimization has been significantly simplified and reliability improved compared to Dianoga 2. File in use issues should be completely eliminated.
  • Unit tests have been added to the codebase (I blame Dan Solovay)
  • Optimization tools are moved to App_Data so that IIS will never consider serving their files (as opposed to /Dianoga Tools in 2.x and earlier)
  • Optimization tools have been updated:
    • libjpeg is now mozjpeg, which results in better optimization for the web specifically
    • SVGO has been added to optimize SVGs
    • PNG optimization remains unchanged using PNGOptimizer, other than now being chainable with the PNGQuant lossy quantizer.

Installing

Dianoga is available from NuGet. You must be using packages.config (NuGet 2.x style) package management for Dianoga, because it installs content items to the project. If you wish to use project.json (NuGet 3.x style), you must manually install the tools and configuration files because of limitations in NuGet 3.x.

Upgrading

Easy. Make sure your host project is targeting .NET 4.5.2 (or later - I use it with 4.6.x), and then upgrade your NuGet package.

Have fun!

Become a Sitecore admin without a login

Suppose someone sends you a Sitecore solution to review, and they forgot to send you a username and password. You could ask for one, or you could just make yourself an account with this handy trick that I call “The Shiv.” This is the entirety of the trick:

  1. Create a Shiv.aspx file somewhere under the webroot. It can be named anything.
  2. Paste the following code in it

    <%@ Page Language="C#" AutoEventWireup="true" %>
    <%@ Import Namespace="Sitecore.Security.Authentication" %>
    
    <%
        AuthenticationManager.Login("sitecore\\admin", false, true);
        Response.Redirect("/sitecore/shell");
    %>
    
  3. Hit the page in a browser

  4. Boom, you’re an administrator
  5. IMPORTANT: Delete the file. For obvious reasons.

Handy, eh?

You can do a very similar thing using the “Login as administrator” option in SIM, however I often find myself in environments without SIM and this code works anywhere.

bang

This code is also a good security reminder: if someone malicious can upload an arbitrary file somewhere in your webroot that is then executed, they can upload this shiv-file and your security is gone. It doesn’t matter if you have encrypted 64-character database passwords, they’re in. It doesn’t matter if you’ve locked down TLS and imposed SAML logins, they’re in. Game over. So secure your filesystem and be awfully wary of accepting users’ uploads anywhere on disk.

Nugetify your Sitecore References

Recently, in a fit of brilliance, Sitecore released a public NuGet feed that you can use to reference Sitecore assemblies from your projects. While some people have been doing this with home grown packages for years, it’s nice to have a stable, official source to get your references from.

If you’re not familiar with how this works, Jeremy Davis wrote a great post about the details of using the official feed.

Okay so what are you on about?

If you’ve got a project of reasonable size, especially one using Helix, you probably have a bucketload of references to Sitecore assemblies. Manually converting all these references to packages is a bit of a tedious process, and you know what we do to tedious processes around here.

nugetify.jpg

That’s right, we script them.

How?

Migrating your direct Sitecore references to NuGet is a quite simple process with this script. For obvious reasons, use source control so you have a fallback in case the script doesn’t work for your particular setup. It worked for me on several projects, but just in case :)

1. Install NuGet.config

Copy this to the root of your solution:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!--
  Used to specify the default Sources for list, install and update.
  -->
  <packageSources>
    <add key="Official Sitecore" value="https://sitecore.myget.org/F/sc-packages/api/v3/index.json" />
  </packageSources>

  <activePackageSource>
    <!-- this tells that all of them are active -->
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

This will add the official Sitecore NuGet package feed to your solution. Unlike adding it via Visual Studio, this will also apply for CI and MSBuild-executed builds.

2. Copy Nugetify.ps1

Copy the following PowerShell script to the root of your solution. Open it in a text editor and set the correct $SitecoreVersion and $FrameworkVersion for your solution. The NuGet feed has packages for Sitecore 7.0-8.2.

# Script to convert all sitecore assembly references to Sitecore Public NuGet feed
# Run from root solution folder (where your packages folder is)
# execute this script with powershell
Param
(
    $sitecoreVersion = '8.2.160729', # NuGet package version to convert to. Format is major.minor.releasedate.
    $frameworkVersion = 'net452' # for 8.2: net452. For 7.0-8.1: net45
)

$ScriptPath = $MyInvocation.MyCommand.Path
$ScriptDir  = Split-Path -Parent $ScriptPath
$MsbNSString = 'http://schemas.microsoft.com/developer/msbuild/2003'
$MsbNS = @{msb = $MsbNSString}
$PackagesConfigFileName = 'packages.config'

#Create project.json from packages.config
Write-Host 'Scanning for projects to update...' -ForegroundColor Green
Get-ChildItem -path '.' -Recurse -Include $PackagesConfigFileName |
    ForEach {
        $PackageFilePath = $_.FullName
        $PackageFileDir = $_.Directory
        Write-Host "Processing $PackageFilePath" -ForegroundColor Green

        # Find existing csproj to match direct references
        $csproj = Resolve-Path "$($_.Directory)\*.csproj"
        $proj = [xml] (Get-Content $csproj)

        # Find existing Sitecore references and NuGet-ify them
        Write-Host "Checking for non-NuGet Sitecore references in $csproj"
        $xpath = "//msb:Reference/msb:HintPath[not(contains(.,'packages\'))]"

        $changedProj = $false
        $sitecoreNuGetPackages = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {
            $node = $_.Node.ParentNode

            $referenceName = $node.Attributes['Include'].Value.Split(',')[0]

              # Filter non-NuGet references to transform into NuGet packages
            if($referenceName.StartsWith("Sitecore") `
                -and -not $referenceName.StartsWith('Sitecore.Modules') `
                -and -not $referenceName.Contains('WFFM') `
                -and -not $referenceName.StartsWith('Sitecore.Forms') `
                -and -not $referenceName.StartsWith('Sitecore.Foundation') `
                -and -not $referenceName.StartsWith('Sitecore.Feature') `
                -and -not ($referenceName.StartsWith('Sitecore') -and $referenceName.EndsWith('Website'))) {

                $changedProj = $true

                Write-Host "NuGet-ifying assembly reference $referenceName"

                # set hintPath to package path
                Push-Location -Path $PackageFileDir
                $hintPathRoot = Resolve-Path "$ScriptDir\packages" -Relative
                Pop-Location

                $hintPath = "$hintPathRoot\$referenceName.NoReferences.$sitecoreVersion\lib\$frameworkVersion\$referenceName.dll"

                $existingHintPath = $node['HintPath', $MsbNSString]
                if($existingHintPath -eq $null) {
                    $hint = $proj.CreateElement("HintPath", $MsbNSString)
                    $hint.InnerXml = $hintPath
                    $foo = $node.AppendChild($hint)
                } else {
                    $existingHintPath.InnerXml = $hintPath
                }

                "$referenceName.NoReferences"
            }
        })

        if($changedProj) {
            Write-Host "Saving NuGet-ified references to csproj" -ForegroundColor Yellow
            $proj.Save($csproj)
        } else {
            Write-Host "Found no references to change."
        }

        # Add packages to packages.config
        $packageXml = [xml] (Get-Content $PackageFilePath)

        $sitecoreNuGetPackages | % {
            $packageNode = $packageXml.CreateElement('package');
            $packageNode.SetAttribute('id', $_)
            $packageNode.SetAttribute('version', $sitecoreVersion)
            $packageNode.SetAttribute('targetFramework', $frameworkVersion)

            $foo = $packageXml.DocumentElement.AppendChild($packageNode)
        }

        Write-Host "Updating packages.config with new packages" -ForegroundColor Yellow
        $packageXml.Save($PackageFilePath)
    }

3. Run Nugetify.ps1

Open a PowerShell and execute Nugetify.ps1.

That’s it. Open Visual Studio and verify that everything was converted correctly, and you should be good to go.

What about NuGet 3 + project.json?

Another option is to use the NuGet 3.x style package management which is integrated into a project.json file that lives next to your csproj files. NuGet 3’s major advantage is that the project.json both references packages and adds them to your project references. So adding packages does not result in alterations to the csproj file, and upgrading packages is as simple as changing a json file.

Sounds idyllic, right? Well there’s one huge downside. Microsoft, in their infinite wisdom, decided that it was not necessary to support content files being deployed into the project the package is installed into. For Sitecore projects, that means packages that come with config files, such as Unicorn, Synthesis, and Glass Mapper, will not install those files into projects using project.json. The content-containing packages can still be used, but then it becomes your task to reverse engineer the content they install, add that to your project, and handle upgrades of those content files when the package is upgraded.

For the moment, I wouldn’t use project.json, but I hope it becomes more tenable in the future. But if you want to use it, I have a script for that too - a script that both nugetifies your Sitecore references and converts all of your packages in packages.config to project.json. This script is based on this one.

# Script to generate project.json for all packages.config file in the solution.
# This script will also migrate non-NuGet Sitecore package references to Sitecore Public NuGet feed

# Run from root solution folder
# execute this script with powershell

# TargetFramework: Use the .NET framework version your projects are targeting, which may NOT be the version Sitecore is built aginst
# SitecoreVersion: NuGet version you want to convert to for local sitecore assembly references
Param
(
      [string] $TargetFramework = "net452",
    $sitecoreVersion = '8.2.160729'
)

# Filter existing installed NuGet packages to transform versions and such
function Filter-Packages {
    $input | % {
        $package = $_.Node.id
        $version = $_.Node.version

        # Translate nuget package generator 8.2 package version to official
        if($version -eq "8.2.0.160729") {
            $version = "8.2.160729"
        }

        # Translate 3rd party refs from old SC versions to target public versions
        if($package.Equals('Microsoft.Extensions.DependencyInjection.Abstractions')) {
            $version = '1.0.0'
        }

        if($package.Equals('MongoDB.Driver')) {
            $package = 'mongocsharpdriver'
            $version = '1.11.0'
        }

        # Blacklist these older sitecore nuget generator metapackages, modules, and nonexistant packages
        if($package.EndsWith("-Core") `
            -or $package.EndsWith("-CoreGroup") `
            -or $package.Equals('MongoDB.Bson') `
            -or $package.Equals("Telerik.Web.UI")) {

            return # skip loop
        }

        # Packages that started with Sitecore before should now get NoReferences for sanity (note: you may need to exclude sitecore modules whose name begins in Sitecore here)
        if($package.StartsWith('Sitecore') `
            -and -not $package.StartsWith('Sitecore.FakeDb') `
            -and -not $package.EndsWith('PatchableIgnoreList')) {

            $package = "$package.NoReferences";
        }

        $_.Node.id = $package
        $_.Node.version = $version

        $_
    }
}

$ScriptPath = $MyInvocation.MyCommand.Path
$ScriptDir  = Split-Path -Parent $ScriptPath
$MsbNS = @{msb = 'http://schemas.microsoft.com/developer/msbuild/2003'}
$PackagesConfigFileName = 'packages.config'

#Create project.json from packages.config
Get-ChildItem -path '.' -Recurse -Include $PackagesConfigFileName |
    ForEach {
        $PackageFilePath = $_.FullName
        $ProjectFilePath = $_.Directory.FullName + '\project.json'
        Write-Host "Processing $PackageFilePath"

        # Find existing csproj to match direct references
        $csproj = Resolve-Path "$($_.Directory)\*.csproj"
        $proj = [xml] (Get-Content $csproj)

        # Find existing Sitecore references and NuGet-ify them
        Write-Host "Checking for non-NuGet Sitecore references in $csproj"
        $xpath = "//msb:Reference/msb:HintPath[not(contains(.,'packages\'))]"

        $changedProj = $false
        $sitecoreNuGetPackages = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {
            $node = $_.Node.ParentNode

            $referenceName = $node.Attributes['Include'].Value.Split(',')[0]

      # Filter non-NuGet references to transform into NuGet packages
            if($referenceName.StartsWith("Sitecore") `
                -and -not $referenceName.StartsWith('Sitecore.Modules') `
        -and -not $referenceName.Contains('WFFM') `
        -and -not $referenceName.StartsWith('Sitecore.Forms') `
                -and -not $referenceName.StartsWith('Sitecore.Foundation') `
                -and -not $referenceName.StartsWith('Sitecore.Feature') `
                -and -not ($referenceName.StartsWith('Sitecore') -and $referenceName.EndsWith('Website'))) {

                $changedProj = $true

                Write-Host "NuGet-ifying assembly reference $referenceName"

                # remove old reference we're NuGet-ing
                [void]$node.ParentNode.RemoveChild($node);

                "$referenceName.NoReferences"
            }
        })

        if($changedProj) {
            Write-Host "Saving NuGet-ified references in $csproj"
            $proj.Save($csproj)
        }

        # Generate project.json
        $file = '{
  "dependencies": {
'

$packages = (Select-xml -xpath '//package' -Path $PackageFilePath | Filter-Packages | % { "    ""{0}"": ""{1}""" -f $_.Node.id,$_.Node.version }) -join ",`r`n"

$file += $packages;

$sitecorePackages = (($sitecoreNuGetPackages | % { "    ""{0}"": ""{1}""" -f $_, $sitecoreVersion }) -join ",`r`n")

# separate the json elements if both converted and sitecore packages exist
if($packages.Length -gt 0) {
    $file += ",`r`n"
} else {
    $file += "`r`n"
}

$file += $sitecorePackages

$file += '
  },
  "frameworks": {
    "' + $TargetFramework + '": {}
  },
  "runtimes":  {
      "win-anycpu": {},
      "win": {}
  }
}'

$file | Out-File $ProjectFilePath

    Remove-Item $PackageFilePath
}

Get-ChildItem -path '.' -Recurse -Include '*.csproj' | ForEach {
    $CsProjFilePath = $_.FullName
    $ProjectFilePath = $_.Directory.FullName + '\project.json'

    Write-Host $csProjFilePath

    $proj = [xml] (Get-Content $CsProjFilePath)

    #Remove all references to ..packages files
    $xpath = "//msb:Reference/msb:HintPath[contains(.,'packages\')]"
    $nodes = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {$_.Node})
    if (!$nodes) { Write-Verbose "RemoveElement: XPath $XPath not found" }
    Write-Output 'Reference Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $referenceNode = $node.ParentNode
        $itemGroupNode = $referenceNode.ParentNode
        [void]$itemGroupNode.RemoveChild($referenceNode)
    }
    [System.XML.XMLElement] $itemGroupNoneNode = $null
    #Find itemgroup with None Elements, if not found add.
    $itemGroupNoneNodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output '$itemGroupNoneNode found: ' $itemGroupNoneNodes.Count
    if($itemGroupNoneNodes.Count -eq 0){
        # create itemgroup element for None nodes.
        Write-Output 'Adding ItemGroup for None Nodes'
        $itemGroupNoneNode =  $proj.CreateElement('ItemGroup',$proj.DocumentElement.NamespaceURI)
        $itemGroupNodes = @(Select-Xml -xpath "//msb:ItemGroup" $proj -Namespace $MsbNS | foreach {$_.Node})
        #$itemGroupNodes.Count
        [void]$proj.DocumentElement.InsertAfter($itemGroupNoneNode,$itemGroupNodes[$itemGroupNodes.Count-1])

    }else{
        $itemGroupNoneNode = $itemGroupNoneNodes[0].ParentNode
    }

    #Remove packages.config from ItemGroup
    $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None[@Include='packages.config']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'packages.config Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Remove packages.config from ItemGroup (if it was set to content)
    $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:Content[@Include='packages.config']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'packages.config Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Remove build target EnsureNuGetPackageBuildImports from csproj
    $nodes = @(Select-Xml -xpath "//msb:Target[@Name='EnsureNuGetPackageBuildImports']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'EnsureNuGetPackageBuildImports target found: ' $nodes.CountAd
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Add project.json to itemGroup
    if( Test-Path $ProjectFilePath){
        $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None[@Include='project.json']" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $projectJsonNoneNode = $proj.CreateElement("None", $proj.DocumentElement.NamespaceURI)
            $projectJsonNoneNode.SetAttribute("Include","project.json")
            [void]$itemGroupNoneNode.AppendChild($projectJsonNoneNode)
            Write-Output 'Adding None node for project.json'
        }
    }

    #add PropertyGroup nodes targetFrameworkProfile, CopyNuGetImplementations, PlatformTarget
    # Find the TargetFrameworkVersion to be used to find the parent PropertyGroup node
    $xpath = "//msb:PropertyGroup/msb:TargetFrameworkVersion"
    $nodes = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {$_.Node})
    if ($nodes.Count -gt 0) {
        [System.XML.XMLElement] $node = $nodes[0]
        $propertyGroupNode = $node.ParentNode
        $nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:TargetFrameworkProfile" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $node = $proj.CreateElement("TargetFrameworkProfile", $proj.DocumentElement.NamespaceURI)
            [void]$propertyGroupNode.AppendChild($node)
            Write-Output 'Adding TargetFrameworkProfile node for PropertyGroup'
        }
        #$nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:CopyNuGetImplementations" $proj -Namespace $MsbNS | foreach {$_.Node})
        #if($nodes.Count -eq 0){
        #    $node = $proj.CreateElement("CopyNuGetImplementations", $proj.DocumentElement.NamespaceURI)
        #    $textnode = $proj.CreateTextNode("true")
        #    $node.AppendChild($textnode)
        #    [void]$propertyGroupNode.AppendChild($node)
        #    Write-Output 'Adding CopyNuGetImplementations node for PropertyGroup'
        #}
        $nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:PlatformTarget[not(@*)]" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $node = $proj.CreateElement("PlatformTarget", $proj.DocumentElement.NamespaceURI)
            $textnode = $proj.CreateTextNode("AnyCPU")
            $foo = $node.AppendChild($textnode)
            [void]$propertyGroupNode.AppendChild($node)
            Write-Output 'Adding PlatformTarget node for PropertyGroup'
        }
    }

    # replace ToolsVersion with 14.0
    $attibutes = Select-Xml -xpath "//@ToolsVersion" $proj -Namespace $MsbNS
    foreach ($attribute in $attibutes){

        $attribute.Node.value = "14.0"
        Write-Output 'Setting ToolsVersion to 14.0'
    }

    $proj.Save($CsProjFilePath)
 }

Till next time, happy NuGetting!

Precompiled Views with Sitecore 8.2

A while ago I wrote a post about speeding up your views with precompilation.

After writing that post I learned about RazorGenerator which Chris van de Steeg blogged about. RazorGenerator is better in just about every way compared to aspnet_compiler.exe. Whereas the aspnet_compiler just warms the compiler cache for the current machine, RazorGenerator builds the generated classes for your Razor files directly into your assemblies.

With RazorGenerator you can compile once and deploy everywhere. Getting started with it is almost stupidly simple. Seriously, I described the whole process in one tweet: Install the RazorGenerator.MsBuild NuGet package. That’s it. So what does that get you?

  • Compile-time view syntax checking. Ever deployed a Razor file that broke at runtime after a “successful” build? Well now your builds will fail if your Razor syntax is invalid. Hoorah!
  • Your Razor views are compiled into your output assembly (look at it in DotPeek and see the ASP namespace)
  • It’s fast. You probably won’t even notice the build time difference.

But that’s not quite all

Right, so how do you get your precompiled views to actually get loaded? ASP.NET MVC doesn’t understand how to resolve classes instead of files out of the box. With Sitecore 8.1 and earlier, you could use the RazorGenerator.Mvc NuGet package and wire up its PrecompiledViewEngine to accomplish this.

Sitecore 8.2 modernizes a lot of the MVC implementation under the covers, and part of that is that the Sitecore infrastructure now supports precompiled views natively! (Wondering why the experience editor starts up a lot faster in 8.2? That’s why.) It’s really easy to use, in fact: all you have to do is tell Sitecore which assemblies to look for precompiled views in and it does the rest. Behold:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <mvc>
      <precompilation>
        <assemblies>
          <assemblyIdentity name="My.Project" />
          <assemblyIdentity name="My.OtherProject" />
        </assemblies>
      </precompilation>
    </mvc>
  </sitecore>
</configuration>

CAVEAT: The Sitecore precompiler is greedy by default. When it is on, if a class exists for your view, the physical file for the view is not checked. You can even delete it. This means that you cannot tweak the cshtml file and have updates be reflected. You can enable timestamp checking if you must by changing this setting:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!--  MVC: Flag for controlling whether Razor View Engine will look at physical view last modified dates 
        when determining to use a view file from a pre-compiled assembly or from the file system.
        Default: "false"
        -->
        <setting name="Mvc.UsePhysicalViewsIfNewer" value="false" />
      </settings>
  </sitecore>
</configuration>

Because there are (minor) performance implications of doing the timestamp checks, I would advise enabling this setting for development and disabling the setting for production. You should never, ever be changing files on production after a deployment anyway.

Recap

If you’re on Sitecore 8.2, you can get view performance gains by:

  • Installing RazorGenerator.MsBuild on each Visual Studio project that contains Razor views
  • Registering each of those assemblies with the Sitecore precompiled view engine so that it uses your precompiled classes for view rendering

Big thanks to Kern for setting up the MVC expert panel, without which this feature might well not exist. And we might still have 1:40 startup times after a compile. :)

Sitecore hangs on startup when using MongoDB

I ran across a tricky issue today where a Sitecore content delivery cluster would simply hang on startup. No errors in the logs. Nothing in the windows event logs. The content editing server worked just fine.

Mystifying, right? So it came down to process of elimination: I disabled analytics and removed all Mongo connection strings and the site worked. The obvious answer here would be “it’s a firewall issue.”

It wasn’t. mongo.exe could connect to the replica set just fine. The connection was using SSL, so I checked the server certificate. It was valid and trusted.

Eventually it came down to a few important facts:

  • We were using SSL.
  • Because we weren’t using client certificates, mongo.exe had to use --sslAllowInvalidCertificates, which meant that it was not verifying the server certificate.
  • The Mongo C# driver does validate the server certificate.

But you said the server certificate was totally valid, right? Right - and it was valid.

Except for the sneaky fact that the certificate had a Certificate Revocation List (CRL) URL embedded in it. The Mongo C# driver defaults to checking the CRL to make sure the server certificate is not revoked.

Normally that’s a good thing, but in this case the CRL was behind the firewall, and the delivery servers were not. So the CRL could not be checked, and the server certificate was treated as invalid due to that.

Why this results in straight up hanging the site instead of an error I’m not sure. But that leads to how it got fixed.

Extending the Mongo Client Settings

The most obvious fix would be to add something to the connection string to disable the CRL check, as it would be undesirable to expose the PKI server that issued the certificate to the DMZ where the delivery servers live. Unfortunately, there is no option to do that in the Mongo connection string format.

Fortunately, There’s A Micro-Pipeline For That™. Sitecore exposes the updateMongoDriverSettings pipeline, which is empty by default. This pipeline allows you to fool with the MongoClientSettings object and alter configurations that are not available in the connection string.

Here’s an example patch to add a processor to said pipeline:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <pipelines>
            <updateMongoDriverSettings>
                <processor type="Me.Pipelines.UpdateMongoDriverSettings.DisableCrlChecks, Me"/>
            </updateMongoDriverSettings>
        </pipelines>
    </sitecore>
</configuration>

And the processor implementation that alters the MongoClientSettings to disable CRL checks:

using MongoDB.Driver;
using Sitecore.Analytics.Pipelines.UpdateMongoDriverSettings;

namespace Me.Pipelines.UpdateMongoDriverSettings
{
    public class DisableCrlChecks : UpdateMongoDriverSettingsProcessor
    {
        public override void UpdateSettings(UpdateMongoDriverSettingsArgs args)
        {
            args.MongoSettings.SslSettings = new SslSettings { CheckCertificateRevocation = false };
        }
    }
}

This technique could also be used to diagnose other SSL issues that might involve invalid certificates, because you can also alter the certificate validity handling to get diagnostic output.

Caveat: Sorry, not for you Mongo Session Provider

vader.jpg

Unfortunately if you’re using the MongoDB Session State provider, it does not respect this pipeline. In fact, it makes itself highly extensible, by hiding the place to put your options in a private static method! Called by internal methods!

private static MongoCollection GetCollection(string connectionString, string databaseName, string collectionName)
{
    return (MongoCollection) new MongoClient(new MongoUrl(connectionString)).GetServer().GetDatabase(databaseName).GetCollection(collectionName);
}

So you’re hosed if you’re using Mongo sessions.

Dependency Injection in Sitecore 8.2

Sitecore 8.2, hot off the presses yesterday, includes built in dependency injection. If you’ve been following the internal architecture of Sitecore for very long, you’ve probably realized that a lot of it is uglier than it needs to be - and less extensible - thanks to the lack of system-wide DI support. It’s kind of a big deal. Even if Nat Mann hates conforming containers ;)

The Sitecore IoC container is based on Microsoft’s Dependency Injection library that was written for .NET Core. It is not the world’s most flexible or performant container (nor was it designed to be), but it works and is a decent choice. If you wish you can change the underlying IoC library being used, but we aren’t going to cover that today.

Sitecore’s built in DI offers two major advantages over the third party dependency injection that most advanced Sitecore teams have been using for a long time:

  • Sitecore itself is using it (and thus you can extend Sitecore itself by replacing dependencies)
  • You can natively dependency inject into pipeline processors (which you could sort of already do)

Controller Injection

As with most ASP.NET DI implementations, most dependency resolution will take place in controllers. If you’ve already been using constructor injection with Sitecore, you may have to change absolutely nothing:

public class FooController : Controller
{
    private readonly IDependency _dependency;

    public Foo(IDependency dependency) 
    {
        _dependency = dependency;
    }

    public ActionResult Index() 
    {
        return View(_dependency.DoStuff());
    }
}

Be aware that the built in IoC container has two major limitations when injecting controllers:

  • You may only have one public constructor for your controller - or any other registered dependency. This is a good thing, as multiple public constructors are a DI antipattern anyway.
  • The controller class must be registered with the container to be resolvable (e.g. you must register it such as container.AddTransient<FooController>()).

The latter limitation we can do something nice about: read on and we’ll get to that when we talk about registering dependencies :)

Pipeline Injection

You may also inject dependencies into pipeline processors using the same constructor injection pattern as controllers. Processors are not injected by default, presumably for performance. As with controllers, processors may only have one public constructor. I suspect, but have not tried, that this trick would also work with commands.

To inject a processor, simply add resolve="true" to its registration. For example:

<processor type="Foo.BarProcessor, Foo" resolve="true" />

Web Forms Dependency Injection

grumpy-cat-no.jpg

(You can actually do Web Forms DI with Sitecore but I’m not going to tell you how. Quit using Web Forms.)

Service Locator

You can also resolve dependencies from the Sitecore container using the Service Locator antipattern. This is where you explicitly ask the container to give you an instance of an object. It’s an antipattern, and you should use it as a weapon of last resort, because it tightly couples your class to the IoC container and makes things difficult to test.

There are actually multiple ways you can use Service Locator:

// the MVC DependencyResolver can be used
DependencyResolver.Current.GetService<IService>();

// the Sitecore ServiceLocator can be used
using Sitecore.DependencyInjection;
ServiceLocator.ServiceProvider.GetService<IServiceCollection>();

Again don’t use these…unless you have no other choice.

Registering Dependencies

Of course an IoC container is useless if it has no registered dependencies to resolve! Sitecore’s container can be configured in multiple ways, all of which involve some level of XML. I heard you groan when you read that ;)

Keep in mind when wiring dependencies that the IoC container is not multitenant. Your dependencies are sharing the container with Sitecore’s - and if you have more than one site, potentially other sites as well. So don’t go expecting to have IFoo resolve to different implementations in different sites!

If you get confused and want to see a list of every dependency that is currently registered, along with its scope and type, there’s a page for that! Just visit /sitecore/admin/showservicesconfig.aspx and there you are. While you’re at it, check out the other handy tools in the admin pages too.

Configurators

A configurator is probably what you think of when you consider IoC configuration if you’ve been using any modern container library. It’s a C# class that conforms to an interface where you are given a container object, and expected to wire your dependencies to it. You can register as many configurators as you like in the <services> section.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <services>
            <configurator type="MyProject.MyConfiguratorClass, MyProject" />
        </services>
    </sitecore>
</configuration>

Here’s an example configurator implementation that registers a couple dependencies. As with most containers Transient and Singleton dependencies are available, and I believe Scoped as well, but I’m not sure what the exact behaviour of that is in this case.

using Microsoft.Extensions.DependencyInjection;
using Sitecore.DependencyInjection;

namespace MyProject
{
    public class MyConfiguratorClass : IServicesConfigurator
    {
        public void Configure(IServiceCollection serviceCollection)
        {
            serviceCollection.AddSingleton<IDependency, Dependency>();
            serviceCollection.AddSingleton<IService>(provider => new Service("withFactory"));
        }
    }
}

Note: You cannot use Sitecore Factory conventions when registering configurators, for example setting properties on the configurator with child elements. This is because the Factory also speaks DI now as a fallback, so it’d be like asking the container to resolve itself :)

Direct Registration

You can also register individual dependencies with XML, just like we did ten years ago! I wouldn’t suggest doing this as it is less expressive than a configurator, not type-checked by compilation, and probably marginally slower as well due to having to convert the string to the type for every dependency.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <services>
            <register 
                serviceType="Type.IName, Assembly" 
                implementationType="Type.Name, Assembly" 
                lifetime="Transient" />
        </services>
    </sitecore>
</configuration>

Automatic Controller Registration

If you’re actually reading this, you may have noticed that I mentioned earlier that you must register every MVC controller you wish to dependency inject with the IoC container. Sounds like a drag, right? Not so fast! Pull out your robe and wizard hat, grab this handy code, and register all your controllers automatically within a configurator:

public void Configure(IServiceCollection serviceCollection)
{
    // configurator per project? Use this:
    serviceCollection.AddMvcControllersInCurrentAssembly();

    // configure all the things from on high by convention? Use this (Habitat as the example):
    serviceCollection.AddMvcControllers(
        "Sitecore.Feature.*", 
        "Sitecore.*.Website");

    // you can also pass Assembly instances directly, but why?
    serviceCollection.AddMvcControllers(
        Assembly.FromName("Foo"), 
        Assembly.FromName("Bar"));
}

And without further ado, here’s the code that makes that possible.

Have a nice day!

Configuring domains from patch files

Sitecore domains are logical security groupings, for example the product ships with “sitecore” (backend users) and “extranet” (frontend users). But you do not have to stick with only the domains the product ships with, as usual for Sitecore you can extend and add your own.

Normally adding a domain means editing App_Config\Security\Domains.config, but we don’t want to do this. Why? Because editing standard Sitecore config files makes it difficult to upgrade Sitecore and introduces error prone file merging.

What we want to do instead is use config patch files. These allow us to add our config completely separately from Sitecore’s standard configuration files. But there’s a problem when it comes to domains: Domains.config does not live in the requisite <sitecore> config section so we cannot patch it.

Fortunately there’s a little known way around this. Someone at Sitecore implemented a config-based domain manager, but it’s not the default - presumably for backwards compatibility. And you can activate that, and add domains to it, using patch files.

So next time you want to add a domain to Sitecore, like say adding a domain for each site to provide logical author role groupings, switch over to the config domain manager. You know you want MySite\Editor instead of sitecore\MySite Editor. It’s easy to do, too.

This patch will activate the config domain manager (the defaults for the config manager already are the same as Domains.config so nothing else needs to be registered):

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <domainManager defaultProvider="file">
            <patch:attribute name="defaultProvider">config</patch:attribute>
        </domainManager>
    </sitecore>
</configuration>

And this patch is an example of adding a domain to the config domain manager:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <domainManager>
            <domains>
                <domain id="MyNewDomain" type="Sitecore.Security.Domains.Domain, Sitecore.Kernel">
                    <param desc="name">$(id)</param>
                    <ensureAnonymousUser>false</ensureAnonymousUser>
                </domain>
            </domains>
        </domainManager>
    </sitecore>
</configuration>

And there you have it. Enjoy!

Synthesis 8.2.1 Released

I am happy to announce that Synthesis 8.2.1 is available on NuGet. This release primarily adds additional features.

What’s New?

Improved Multiple Configuration Support

Previously registering multiple configurations in Synthesis was possible but way too hard. Configurations may now register themselves using code, similar to MVC area registrations.

To register a new configuration with Synthesis 8.2.1:

  • Add a class to the assembly you want the configuration’s model to live in that derives from SynthesisConfigurationRegistration
  • Implement the abstract members of SynthesisConfigurationRegistration. Much deeper customization is also available by overriding other optional members.
  • Add a SynthesisConfigRegistrar processor to the initialize pipeline that is set to scan your assembly, e.g.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    <pipelines>
    <initialize>
    <!-- IMPORTANT: Each registrar instance must have a unique hint value for the patch to work correctly. -->
    <processor type="Synthesis.Pipelines.Initialize.SynthesisConfigRegistrar, Synthesis" hint="Hinty McHintface">
    <assemblies hint="list:AddAssembly">
    <meAssembly>My.Assembly.WithModel</meAssembly>
    </assemblies>
    </processor>
    </initialize>
    </pipelines>
  • That’s it! Your configuration will now be activated.

Auto Friending

WHAT DID I EVER DO TO YOU?

Along with simpler configuration registration, allowing your configurations to reference each other’s generated classes is also far easier with “Auto-Friending.”

To illustrate this, suppose you were using Habitat and in your Project layer you had templates that inherited Feature templates. Without auto-friending (or previously manual friending), the Project would generate duplicate interfaces for the Feature templates. This is a bad thing. But with friending, the Project generated template will simply inherit from the already existing Feature model, extending implicit dependency in the database into explicit dependency at a code level (also a good thing!).

With auto-friending, configurations automatically friend each other in the order they are registered. So for the example above, as long as the Feature’s model configuration was registered before the Project’s model, everything would Just Work. You can control the order of registration by the order in the SynthesisConfigRegistrar assemblies.

IRenderingContext and improved IoC support (Synthesis.Mvc)

A pattern that I’ve been using for a while (as have others) to improve testability is to make a facade over the RenderingContext that turns it into a Synthesis API and removes all Item dependencies. This IRenderingContext can be registered with your IoC container (to SitecoreRenderingContext) and constructor-injected into controller renderings to make Item-free controllers that are easy to test without any hacks. Even awesome hacks like FakeDb.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
public class FooController : Controller
{
private readonly IRenderingContext _renderingContext;
public FooController(IRenderingContext renderingContext)
{
_renderingContext = renderingContext;
}
public ActionResult Foo()
{
var dataSource = _renderingContext.GetRenderingDatasource<IExpectedTypeItem>();
if(dataSource == null)
{
// no datasource set, or datasource is wrong template type (or context item, if no datasource set)
return Content("Derp.");
}
var model = new FooViewModel(dataSource);
// set other model props here
// Note that none of this controller directly used Sitecore APIs and thus does not require FakeDb nor HTTP context
// to have unit tests written against it.
return View(model);
}
}

Additional, more specific interfaces are also available for more specific use cases: IContextItem, IContextDatabase, and IContextSite. These can all be bound to SitecoreRenderingContext and provide smaller interfaces for more specific tasks.

Config Patching by Default

Synthesis now ships with Synthesis.LocalConfig.config.example, which is designed to be duplicated to form your own configuration patch. This encourages leaving the default configurations alone, which in turn greatly simplifies upgrading. The documentation and README has also been updated to reflect this.

Improvements

  • The default settings have been improved:
    • Creating model backup files is now disabled by default because it is of questionable utility when using source control
    • The InterfaceOutputPath and ItemOutputPath settings have been deprecated and merged into a single ModelOutputPath because it’s silly to emit more than one model file. The separate settings will still operate if you wish to use them.
    • Uncommonly used settings have been removed from Synthesis.config (SitecoreKernelAssemblyPath, SynthesisAssemblyPath, and InterfaceSuffix). They continue to work if set, but are removed for brevity as they are generally not used other than at default values.
  • The SynthesisEditContext class has been marked obsolete because the pattern is a bad idea
  • WebForms related classes have been marked obsolete because don’t use WebForms
  • The ability to attempt to automatically rebuild the project containing the model on startup has been removed due to being a generally bad idea
  • The ModelOutputBasePath setting has been added. This path is prepended to the ModelOutputPath for all configurations. The advantage of using this is that for people who work out of webroot, they can use <sc.variable> values in a setting (e.g. the out of webroot project location) whereas Sitecore does not expand variables in the ModelOutputPath. #27

Bug Fixes

  • Setting max backups to 0 no longer results in an infinite loop and now does not make backup files #28
  • Registering the same assembly twice in the type list provider will no longer scan the assembly twice
  • Wildcards that do not end in (e.g. Foo..Web) will not operate correctly when adding assemblies to the type list provider

Upgrading

Upgrading should be as simple as a NuGet upgrade*. If you have customized your Synthesis.config you may need to merge it with the default (or even better make it a patch file).

* as long as you are using Sitecore 8.1. As with the 8.2.0 release, it is designed only for Sitecore 8.1 due to breaking Sitecore API changes in 8.1. Sorry about that :(

Thanks!

Thank you to the community members who contributed to this release:

As always, happy coding!

Unicorn 3.1.5 Released

Unicorn 3.1.5 and Rainbow 1.3 are released to NuGet right now. This is a maintenance release. Upgrading is recommended, particularly if you are using Transparent Sync.

New Features

The new <syncConfiguration> dependency (see Unicorn.config) enables configurations to opt in to having synced items be updated into the link database and/or the search indexes after being changed by a sync.

This improves Unicorn’s compatibility with content syncing scenarios that are less developer focused, at the expense of reduced sync performance.

Transparent Sync also supports these flags, and items changed while Sitecore is alive to see them will have their links and indexes taken care of if set to do so. However if the IIS App Pool is not active during the file changes, nothing is updated.

Per-configuration concurrency

Unicorn 3 has always supported multithreaded sync and reserialize, but it has been disabled since release due to a concurrency bug in the Sitecore Template Engine that results in sync deadlocks when it is used to sync template item changes.

3.1.5 brings the ability to set the max thread count per-configuration (using the same <syncConfiguration> as indexing support) so that for configurations that are not syncing templates (or on Sitecore prior to 8.0U2) you may again enable concurrency. Threaded syncing can be 30-50% faster than non threaded.

Control panel UX improvements

Unicorn 3.1.5 brings a ‘Sync all’ button to the control panel, and a few minor UI tweaks such as relegating the log verbosity to an options modal.

Bug Fixes

  • Checkboxes inherited through several layers of standard values will no longer result in the derived template being reset to base standard values
  • Auto-publish no longer will miss publishing unversioned fields in certain cases
  • Renaming or moving items with transparent sync enabled and data cache turned on no longer results in a corrupted data cache
  • Change Template now works correctly with transparent sync
  • Unicorn now understands content items that are using Sitecore 8.1 language fallback without creating extra versions on the serialized copy
  • The log verbosity feature has had several issues fixed that would result in logs incorrectly not being shown at the selected level
  • Unicorn’s Micro IoC container now understands int constructor parameters
  • Adding or changing serialized template items when using transparent sync will now correctly clear the template engine, preventing an error
  • Automated build powershell examples are now more flexible when invoked in unusual path locations
  • In the control panel, the batch action buttons will no longer ‘overlay’ configuration checkboxes in certain situations
  • Logging of unversioned field updates will no longer incorrectly show a version number alongside the update in the logs
  • Certain unusual situations when syncing templates and items of that template will no longer throw a null reference due to stale caches

Upgrading

Upgrading from 3.1.4 is just a NuGet upgrade away. There are some config changes, so overwrite files and merge anything custom back in. Or even better use config patches and leave the default config alone :)

Note that Unicorn 3.1.5 changes the storage format for checkbox fields so that unchecked checkboxes are stored as ‘0’ instead of Sitecore’s standard blank value. This makes sync more reliable, however it does means that the first sync after serializing a new item with an unchecked checkbox will result in a ‘change’ as the zero value is pushed back into the Sitecore database. Checkboxes that are already serialized with blank values will continue to work just fine as they are.

Credits

Unicorn is a team effort of the Sitecore community. I’d like to thank the following folks who contributed to this release:

Dana Hanna
Kasper Gadensgaard
Mark Cassidy
Mike Carlisle
Søren Kruse
WizX20

And extra special thanks to Robin Hermanussen, Richard Seal, Alex Washtell, Mark Cassidy, and everyone else I missed for tirelessly answering questions in #unicorn on Sitecore Slack when I’m busy or not around!

The Solr Cannon: Rapid Solr Setup for Sitecore

Have you ever wished that standing up a Solr server for Sitecore was easy? I have. So I made a script to make it so.

The Solr Cannon automates the setup and configuration of Solr on Windows (Note: Linux may be a better production choice if you have the option), including installation as a background service, Sitecore schema installation, core creation, and generating a Sitecore config patch file that will point Sitecore at your shiny new Solr server and its cores.

What you’ll need

  • A copy of the Bitnami Solr Stack for Windows (I used 5.5.0-1)
  • A copy of the script and the solr schema file (the schema was generated with Sitecore 8.1 Update 1, you might need to generate your own if it doesn’t work)
  • PowerShell 3.0 or later

Let’s do this

  • Copy the Solr stack and scripts to a folder on the server that will run Solr
  • Review the Install Solr.ps1 script to make sure the variables are to your liking (project name, ports, solr stack installer path, etc)
  • Open an administrative PowerShell prompt
  • Execute Install Solr.ps1
  • Kick back for a couple minutes
  • Bask in the glow of your new Solr server and all of its Sitecore cores

But what about the rest?

Ok ok, that’s not all you need to do in order to configure Solr and Sitecore.

  • Check out Patrick Perrone’s post to help you swap in Solr as the search provider and configure the Solr IoC container. Note: I’d recommend Windsor for the IoC. Autofac and Ninject were less than awesome install experiences…
  • Once you have the Sitecore configuration flipped to Solr, you can grab the Solr.config file that the Solr Cannon produced when it installed Solr (written to the script directory). This config patch will:
    • Point Sitecore at your Solr server
    • Make Sitecore look at the correct core names for each index
    • Enable SwitchOnRebuildSolrIndex which allows you to rebuild indexes without any index downtime (great for production…or dev)
  • Load up Sitecore, open the indexing manager, and rebuild all your indexes. With a bit of luck, you’ll be good to go!

Multi-tenant Solr

Suppose you’re working on your own machine and you’ve got more than one dev site that’s using Solr. The Solr Cannon scripts can act to create a new config set and cores for several Sitecore installations on the same Solr server. Edit the script to have a different $ProjectName set, run it again, and say no when asked if you need to install a Solr server. In this mode, the script will build out a config set and cores for a new Sitecore site, as well as the patch file.

Have fun :)