Not too long ago I was considering the future of Unicorn. Rather naïvely, I couldn’t think of all that much that needed fixing. I wondered if there would ever be a Unicorn 4.

Now since you’ve read the title of this blog post, something tells me you don’t think that attitude lasted very long. It didn’t. Because there’s a pretty simple truth about Unicorn, even with all the optimizations I put in to Unicorn 3:

There were two primary causes of this:

1. The Sitecore API is slow and not suited to dealing with large quantities of items
2. Unicorn was originally designed to optimize for a few configurations, and Helix was blasting the number of active configurations to large numbers

I found that the sync time was becoming annoyingly long. Long enough that it was a distraction, long enough that a piddly 20% faster from more profiling just wasn’t good enough.

What to do, what to do.

Introducing Project Dilithium

I started on a crazy quest to speed up syncing performance. Sitecore’s recently added Publishing Service gained drastic speed gains in publishing by leveraging direct database access. Publishing, like syncing, is a batch-oriented operation. The Sitecore API is a single-item-oriented API. It does no optimizations, aside from caching, when you want more than one item. I resolved to try out SQL and see how it went. It went. See for yourself:

(benchmarks on i5-3570k@4GHz, SSDs, freshly restarted IIS + SQL, average of 3 runs each)

So what is Dilithium?

There are two pieces of the Dilithium project: SQL and Serialized. Both operate in a similar fashion: before a sync operation begins, all items that will be involved in that sync are read in a single big batch either from SQL or disk. The actual sync then occurs against the pre-batched items. In case you’re wondering, the graph above includes the batching times for Dilithium - no cheating :)

The SQL version takes advantage of the Descendants table to grab all predicate-included items for every configuration in a single huge SQL query (this does mean that FastQueryDescendantsDisabled must be turned off). These items are then indexed and prepared for fast access. In this way, the entire data contents of the stock core database can be read in about 1500ms, compared to the Sitecore API’s 8 seconds.

The serialized version skips all the checks that usually are needed to traverse a Serialization File System tree - expensive operations that guarantee that the children you’re getting all belong to the same item ID, for example. It’s essentially reading all files sequentially and caching them for sync just like the SQL version.

When both DiSerialized and DiSQL are used together, they enable parallel loading of both sources at once. This reduces the load time quite significantly, as you can see on the shortest bar on the chart above.

Questions?

Direct SQL, are you nuts?

Yes. Yes I am. Directly querying the Sitecore database is a terrible idea in almost every situation…except this. The schema for content has also been very stable for several major revisions of the product. And Stephen Pope said it was ok :)

Is it any faster adding or modifying items?

No. I’m not crazy enough to also do direct SQL writes, so all modifications go through normal Sitecore item saving APIs and perform pretty much the same as Unicorn 3. This also means that they’re mature, well-tested code - Unicorn 4 under the hood is not much different from v3, except for Dilithium.

Can I turn it off?

Heck yes. You can enable or disable Dilithium on a per-configuration basis. It also works just fine if you sync Dilithium and standard configurations together.

Serialized items and formats are identical between Dilithium and normal, so you can also enable and disable at will with no migration process.

Will this eat up all my RAM?

If you’re syncing gigabytes of media items…yes, definitely. For more normal developer and light content items, nope. DiSQL does not read blob data, so if you become RAM constrained turn off DiSerialized first.

Note that the precaches are disposed of immediately after the sync completes, so memory spikes are temporary.

Let’s address the 🐘 in the room, how is Unicorn 4 60% faster than 3 without Dilithium?

Actually pretty simple: eliminating a cache bombing that was done between syncing configurations that was needed for Transparent Sync. Well, it wasn’t needed if you’re syncing a non-Transparent configuration. Doing that meant that sites like Habitat with a ton of small non-Transparent configurations perform a lot faster.

Any other performance tricks?

In my daily development I use a mix of Transparent Sync for developer items (templates, renderings), and standard sync for content items. Since Transparent Sync is disk-based anyway there is little point to running a sync on Transparent Sync configurations when you’re just doing local development. As such, there’s now an option to skip Transparent Sync-enabled configurations when you sync all. This is an option in the PowerShell API in Unicorn 4, as well as an option in the control panel (same place as log level).

Depending on how many Transparent Sync configurations you have, your time savings can be significant from enabling this. You should not however do this when deploying transparent configurations to a shared server.

Can I have it?

Not yet, it still needs some additional testing and vetting before I call it alpha. If you’re interested in being a brave early tester, get ahold of me on Sitecore Slack :)

Is this the only new thing in Unicorn 4?

Nope. I’ve got some more tricks up my sleeve, so stay tuned for some more preview blogs coming soon™

Back in the bad old Web Forms days, rendering a page was a multi-step process. There were Init, Load, PreRender, and Render, just to name a few. This was mostly a pain to deal with as it made it hard to know when in the page lifecycle you should place your code to deal with framework side effects. It also made its fair share of inane interview questions.

But it did mean one useful thing: there was a place you could put code where it would always run even if a rendering was output cached, that went before the page actually rendered. Why was this useful? Suppose you needed a certain CSS class on <body> when a specific rendering was added to the page. Maybe a rendering would override the <title>, or any number of other techniques* which require code to execute prior to page layout composition.

* most of these techniques are just as bad as Web Forms :P

When MVC became popular, it brought its own single-step rendering pipeline. This is awesome because it’s easy to understand: the page rendering starts at the top and works its way down, rendering partials and the like in order. No more “the form value is missing because you added the dynamic field after init so data binding had already run.” (Seriously?) Simple.

But in a single step pipeline, there’s no room for a later component to signal an earlier component anything. Take this for example:

// support class
public class Body
{
    public static string CssClass { get; set; }
}

// layout
<body class="@Body.CssClass">
    @Html.Sitecore().Placeholder("wrapper")
</body>

// inside wrapper, in a controller
public ActionResult Rendering()
{
    Body.CssClass = "🐘";
    return View();
}

What will happen? Well, first the body tag will execute, then the wrapper placeholder will be executed. The body class will be set after the <body> tag has been rendered: there will be no class on the body.

There are two ways I know of to get around this.

Use a Layout

A layout? Aren’t we already using one of those? Nope: an MVC layout, not a Sitecore layout. In our example above, we could refactor the layout like this:

// _Layout.cshtml
<body class="@Body.CssClass">
    @RenderBody()
</body> 

// SitecoreLayout.cshtml
@{
    Layout = "~/Path/To/_Layout.cshtml";
}

// the body of this is placed in the @RenderBody() in the MVC layout
@Html.Sitecore().Placeholder("wrapper")

Here the Sitecore layout inherits the MVC layout by setting the Layout property. This also has the side effect that the Sitecore view is executed before the MVC layout. So in our example, now the SitecoreLayout.cshtml is rendered (including the rendering which sets the body class), and then the _Layout.cshtml renders and places the Sitecore content in the @RenderBody(). The body class is thus set how we want.

MVC layouts are also a lovely tool if you have several Sitecore layouts which share a lot of common boilerplate wrapper code - for example, the same meta tags, favicon, or version footer - without duplicating them several times. Sections enable you to have your Sitecore layout also write to specific places in the MVC layout. For example you might choose to expose a head section to enable the Sitecore layout to add CSS files or analytics scripts to the MVC layout <head>.

Reversed Rendering

The second way to control your renderings’ order is to use a technique I call Reversed Rendering. This is a simple trick that relies on the fact that HTML helpers are really just functions. This is a completely valid layout:

@{
    var wrapper = Html.Sitecore().Placeholder("wrapper");
}

<body class="@Body.CssClass">
    @wrapper
</body>

The Placeholder() method returns a HtmlString object, which contains the markup for the placeholder. You can take advantage of this to change the order by assigning the placeholder to a variable instead of rendering it directly into the output. Then you can control the order in which things render to make the body content render before the parent tags, thus facilitating communications.

I haven’t tried it (and performance wouldn’t be great dealing with large strings), but you could also theoretically use this trick to perform HTML post-processing on rendered renderings.

What about output caching?

Output caching gains performance by not executing the whole rendering pipeline and instead serving a pre-cached static version or a rendering. The problem is that by bypassing the rendering pipeline it also bypasses the code (controller or view) that pushes data up to the layout.

So when using these techniques, make sure to not enable output caching on any renderings that are setting variables used in the layout - or else the variables won’t be set as soon as the caching kicks in.

So there you have it. Go forth and have fun!

If you’ve worked with Sitecore for very long, you have probably needed to update an item’s content programmatically. A pattern in common usage for this is to use the EditContext class, for example:

Item item;
using(new EditContext(item))
{
    item["FieldName"] = "A new value";
}

Unfortunately, it has a fatal flaw. A fatal flaw that Jakob Christensen pointed out in 2006. Unfortunately it’s still in wide use, and is not marked as obsolete. Hopefully this post can help change that!

The problem lies in the way the using statement works. It is effectively equivalent to this:

var context = new EditContext(item);
try
{
    item["FieldName"] = "A new value";
}
finally
{
    context.Dispose();
}

An astute observer will notice that due to the finally semantic, the EditContext is always disposed, regardless of whether an exception occurs. Take this code for example:

Item item;
using(new EditContext(item))
{
    item["FieldA"] = "Sample";
    item["FieldB"] = GetFieldValue();
    item["FieldC"] = "A new value";
}

public string GetFieldValue() 
{
    throw new ArgumentException("Let's address the elephant in the room.");
}

In this example GetFieldValue() throws an exception. This will cause an immediate drop through to the finally of the using statement, which executes whether an exception is thrown or not. Which will commit the partial item change to FieldA without FieldB or FieldC, leaving the Sitecore item in an inconsistent state. Which you probably do not want.

So what should we use?

The correct way to edit an item is with the Editing property, for example:

Item item;
item.Editing.BeginEdit();
item["FieldA"] = "Sample";
item["FieldB"] = GetFieldValue();
item["FieldC"] = "A new value";
item.Editing.EndEdit();

Note how EndEdit() will not be called if an exception occurs. This prevents Sitecore from writing any field changes to the database (changes are queued up and committed all together when EndEdit() is called).

This syntax is however a bit hard to read, because it is difficult to parse the code compared to the neat indentation of EditContext. I would like to propose an alternative syntax, taking advantage of the fact that you can use arbitrary braces in C#:

Item item;

item.Editing.BeginEdit();
{
    item["FieldA"] = "Sample";
    item["FieldB"] = GetFieldValue();
    item["FieldC"] = "A new value";
}
item.Editing.EndEdit();

There! Now it reads nearly like EditContext, but does not suffer from its downsides :)

I am happy to announce the release of Synthesis 8.2.2.

Synthesis is an object mapping framework for Sitecore that enables developing more reliable and maintainable sites in less time than traditional Sitecore development. It is a strongly typed template object generator that is easily understandable for developers with either a Sitecore or traditional .NET background.

What’s new?

Automatic Model Regeneration

Synthesis now ships by default with event handlers that automatically regenerate your model classes when templates are changed, renamed, moved, or deleted. Whereas before you’d have to manually request a regeneration, now you can just save the template and within 2-3 seconds your model classes have been updated.

Automatic Model Regeneration turns itself off if <compilation debug="false"> is set in the web.config. You can choose to disable it in published scenarios either this way or by deleting its Synthesis.AutoRegenerate.config file when deploying. #37

Hyperlink Tag Body Rendering

Previously Synthesis’ MVC helpers did not have a way to render a hyperlink field with an arbitrary HTML body within (e.g. an image, etc). This has been rectified in 8.2.2, with the new BeginHyperlinkFor helper. #31

@using(Html.BeginHyperlinkFor(m => m.HyperlinkField)) {
    <h1>Woohoo!</h1>
    <img src="homer.gif">
}

Config Files In Their Own Folder

The Synthesis configuration files have been moved to App_Config/Include/Synthesis for clarity. The NuGet package upgrade will take care of the migration.

.NET Framework 4.5.2

Synthesis 8.2.2 requires that the project it is installed on be targeting the .NET Framework 4.5.2 or later. This enables full Sitecore 8.2 compatibility. Synthesis 8.2.2 should work on Sitecore 8.1 or later.

Bug Fixes

• Model source files whose contents have not changed in the current regeneration are now not rewritten to disk. This prevents their timestamp from changing and thus triggering a need to rebuild their host project. When using modular architecture, this can greatly reduce build times. #32
• Fixed a bug when using Solr indexes where Synthesis’ regenerate process would throw an exception and possibly result in an incomplete model #34
• Specifying a model output path in a directory that does not exist will now create that directory instead of erroring #35
• The content search configuration has been adapted to register the Synthesis _templatesimplemented computed field in such a way that it works with 8.1 and 8.2’s new registration scheme, as well as with Solr indexes. Note that the Synthesis content search integrations (querying) do not otherwise support Solr still.
• You can now disable the generation of Content Search elements in your models if you wish, using the EnableContentSearch setting. The default value is set in Synthesis.config. This will enable using models with fewer project reference requirements, if you do not need the search integrations.

Upgrading

If coming from Synthesis 8.2.x, the upgrade is via a simple NuGet upgrade. For earlier versions, consult the instructions on the release posts for the versions between what you’re on and where you’re going.

Thanks

Thanks to the community members who contributed to this release.

• Peter Holm
• rolek

Conditional inversion is a simple but powerful technique that makes your code easier to read. I’ve noticed that not a lot of people in the Sitecore community seem to know about it, so I thought I’d blog about it.

Inverting your ifs

Deeply nested code is pretty hard to read. Let’s take this example:

public class Class1
{
    private static object Lock = new object();
    private static string Value = false;

    public string GetValue()
    {
        if (Value == null)
        {
            lock (Lock)
            {
                if (Value == null)
                {
                    Value = ExpensiveCreateValueMethod();
                }
            }
        }

        return Value;
    }
}

This is a very common pattern in multithreaded code, the double-check lock. It ensures that the Value is not initialized more than once at the same time by different threads. It’s also fairly hard to follow the code: it has a lot of nested blocks, and the information density is not very high.

Let’s take a different approach to the same code.

Let’s invert our if statements so we can get out of our method as quickly as possible.

public class Class1
{
    private static object Lock = new object();
    private static string Value = false;

    public string GetValue()
    {
        if (Value != null) return Value;
        
        lock (Lock)
        {
            if (Value != null) return Value;
            
            return Value = ExpensiveCreateValueMethod();
        }
    }
}

With inverted if statements we check not for success but for failure. This method has two fewer nesting levels, is four lines shorter, and is significantly more readable because you don’t have to traverse a long if block to see what code will be executed next on success. Imagine outside a contrived example like this one, where your if might be 50 lines long.

So next time you’re writing conditionals, think about handling the failure before the success. Instead of:

public string Foo(string bar) 
{
    if(bar != null) {
        // do
        // stuff
        return something;
    }

    return null;
}

Do this:

public string Foo(string bar) 
{
    if(bar == null) return null;

    // do
    // stuff
    return something;
}

Nicer, right?

Feeling loopy?

The same inversion technique is also lovely for loop control. Have you ever written a loop like this?

foreach (var foo in bar)
{
    if (foo != null)
    {
        // do
        // stuff
    }
}

When inverting loop control you make use of the continue statement. This is an underused gem that skips the current loop iteration just like return skips the rest of a method. We can rewrite the loop above for better readability like so:

foreach (var foo in bar)
{
    if (foo == null) continue;
    
    // do
    // stuff
}

Now if foo is null, we just skip the rest of the loop body and go to the next item. Handy, right?

Bonus points: asserting reference types

A special case of this kind of inversion is the use of the Sitecore.Diagnostics.Assert class. When you’re designing a method, any reference type parameters to that method are allowed to be null. I’ll bet you didn’t expect someone to pass null, right? (I’ll bet I didn’t either!)

public void DoStuff()
{
    // null is usually not overtly passed, e.g. looping over dynamic data
    var stuffFromRestService = new[] { "foo", null, "bar" };
    foreach (var attribute in stuffFromRestService)
    {
        DoAThing(attribute);
    }
}

public void DoAThing(string attribute)
{
    // boom
    attribute.ToLowerInvariant();
}

Then you get this most favourite error, with a ‘helpful’ stack trace in the middle of the offending method:

Not helpful, right? Well we can handle this more gracefully if we assert that our parameters that are reference types are not null - then we get an exception at the top of the method that is useful and tells us what really happened:

// using Sitecore.Diagnostics;
public void DoAThing(string attribute) 
{
    Assert.ArgumentNotNull(attribute, nameof(attribute));

    // do stuff
    attribute.ToLowerInvariant();
}

And we get a better error message that tells us the argument name and what happened:

Pretty handy, right? Now go forth and code!

I’m happy to announce the release of Unicorn 3.3 and Rainbow 1.4. This is primarily a maintenance and bug fix release and is recommended for all users of Unicorn 3.x.

As there was no official announcement for Unicorn 3.2, this will also note what was new in 3.2. Items from 3.2 will be noted as such.

What’s new?

Improved Configuration Dependencies

Dependencies between configurations have been improved to include implicit dependencies that are created when one configuration includes a path that is a child of another configuration. This simplifies the need to define explicit dependencies in many cases. This config file documents how this works and how to disable it if you need to. #165

Improved Debugging with PowerShell API

You can now enable diagnostic logging of the CHAP base signature on the client and server side, so if you have issues getting the tool authentication working you can diagnose why. See this and this for configuration information. #182

Full Sitecore 8.2 Compatibility

Unicorn and Rainbow now build against and require .NET 4.5.2 for Sitecore 8.2 compatibility. This means that your project referencing Unicorn must also target framework 4.5.2 or later, even if not using Sitecore 8.2. Unicorn should work with Sitecore 7 and later. #175

(3.2) Improved Exclusion Grammar

You can now exclude children of a child of an included path. #138

(3.2) User and Role Serialization

Unicorn 3.2 introduced YAML-based user and role serialization via the Unicorn.Users and Unicorn.Roles NuGet packages. User and role serialization works similarly to item serialization. To add this capability to your Unicorn installation you simply install the correct NuGet package and the readme will guide you, or you can read the documentation using the links below.

• Roles README
• How to configure roles
• Users README
• How to configure users

Bug fixes

• When deserializing an item that contains a field missing in Sitecore, the field hint is shown instead of just its ID. This aids in tracking down the offending field, since it does not exist to find in the DB. #169
• The Unicorn.SharedSecret.config.example has been renamed to include a z so that simply renaming it to use does not cause a config load order issue #181
• Fixed an issue with null vs empty field values that could cause an exception in Rainbow #16
• Fixed an issue that would cause missing language data when you removed a version from a tracked item using the Sitecore API within an EventDisabler block #168
• Fixed an error that occurred when an item changed template, but the old template was already deleted as part of the same sync process and thus no longer existed #164
• An issue that could cause spurious “Item changed, do you want to overwrite?” popups when saving an item in Experience Editor and using Transparent Sync has been fixed #163
• Fixed an issue that could cause a spurious Unicorn conflict warning when saving items with checkboxes in certain cases #151
• Serialized items with field values that equaled the standard values defined in the database are no longer left as standard values, and are correctly explicitly set to their serialized value. This was a common cause of items that ‘synced every time’. #162
• Moving an item between Unicorn configurations will correctly respect exclusions that exclude some children of the target location #161
• Unicorn is now free of uses of EditContext, which is a 10-year-old antipattern #160
• Pathing that escaped from the webroot using ~/ no longer breaks when set as the serialization path #157
• Errors about being unable to send headers after they have already been sent in Sitecore 8.1 Update-3 and later have been resolved #155
• Unicorn Auto Publish now works correctly if it is set to publish to more than one publishing target #152
• Renaming items whose filenames have been truncated on disk (very long db item names) will no longer result in the creation of duplicate files #146
• Items whose only change between db and disk is the BranchID are now synced correctly #144
• YAML values containing \ and " are now correctly treated as multiline values so that serialized files are valid YAML. This is a non-breaking change, but you may notice values such as __Created by being altered as items are saved. All Unicorn 3.x versions can parse either format without issue. #143
• You can now explicitly add a wildcard item (an item named *) to a Unicorn configuration without it being interpreted as a wildcard character. See test cases #142
• The Unicorn data provider now returns null for ChildIDs if no children exist to enhance compatibility with the Sitecore data provider’s behaviour #141
• A race condition when using multithreaded sync and auto publishing has been fixed #183
• Rapidly clicking a checkbox in the control panel will no longer potentially corrupt the page state #158
• Unicorn and Rainbow both now reference Sitecore from the Official NuGet Feed, so if you need to build from source copying Sitecore assemblies is no longer required. Just clone and build!
• 3.2 Setting exclusions on a configuration that uses Transparent Sync is now an exception (this has never been a supported configuration, just now it’s an error too)
• 3.2 Configuration predicates with no include nodes are now an error instead of including all items

Upgrading

To upgrade to Unicorn 3.3, you need to make sure the project hosting Unicorn and Rainbow is targeting .NET 4.5.2 or later. Then simply upgrade the NuGet package, if you have Unicorn 3.1 or later.

For earlier versions, follow the upgrade instructions on the appropriate launch blog post (e.g. for 3.0 or 3.1) before you upgrade to 3.3.

Installing

Install the Unicorn package from NuGet. A readme will be shown to help you get started after install. You must be using packages.config (NuGet 2.x style) package management for Unicorn and Rainbow, because they install content items to the project. If you wish to use project.json (NuGet 3.x style), you must manually install the configuration files for Unicorn and Rainbow because of limitations in NuGet 3.x.

Thanks

As usual Unicorn is not just me, it’s a community project. Thank you all for your pull requests, issue reports, and for using my work.

I’d like to specially thank Mark Cassidy for spending a lot of time tracking down - and fixing - some Unicorn issues he was having. All contributors to this release include:

• Tyler Allen
• Jason Booth
• Mark Cassidy
• Steve Crabb
• Jeff Darchuk
• Mads Due
• Thomas Eldblom
• HenkMahieu
• Ben Lipson
• mzizka
• Jawad Sabra
• Thomas Stern
• Tony Wang
• wolfeno
• Una Verhoeven

I’m happy to announce the release of Dianoga 3.0. This release brings significant improvements compared to previous releases, and is recommended for existing users.

What’s Dianoga?

Dianoga is a tool that optimizes images uploaded to the Sitecore media library to save bandwidth and improve page loading times as a result.

Dianoga ensures that your site is always serving fully optimised media library images even if you are using Sitecore’s dynamic resizing features (for example with Adaptive Images). Even if you have already optimized every image uploaded into the media library, after Sitecore performs image processing the result is not optimized (an unfortunate limitation of other image optimization libraries for Sitecore is that they only apply to the original image).

Dianoga is also great for situations where content editors may not be image editing experts and upload images that contain gobs of EXIF data and other nonessential metadata - these are removed automatically before being shown to visitors. All of the optimizations done are lossless (no quantizing, etc) so you lose no image quality.

What’s new in 3.0?

SVG support

With the rise of SVG as a common format for media library media, it became apparent Dianoga needed to support optimizing SVG. Dianoga 3 includes SVG optimization!

SVG media is automatically:

• Optimized for size (the SVG is processed using SVGO)
• Gzipped before going in the media cache, and served using the cached gzipped version (reduces size over the wire)
• Apropos configuration to enable SVG support in media library on Sitecore < 8.1 is enabled by default (thanks to Kamruz Jaman and Anders Laub for the blogs e.g. this and this)

Optimization strategies

Dianoga 1.x used a synchronous optimization technique that resulted in a slower initial image load time, but always served optimized images. Dianoga 2.x instead relied on an asynchronous technique where the first media served would be unoptimized and subsequent hits were optimized, but the response time was never impacted.

It became apparent that both of these strategies have their place, for example you need to be synchronous when uploading to a CDN, and so Dianoga 3 supports optimization strategies that let you choose when to optimize. If you don’t like any of the strategies or want to optimize only when sending media to a CDN programmatically, you can also invoke the optimization pipeline directly to optimize precisely when you need to (the MediaOptimizer class is what you’re after here).

Pipeline-based optimization

Dianoga is now powered by Sitecore pipelines, providing simple and flexible extension options. The root <dianogaOptimize> pipeline runs for all optimizations, and spins off into <dianogaOptimizeJpeg> and other similar sub-pipelines for individual file types.

Optimizer chaining

With Dianoga 2, you could not simply apply more than one optimizer to a file format. For example, you might wish to quantize a PNG (which is lossy), and then also optimize its encoding after quantization to further reduce file size. This is very simple with Dianoga 3 - now each optimizer is just a step in a pipeline. Add or remove optimizers as you wish.

Media path exclusion

Got a folder of huge photos you don’t want optimized because they should be kept pristine for downloads? Have another reason you want to not optimize specific parts of the media library? Great - Dianoga 3 now supports that.

Because optimization is pipeline based, this just takes the form of a processor that aborts optimization when the input is to be ignored. See Dianoga.ExcludePaths.config.example which ships with the NuGet package for how to use this.

Framework version requirements

Dianoga 3 requires .NET 4.5.2 or later to be the target framework of the project it is referenced in. This provides compatibility with Sitecore 8.2. Dianoga 3 requires Sitecore 7.x or later.

Bugs and fixes

• Asynchronous optimization has been significantly simplified and reliability improved compared to Dianoga 2. File in use issues should be completely eliminated.
• Unit tests have been added to the codebase (I blame Dan Solovay)
• Optimization tools are moved to App_Data so that IIS will never consider serving their files (as opposed to /Dianoga Tools in 2.x and earlier)
• Optimization tools have been updated:
  • libjpeg is now mozjpeg, which results in better optimization for the web specifically
  • SVGO has been added to optimize SVGs
  • PNG optimization remains unchanged using PNGOptimizer, other than now being chainable with the PNGQuant lossy quantizer.

Installing

Dianoga is available from NuGet. You must be using packages.config (NuGet 2.x style) package management for Dianoga, because it installs content items to the project. If you wish to use project.json (NuGet 3.x style), you must manually install the tools and configuration files because of limitations in NuGet 3.x.

Upgrading

Easy. Make sure your host project is targeting .NET 4.5.2 (or later - I use it with 4.6.x), and then upgrade your NuGet package.

Have fun!

Suppose someone sends you a Sitecore solution to review, and they forgot to send you a username and password. You could ask for one, or you could just make yourself an account with this handy trick that I call “The Shiv.” This is the entirety of the trick:

1. Create a Shiv.aspx file somewhere under the webroot. It can be named anything.

2. Paste the following code in it

    <%@ Page Language="C#" AutoEventWireup="true" %>
    <%@ Import Namespace="Sitecore.Security.Authentication" %>
   
    <%
        AuthenticationManager.Login("sitecore\\admin", false, true);
        Response.Redirect("/sitecore/shell");
    %>
   

3. Hit the page in a browser

4. Boom, you’re an administrator

5. IMPORTANT: Delete the file. For obvious reasons.

Handy, eh?

You can do a very similar thing using the “Login as administrator” option in SIM, however I often find myself in environments without SIM and this code works anywhere.

This code is also a good security reminder: if someone malicious can upload an arbitrary file somewhere in your webroot that is then executed, they can upload this shiv-file and your security is gone. It doesn’t matter if you have encrypted 64-character database passwords, they’re in. It doesn’t matter if you’ve locked down TLS and imposed SAML logins, they’re in. Game over. So secure your filesystem and be awfully wary of accepting users’ uploads anywhere on disk.

Recently, in a fit of brilliance, Sitecore released a public NuGet feed that you can use to reference Sitecore assemblies from your projects. While some people have been doing this with home grown packages for years, it’s nice to have a stable, official source to get your references from.

If you’re not familiar with how this works, Jeremy Davis wrote a great post about the details of using the official feed.

Okay so what are you on about?

If you’ve got a project of reasonable size, especially one using Helix, you probably have a bucketload of references to Sitecore assemblies. Manually converting all these references to packages is a bit of a tedious process, and you know what we do to tedious processes around here.

That’s right, we script them.

How?

Migrating your direct Sitecore references to NuGet is a quite simple process with this script. For obvious reasons, use source control so you have a fallback in case the script doesn’t work for your particular setup. It worked for me on several projects, but just in case :)

1. Install NuGet.config

Copy this to the root of your solution:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!--
  Used to specify the default Sources for list, install and update.
  -->
  <packageSources>
    <add key="Official Sitecore" value="https://sitecore.myget.org/F/sc-packages/api/v3/index.json" />
  </packageSources>
  
  <activePackageSource>
    <!-- this tells that all of them are active -->
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

This will add the official Sitecore NuGet package feed to your solution. Unlike adding it via Visual Studio, this will also apply for CI and MSBuild-executed builds.

2. Copy Nugetify.ps1

Copy the following PowerShell script to the root of your solution. Open it in a text editor and set the correct $SitecoreVersion and $FrameworkVersion for your solution. The NuGet feed has packages for Sitecore 7.0-8.2.

# Script to convert all sitecore assembly references to Sitecore Public NuGet feed
# Run from root solution folder (where your packages folder is)
# execute this script with powershell
Param
(
    $sitecoreVersion = '8.2.160729', # NuGet package version to convert to. Format is major.minor.releasedate.
    $frameworkVersion = 'net452' # for 8.2: net452. For 7.0-8.1: net45
)

$ScriptPath = $MyInvocation.MyCommand.Path
$ScriptDir  = Split-Path -Parent $ScriptPath
$MsbNSString = 'http://schemas.microsoft.com/developer/msbuild/2003'
$MsbNS = @{msb = $MsbNSString}
$PackagesConfigFileName = 'packages.config'

#Create project.json from packages.config
Write-Host 'Scanning for projects to update...' -ForegroundColor Green
Get-ChildItem -path '.' -Recurse -Include $PackagesConfigFileName |
    ForEach {
        $PackageFilePath = $_.FullName
        $PackageFileDir = $_.Directory
        Write-Host "Processing $PackageFilePath" -ForegroundColor Green

        # Find existing csproj to match direct references
        $csproj = Resolve-Path "$($_.Directory)\*.csproj"
        $proj = [xml] (Get-Content $csproj)

        # Find existing Sitecore references and NuGet-ify them
        Write-Host "Checking for non-NuGet Sitecore references in $csproj"
        $xpath = "//msb:Reference/msb:HintPath[not(contains(.,'packages\'))]"

        $changedProj = $false
        $sitecoreNuGetPackages = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {
            $node = $_.Node.ParentNode

            $referenceName = $node.Attributes['Include'].Value.Split(',')[0]

              # Filter non-NuGet references to transform into NuGet packages
            if($referenceName.StartsWith("Sitecore") `
                -and -not $referenceName.StartsWith('Sitecore.Modules') `
                -and -not $referenceName.Contains('WFFM') `
                -and -not $referenceName.StartsWith('Sitecore.Forms') `
                -and -not $referenceName.StartsWith('Sitecore.Foundation') `
                -and -not $referenceName.StartsWith('Sitecore.Feature') `
                -and -not ($referenceName.StartsWith('Sitecore') -and $referenceName.EndsWith('Website'))) {

                $changedProj = $true

                Write-Host "NuGet-ifying assembly reference $referenceName"

                # set hintPath to package path
                Push-Location -Path $PackageFileDir
                $hintPathRoot = Resolve-Path "$ScriptDir\packages" -Relative
                Pop-Location

                $hintPath = "$hintPathRoot\$referenceName.NoReferences.$sitecoreVersion\lib\$frameworkVersion\$referenceName.dll"

                $existingHintPath = $node['HintPath', $MsbNSString]
                if($existingHintPath -eq $null) {
                    $hint = $proj.CreateElement("HintPath", $MsbNSString)
                    $hint.InnerXml = $hintPath
                    $foo = $node.AppendChild($hint)
                } else {
                    $existingHintPath.InnerXml = $hintPath
                }

                "$referenceName.NoReferences"
            }
        })

        if($changedProj) {
            Write-Host "Saving NuGet-ified references to csproj" -ForegroundColor Yellow
            $proj.Save($csproj)
        } else {
            Write-Host "Found no references to change."
        }

        # Add packages to packages.config
        $packageXml = [xml] (Get-Content $PackageFilePath)

        $sitecoreNuGetPackages | % {
            $packageNode = $packageXml.CreateElement('package');
            $packageNode.SetAttribute('id', $_)
            $packageNode.SetAttribute('version', $sitecoreVersion)
            $packageNode.SetAttribute('targetFramework', $frameworkVersion)

            $foo = $packageXml.DocumentElement.AppendChild($packageNode)
        }

        Write-Host "Updating packages.config with new packages" -ForegroundColor Yellow
        $packageXml.Save($PackageFilePath)
    }

3. Run Nugetify.ps1

Open a PowerShell and execute Nugetify.ps1.

That’s it. Open Visual Studio and verify that everything was converted correctly, and you should be good to go.

What about NuGet 3 + project.json?

Another option is to use the NuGet 3.x style package management which is integrated into a project.json file that lives next to your csproj files. NuGet 3’s major advantage is that the project.json both references packages and adds them to your project references. So adding packages does not result in alterations to the csproj file, and upgrading packages is as simple as changing a json file.

Sounds idyllic, right? Well there’s one huge downside. Microsoft, in their infinite wisdom, decided that it was not necessary to support content files being deployed into the project the package is installed into. For Sitecore projects, that means packages that come with config files, such as Unicorn, Synthesis, and Glass Mapper, will not install those files into projects using project.json. The content-containing packages can still be used, but then it becomes your task to reverse engineer the content they install, add that to your project, and handle upgrades of those content files when the package is upgraded.

For the moment, I wouldn’t use project.json, but I hope it becomes more tenable in the future. But if you want to use it, I have a script for that too - a script that both nugetifies your Sitecore references and converts all of your packages in packages.config to project.json. This script is based on this one.

# Script to generate project.json for all packages.config file in the solution.
# This script will also migrate non-NuGet Sitecore package references to Sitecore Public NuGet feed

# Run from root solution folder
# execute this script with powershell

# TargetFramework: Use the .NET framework version your projects are targeting, which may NOT be the version Sitecore is built aginst
# SitecoreVersion: NuGet version you want to convert to for local sitecore assembly references
Param
(
      [string] $TargetFramework = "net452",
    $sitecoreVersion = '8.2.160729'
)

# Filter existing installed NuGet packages to transform versions and such
function Filter-Packages {
    $input | % {
        $package = $_.Node.id
        $version = $_.Node.version

        # Translate nuget package generator 8.2 package version to official
        if($version -eq "8.2.0.160729") {
            $version = "8.2.160729"
        }

        # Translate 3rd party refs from old SC versions to target public versions
        if($package.Equals('Microsoft.Extensions.DependencyInjection.Abstractions')) {
            $version = '1.0.0'
        }

        if($package.Equals('MongoDB.Driver')) {
            $package = 'mongocsharpdriver'
            $version = '1.11.0'
        }

        # Blacklist these older sitecore nuget generator metapackages, modules, and nonexistant packages
        if($package.EndsWith("-Core") `
            -or $package.EndsWith("-CoreGroup") `
            -or $package.Equals('MongoDB.Bson') `
            -or $package.Equals("Telerik.Web.UI")) {

            return # skip loop
        }

        # Packages that started with Sitecore before should now get NoReferences for sanity (note: you may need to exclude sitecore modules whose name begins in Sitecore here)
        if($package.StartsWith('Sitecore') `
            -and -not $package.StartsWith('Sitecore.FakeDb') `
            -and -not $package.EndsWith('PatchableIgnoreList')) {

            $package = "$package.NoReferences";
        }

        $_.Node.id = $package
        $_.Node.version = $version

        $_
    }
}

$ScriptPath = $MyInvocation.MyCommand.Path
$ScriptDir  = Split-Path -Parent $ScriptPath
$MsbNS = @{msb = 'http://schemas.microsoft.com/developer/msbuild/2003'}
$PackagesConfigFileName = 'packages.config'

#Create project.json from packages.config
Get-ChildItem -path '.' -Recurse -Include $PackagesConfigFileName |
    ForEach {
        $PackageFilePath = $_.FullName
        $ProjectFilePath = $_.Directory.FullName + '\project.json'
        Write-Host "Processing $PackageFilePath"

        # Find existing csproj to match direct references
        $csproj = Resolve-Path "$($_.Directory)\*.csproj"
        $proj = [xml] (Get-Content $csproj)

        # Find existing Sitecore references and NuGet-ify them
        Write-Host "Checking for non-NuGet Sitecore references in $csproj"
        $xpath = "//msb:Reference/msb:HintPath[not(contains(.,'packages\'))]"

        $changedProj = $false
        $sitecoreNuGetPackages = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {
            $node = $_.Node.ParentNode

            $referenceName = $node.Attributes['Include'].Value.Split(',')[0]

      # Filter non-NuGet references to transform into NuGet packages
            if($referenceName.StartsWith("Sitecore") `
                -and -not $referenceName.StartsWith('Sitecore.Modules') `
        -and -not $referenceName.Contains('WFFM') `
        -and -not $referenceName.StartsWith('Sitecore.Forms') `
                -and -not $referenceName.StartsWith('Sitecore.Foundation') `
                -and -not $referenceName.StartsWith('Sitecore.Feature') `
                -and -not ($referenceName.StartsWith('Sitecore') -and $referenceName.EndsWith('Website'))) {

                $changedProj = $true

                Write-Host "NuGet-ifying assembly reference $referenceName"

                # remove old reference we're NuGet-ing
                [void]$node.ParentNode.RemoveChild($node);

                "$referenceName.NoReferences"
            }
        })

        if($changedProj) {
            Write-Host "Saving NuGet-ified references in $csproj"
            $proj.Save($csproj)
        }

        # Generate project.json
        $file = '{
  "dependencies": {
'

$packages = (Select-xml -xpath '//package' -Path $PackageFilePath | Filter-Packages | % { "    ""{0}"": ""{1}""" -f $_.Node.id,$_.Node.version }) -join ",`r`n"

$file += $packages;

$sitecorePackages = (($sitecoreNuGetPackages | % { "    ""{0}"": ""{1}""" -f $_, $sitecoreVersion }) -join ",`r`n")

# separate the json elements if both converted and sitecore packages exist
if($packages.Length -gt 0) {
    $file += ",`r`n"
} else {
    $file += "`r`n"
}

$file += $sitecorePackages

$file += '
  },
  "frameworks": {
    "' + $TargetFramework + '": {}
  },
  "runtimes":  {
      "win-anycpu": {},
      "win": {}
  }
}'

$file | Out-File $ProjectFilePath

    Remove-Item $PackageFilePath
}

Get-ChildItem -path '.' -Recurse -Include '*.csproj' | ForEach {
    $CsProjFilePath = $_.FullName
    $ProjectFilePath = $_.Directory.FullName + '\project.json'

    Write-Host $csProjFilePath

    $proj = [xml] (Get-Content $CsProjFilePath)

    #Remove all references to ..packages files
    $xpath = "//msb:Reference/msb:HintPath[contains(.,'packages\')]"
    $nodes = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {$_.Node})
    if (!$nodes) { Write-Verbose "RemoveElement: XPath $XPath not found" }
    Write-Output 'Reference Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $referenceNode = $node.ParentNode
        $itemGroupNode = $referenceNode.ParentNode
        [void]$itemGroupNode.RemoveChild($referenceNode)
    }
    [System.XML.XMLElement] $itemGroupNoneNode = $null
    #Find itemgroup with None Elements, if not found add.
    $itemGroupNoneNodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output '$itemGroupNoneNode found: ' $itemGroupNoneNodes.Count
    if($itemGroupNoneNodes.Count -eq 0){
        # create itemgroup element for None nodes.
        Write-Output 'Adding ItemGroup for None Nodes'
        $itemGroupNoneNode =  $proj.CreateElement('ItemGroup',$proj.DocumentElement.NamespaceURI)
        $itemGroupNodes = @(Select-Xml -xpath "//msb:ItemGroup" $proj -Namespace $MsbNS | foreach {$_.Node})
        #$itemGroupNodes.Count
        [void]$proj.DocumentElement.InsertAfter($itemGroupNoneNode,$itemGroupNodes[$itemGroupNodes.Count-1])

    }else{
        $itemGroupNoneNode = $itemGroupNoneNodes[0].ParentNode
    }

    #Remove packages.config from ItemGroup
    $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None[@Include='packages.config']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'packages.config Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Remove packages.config from ItemGroup (if it was set to content)
    $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:Content[@Include='packages.config']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'packages.config Nodes found: ' $nodes.Count
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Remove build target EnsureNuGetPackageBuildImports from csproj
    $nodes = @(Select-Xml -xpath "//msb:Target[@Name='EnsureNuGetPackageBuildImports']" $proj -Namespace $MsbNS | foreach {$_.Node})
    Write-Output 'EnsureNuGetPackageBuildImports target found: ' $nodes.CountAd
    foreach($node in $nodes) {
        $itemGroupNode = $node.ParentNode
        [void]$itemGroupNode.RemoveChild($node)
    }

    #Add project.json to itemGroup
    if( Test-Path $ProjectFilePath){
        $nodes = @(Select-Xml -xpath "//msb:ItemGroup/msb:None[@Include='project.json']" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $projectJsonNoneNode = $proj.CreateElement("None", $proj.DocumentElement.NamespaceURI)
            $projectJsonNoneNode.SetAttribute("Include","project.json")
            [void]$itemGroupNoneNode.AppendChild($projectJsonNoneNode)
            Write-Output 'Adding None node for project.json'
        }
    }

    #add PropertyGroup nodes targetFrameworkProfile, CopyNuGetImplementations, PlatformTarget
    # Find the TargetFrameworkVersion to be used to find the parent PropertyGroup node
    $xpath = "//msb:PropertyGroup/msb:TargetFrameworkVersion"
    $nodes = @(Select-Xml -xpath $xpath $proj -Namespace $MsbNS | foreach {$_.Node})
    if ($nodes.Count -gt 0) {
        [System.XML.XMLElement] $node = $nodes[0]
        $propertyGroupNode = $node.ParentNode
        $nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:TargetFrameworkProfile" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $node = $proj.CreateElement("TargetFrameworkProfile", $proj.DocumentElement.NamespaceURI)
            [void]$propertyGroupNode.AppendChild($node)
            Write-Output 'Adding TargetFrameworkProfile node for PropertyGroup'
        }
        #$nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:CopyNuGetImplementations" $proj -Namespace $MsbNS | foreach {$_.Node})
        #if($nodes.Count -eq 0){
        #    $node = $proj.CreateElement("CopyNuGetImplementations", $proj.DocumentElement.NamespaceURI)
        #    $textnode = $proj.CreateTextNode("true")
        #    $node.AppendChild($textnode)
        #    [void]$propertyGroupNode.AppendChild($node)
        #    Write-Output 'Adding CopyNuGetImplementations node for PropertyGroup'
        #}
        $nodes = @(Select-Xml -xpath "//msb:PropertyGroup/msb:PlatformTarget[not(@*)]" $proj -Namespace $MsbNS | foreach {$_.Node})
        if($nodes.Count -eq 0){
            $node = $proj.CreateElement("PlatformTarget", $proj.DocumentElement.NamespaceURI)
            $textnode = $proj.CreateTextNode("AnyCPU")
            $foo = $node.AppendChild($textnode)
            [void]$propertyGroupNode.AppendChild($node)
            Write-Output 'Adding PlatformTarget node for PropertyGroup'
        }
    }

    # replace ToolsVersion with 14.0
    $attibutes = Select-Xml -xpath "//@ToolsVersion" $proj -Namespace $MsbNS
    foreach ($attribute in $attibutes){

        $attribute.Node.value = "14.0"
        Write-Output 'Setting ToolsVersion to 14.0'
    }

    $proj.Save($CsProjFilePath)
 }

Till next time, happy NuGetting!

A while ago I wrote a post about speeding up your views with precompilation.

After writing that post I learned about RazorGenerator which Chris van de Steeg blogged about. RazorGenerator is better in just about every way compared to aspnet_compiler.exe. Whereas the aspnet_compiler just warms the compiler cache for the current machine, RazorGenerator builds the generated classes for your Razor files directly into your assemblies.

With RazorGenerator you can compile once and deploy everywhere. Getting started with it is almost stupidly simple. Seriously, I described the whole process in one tweet: Install the RazorGenerator.MsBuild NuGet package. That’s it. So what does that get you?

• Compile-time view syntax checking. Ever deployed a Razor file that broke at runtime after a “successful” build? Well now your builds will fail if your Razor syntax is invalid. Hoorah!
• Your Razor views are compiled into your output assembly (look at it in DotPeek and see the ASP namespace)
• It’s fast. You probably won’t even notice the build time difference.

But that’s not quite all

Right, so how do you get your precompiled views to actually get loaded? ASP.NET MVC doesn’t understand how to resolve classes instead of files out of the box. With Sitecore 8.1 and earlier, you could use the RazorGenerator.Mvc NuGet package and wire up its PrecompiledViewEngine to accomplish this.

Sitecore 8.2 modernizes a lot of the MVC implementation under the covers, and part of that is that the Sitecore infrastructure now supports precompiled views natively! (Wondering why the experience editor starts up a lot faster in 8.2? That’s why.) It’s really easy to use, in fact: all you have to do is tell Sitecore which assemblies to look for precompiled views in and it does the rest. Behold:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <mvc>
      <precompilation>
        <assemblies>
          <assemblyIdentity name="My.Project" />
          <assemblyIdentity name="My.OtherProject" />
        </assemblies>
      </precompilation>
    </mvc>
  </sitecore>
</configuration>

CAVEAT: The Sitecore precompiler is greedy by default. When it is on, if a class exists for your view, the physical file for the view is not checked. You can even delete it. This means that you cannot tweak the cshtml file and have updates be reflected. You can enable timestamp checking if you must by changing this setting:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!--  MVC: Flag for controlling whether Razor View Engine will look at physical view last modified dates 
        when determining to use a view file from a pre-compiled assembly or from the file system.
        Default: "false"
        -->
        <setting name="Mvc.UsePhysicalViewsIfNewer" value="false" />
      </settings>
  </sitecore>
</configuration>

Because there are (minor) performance implications of doing the timestamp checks, I would advise enabling this setting for development and disabling the setting for production. You should never, ever be changing files on production after a deployment anyway.

Recap

If you’re on Sitecore 8.2, you can get view performance gains by:

• Installing RazorGenerator.MsBuild on each Visual Studio project that contains Razor views
• Registering each of those assemblies with the Sitecore precompiled view engine so that it uses your precompiled classes for view rendering

Big thanks to Kern for setting up the MVC expert panel, without which this feature might well not exist. And we might still have 1:40 startup times after a compile. :)