Unicorn 4 Released

unicorn

I’m happy to announce the final release of Unicorn 4.0! Unicorn 4 comes with significant performance and developer experience improvements, along with bug fixes. Unicorn 4 is available from NuGet or GitHub.

Project Dilithium and Performance

Unicorn 4 is faster - a lot faster. Check out these benchmarks:

performance benchmarks

The speed increase is due to optimized caching routines, as well as the Dilithium batch processors. Dilithium is an optional feature that is off by default: because of its newness, it’s still experimental. I’m using it in production though. Give it a try - it can always be turned off without hurting anything.

For more detail into how Unicorn 4 is faster, and what Dilithium does, check out this detailed blog post.

Improved Modular Configuration

Unicorn 4 features a refactored configuration system that is designed to support Sitecore Helix projects with an improved configuration experience. The new config system is completely backwards-compatible, but now enables configuration inheritance, configuration variables, and configuration extension so that modular projects can encode their conventions (e.g. paths to include, physical paths) into one base config and all the module configs can extend it.

This drastically reduces the verbosity of the module configurations, and improves their maintainability by allowing conventions to be DRY. Here’s a very simple example of a base conventions configuration:

1
2
3
4
5
6
<configuration name="Habitat.Feature.Base" abstract="true">
<targetDataStore physicalRootPath="$(sourceFolder)\$(layer)\$(module)\serialization" />
<predicate>
<include name="$(layer).$(module).Templates" database="master" path="/sitecore/templates/$(layer)/$(module)" />
</predicate>
</configuration>

And here’s a module configuration that extends it:

1
2
3
4
5
6
<configuration 
name="Feature.News"
extends="Habitat.Feature.Base">
<!-- automatically stores items at $(sourceFolder)\Feature\News\serialization -->
<!-- automatically includes /sitecore/templates/Feature/News -->
</configuration>

There’s a lot more that you can do with the configuration enhancements in Unicorn 4 too. For additional details, read this extensive blog post.

Sitecore PowerShell Extensions Support

unicorn + spe

Just about anything you can do with Unicorn can now be automated using Sitecore PowerShell Extensions in Unicorn 4. You can now run Unicorn SPE cmdlets to…

Console Output Scaling

The Unicorn console has received a serious upgrade in Unicorn 4. If you’ve ever run a sync that changed a large number of items from the Unicorn Control Panel, you may have noticed the browser slow to a crawl and the sync seem to almost stop. The console that underpins Unicorn 3 and earlier started to choke at around 500 lines.

No longer! Unicorn 4’s upgraded console has spit out 100,000 lines without a hitch, and it should scale beyond that.

The automated tool console (PowerShell API) has also received an upgrade. Previously the tool console buffered all the output of a sync before sending it back. This caused problems in certain environments, namely Azure, where TCP connections that don’t send any data for more than four minutes are terminated. This would cause long-running syncs in Azure to die unexpectedly.

In Unicorn 4 the automated tool console emits data in a stream just like the control panel console. There’s also a heartbeat timer where if no new console entries are made for 30 seconds, a . will be sent to make sure the connection is kept active.

The streaming tool console also requires updating your Unicorn.psm1 file - not only will you get defense against TCP timeouts, you’ll also be able to see the sync occur in real time using the PSAPI just like you would from the control panel. No more waiting until it’s done to see how things are going :)

Predicates can Exclude by Template ID or Name

Unicorn 4 can now exclude items from a configuration by template ID, thanks to Alan Płócieniak. See also Alan’s original post on the technique.

1
2
3
<include name="Template ID" database="master" path="/sitecore/allowed">
<exclude templateId="{3B4F2B85-778D-44F3-9B2D-BEFF1F3575E6}" />
</include>

You can also exclude items by a regular expression of their name. This enables scenarios such as wanting to include all templates, but exclude all __Standard values items.

1
2
3
<include name="Name pattern" database="master" path="/sitecore/namepattern">
<exclude namePattern="^__Standard values$" />
</include>

The complete grammar for predicates is always in the predicate test config.

Breaking Changes

Unicorn 4’s breaking changes do not break any common use-cases of Unicorn, but review them to see if they affect you.

  • The __Originator field is now serialized by default. This enables proper tracking of the origin of items instantiated from branch templates.
  • Multithreaded sync support has been removed due to Sitecore bugs preventing realistic use. This was disabled by default already. Dilithium is faster than multithread sync ever was.
  • The Rainbow UseLegacyAttributeFormatting (formats items in Unicorn < 3.1 format) setting has been removed. The new format is now always used. This has always been off by default.
  • Rainbow FieldComparers are no longer activated using the Sitecore Factory, so they only support parameterless constructors (this would only affect custom comparers; the stock ones have always been parameterless)
  • Due to the console upgrades, a new Unicorn.psm1 is required if you are using Unicorn’s PowerShell API. This file also now ships in the NuGet package, so you can be sure you’re getting the right version for your Unicorn.

Bug Fixes

  • Transparent sync misc fixes (to renaming, saving display names, instantiating branch templates into TpSync areas)
  • Renaming an item and changing fields on it in one operation (only possible with Sitecore API) now no longer loses the additional field data in the serialized item
  • Improved output and logging, clarified messaging, improved developer experience
  • Content editor warnings now handle items in more than one configuration correctly
  • Control panel now displays which paths are invalid when initial serialization is blocked due to invalid include paths
  • The required password length for user serialization is now configurable, should you really really want to use b
  • Using Fast Query while any Transparent Sync configuration is active will now log a warning to the Sitecore logs (fast query cannot find transparent sync items, so items may not be returned as expected). This can be disabled if your logs start to fill with spam and you understand the issue.
  • PowerShell API now defaults to not write secrets to the console (debug is off by default) for secure-by-default-ness
  • Fixed a background exception that could occur when modifying serialized items on disk in rare cases, which could cause the app pool to terminate #222
  • The Rainbow data cache now correctly invalidates if an item is moved or renamed on disk after being added to the cache
  • Choosing many configurations to sync will no longer push the sync button off the page #232
  • The default console logging level for interactive syncing has been changed from Info to Debug, since there is no longer a scaling issue with the console output. This provides better insights into what has been changed on items without needing to see the Sitecore logs

Upgrading

If you’re coming from classical Unicorn 3.1 or later, upgrading is actually really simple: just upgrade your NuGet package. Unicorn 4 changes nothing about storage or formatting (except that the __Originator field is no longer ignored by default), so all existing serialized items are compatible.

If you’re invoking Unicorn via its remote PowerShell API, make sure to upgrade your Unicorn.psm1 to the Unicorn 4 version to ensure correct error handling with the streaming console.

Thanks

Thank you to the community members who contributed code and bug reports to this release.

Address

Unicorn 4 Part III: Configuration Enhancements

TL;DR: Unicorn 4 prerelease is on NuGet right now!

Now that that’s out of the way, let’s talk about another new Unicorn 4 feature: modular architecture friendly configurations.

When Habitat first launched, I was mildly incredulous at the amount of duplication in its Unicorn configurations. Tons of tiny modules, all of which shared similar but not identical configurations (such as custom root folders) was not really a consideration when multiple configurations were originally conceived. Fast forward to today, and that’s a major use case that is more difficult than it needs to be.

Here’s an example of a Habitat Unicorn configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
<sitecore>
<unicorn>
<configurations>
<configuration
name="Feature.News"
description="Feature News"
dependencies="Foundation.Serialization,Foundation.Indexing"
patch:after="configuration[@name='Foundation.Serialization']">
<targetDataStore physicalRootPath="$(sourceFolder)\feature\news\serialization"
type="Rainbow.Storage.SerializationFileSystemDataStore, Rainbow" useDataCache="false"
singleInstance="true" />
<predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
<include name="Feature.News.Templates" database="master" path="/sitecore/templates/Feature/News" />
<include name="Feature.News.Renderings" database="master" path="/sitecore/layout/renderings/Feature/News" />
<include name="Feature.News.Media" database="master" path="/sitecore/media library/Feature/News" />
</predicate>
<roleDataStore type="Unicorn.Roles.Data.FilesystemRoleDataStore, Unicorn.Roles" physicalRootPath="$(sourceFolder)\feature\news\serialization\Feature.News.Roles" singleInstance="true"/>
<rolePredicate type="Unicorn.Roles.RolePredicates.ConfigurationRolePredicate, Unicorn.Roles" singleInstance="true">
<include domain="modules" pattern="^Feature News .*$" />
</rolePredicate>
</configuration>
</configurations>
</unicorn>
</sitecore>
</configuration>

It’s long and it has a ton of boilerplate that is either identical in every module, or else defined by system conventions (e.g. physicalRootPath). We don’t need to be that verbose when using Unicorn 4. When we setup a modular, convention-based system using Unicorn 4 we can start by using abstract configurations to define the conventions of our system:

1
2
3
4
5
6
7
8
9
10
11
12
<configuration name="Habitat.Feature.Base" abstract="true">
<targetDataStore physicalRootPath="$(sourceFolder)\$(layer)\$(module)\serialization" />
<predicate>
<include name="$(layer).$(module).Templates" database="master" path="/sitecore/templates/$(layer)/$(module)" />
<include name="$(layer).$(module).Renderings" database="master" path="/sitecore/layout/renderings/$(layer).$(module)" />
<include name="$(layer).$(module).Media" database="master" path="/sitecore/media library/$(layer).$(module)" />
</predicate>
<roleDataStore type="Unicorn.Roles.Data.FilesystemRoleDataStore, Unicorn.Roles" physicalRootPath="$(sourceFolder)\$(layer)\$(module)\serialization\$(layer).$(module).Roles" singleInstance="true"/>
<rolePredicate type="Unicorn.Roles.RolePredicates.ConfigurationRolePredicate, Unicorn.Roles" singleInstance="true">
<include domain="modules" pattern="^$(layer) $(module) .*$" />
</rolePredicate>
</configuration>

This configuration defines a configuration that other configurations can extend. Because of its abstract-ness it is not a Unicorn configuration itself, only a template. Non-abstract configurations may also be extended.

This abstract configuration is also making use of Unicorn 4’s ability to do variable replacement in configurations. The $(layer) and $(module) variables are expanded in the extending configuration and are based on the convention of naming your configurations Layer.Module. You can also expand more than one config per module and use your own variables. Using our abstract Habitat.Feature.Base configuration above, the same Feature.News configuration we started with can now be expressed much more simply:

1
2
3
4
5
6
<configuration 
name="Feature.News"
description="Feature News"
dependencies="Foundation.Serialization,Foundation.Indexing"
extends="Habitat.Feature.Base">
</configuration>

Nice huh? But what if you want to extend or replace a dependency in the inherited configuration? You can do that, too - and using Unicorn 4’s element inheritance system you can also do it very cleanly. Unicorn configurations have always been architecturally a set of independent IoC containers. The <defaults> node in Unicorn.config sets up the defaults, and then each configuration’s nodes override and replace the defaults if they exist. This is how you can deploy only new items with the NewItemsOnlyEvaluator - you’re replacing the default evaluator with a different dependency implementation.

Unicorn 4 takes this a step further: with config inheritance, dependencies can be partially extended at an element level. You might have noticed this already in the Habitat.Feature.Base configuration, when we did this:

1
<targetDataStore physicalRootPath="$(sourceFolder)\$(layer)\$(module)\serialization" />

In Unicorn 3, this would have required a type attribute. In Unicorn 4, unless you specify a type attribute, any attributes you add either replace or add to the default (or inherited) implementation. So instead this kept the same default dependency definition and changed an attribute on it - the physicalRootPath.

If you do specify a type, nothing is inherited and it works like Unicorn 3. Thus existing configurations will also work without modification :)

But what about things that have more than just attributes, like the predicate‘s include nodes? You can append elements in the inherited configuration in that case. If we take our Habitat.Feature.Base configuration above and extend it like this:

1
2
3
<predicate>
<include name="Foo" database="master" path="/sitecore/Foo" />
</predicate>

The end result is effectively:

1
2
3
4
5
6
<predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
<include name="Feature.News.Templates" database="master" path="/sitecore/templates/Feature/News" />
<include name="Feature.News.Renderings" database="master" path="/sitecore/layout/renderings/Feature/News" />
<include name="Feature.News.Media" database="master" path="/sitecore/media library/Feature/News" />
<include name="Foo" database="master" path="/sitecore/Foo" />
</predicate>

You cannot remove inherited predicate nodes (or other dependencies that use children like fieldFilter), so plan accordingly: adding elements only.

And there you have it: with Unicorn 4 you can reasonably simply create serialization conventions for your modules and avoid configuration duplication - or if you’re not ready to go modular, you can at least enjoy not needing to have a type on most configuration nodes.

But Wait, There’s More 🐘: The Console No Longer Sucks

In the Control Panel…

The Unicorn console has also received a serious upgrade in Unicorn 4. If you’ve ever run a sync that changed a large number of items from the Unicorn Control Panel, you may have noticed the browser slow to a crawl and the sync seem to almost stop. The console that underpins Unicorn 3 and earlier started to choke at around 500 lines.

No longer! Unicorn 4’s console has spit out 100,000 lines without a hitch.

Automated Tools (PowerShell API)

The automated tool console has also received an upgrade. Previously the tool console buffered all the output of a sync before sending it back. This caused problems in certain environments, namely Azure, where TCP connections that don’t send any data for more than 4 minutes are terminated. This would cause any long-running syncs in Azure to die unexpectedly.

In Unicorn 4 the automated tool console emits data in a stream just like the control panel console. There’s also a heartbeat timer where if no new console entries are made for 30 seconds, a . will be sent to make sure the connection is kept active.

The streaming console also requires updating your Unicorn.psm1 file - not only will you get defense against TCP timeouts, you’ll also be able to see the sync occur in real-time using the PSAPI just like you would from the control panel. No more waiting until it’s done to see how things are going :)

Can I have it yet?

Absolutely. You can find Unicorn 4.0.0-pre03 on NuGet right now!

How stable is this?

More stable than you might think. Unicorn 4 is largely additions, fixes, and enhancements to the already stable codebase behind Unicorn 3. The core pieces have not changed very much, unless you enable Dilithium and that’s optional. The new config inheritance stuff has 97% code coverage. That’s not to say it’s bug free either. If you find bugs let me know and I’ll fix them :)

What about installing it?

Installation is just like Unicorn 3: Install the Unicorn NuGet package, and follow the directions in the README that will launch on installation to set up configuration(s).

NOTE: Dilithium ships disabled by default. If you want to enable it, make a copy of Unicorn.Dilithium.config.example and enable it.

What about upgrading to it?

If you’re coming from classical Unicorn 3.1 or later, upgrading is actually really simple: just upgrade your NuGet package. Unicorn 4 changes nothing about storage or formatting (except that the __Originator field is no longer ignored by default), so all existing serialized items are compatible.

Taking advantage of the config enhancements detailed above is also entirely optional: Unicorn 3 configurations are totally readable by Unicorn 4.

If you’re invoking Unicorn via its PowerShell API, make sure to upgrade your Unicorn.psm1 to the Unicorn 4 version to ensure correct error handling with the streaming console.

Have fun!

Unicorn 4 Preview Part 2.5: Generating Unicorn Packages with SPE

Last time we talked about how Sitecore PowerShell Extensions support was coming to Unicorn 4. This time, we’ve got a new cmdlet to share.

Over time, many people have asked if there was a way to generate Sitecore packages from Unicorn. The answer has always been no, for many good reasons: packages install slowly, cannot ignore specific fields, or process advanced exclusions like a Unicorn predicate can. This makes them much less safe (and much slower) for deployment purposes compared to a remotely invoked Sync using deployed serialized items.

But there is a great use case for generating packages from Unicorn: authoring modules. As a module author, a method is needed to track the items that belong to your module and also to reliably create Sitecore packages for distribution of your module which contain those items. Unicorn is a natural fit for simply tracking module items, but it has lacked the ability to automatically push updates to release packages like it can to serialized items. This unnecessarily complicates things and reduces release reliability. That’s bad.

So when Michael West and Adam Najmanowicz, the authors of Sitecore PowerShell Extensions, asked if there was a way we could export Unicorn configurations to packages my answer was absolutely.

SPE has long had packaging support built into it, and in fact SPE’s release packages are built using SPE. Unicorn packaging support is also implemented through SPE, and here’s how it works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Create a new Sitecore Package (SPE cmdlet)
$pkg = New-Package

# Get the Unicorn Configuration(s) we want to package
$configs = Get-UnicornConfiguration "Foundation.*"

# Pipe the configs into New-UnicornItemSource
# to process them and add them to the package project
# (without -Project, this would emit the source object(s)
# which can be manually added with $pkg.Sources.Add())
$configs | New-UnicornItemSource -Project $pkg

# Export the package to a zip file on disk
Export-Package -Project $pkg -Path "C:\foo.zip"

And when you’re done with that, c:\foo.zip would contain a package that when installed will contain the entire contents of any Unicorn configuration matching Foundation.*.

New-UnicornItemSource also accepts parameters to specify package installation options, exactly like SPE’s New-ExplicitItemSource. This cmdlet is also very similar to how New-UnicornItemSource works: each item that is included in the configuration is added to the package as an explicit item source. Doing this also means that the exported package completely respects the Unicorn Predicate, including exclusions of child paths (note that if you specify -InstallMode Overwrite, excluded children may be deleted by the package).

Questions?

Are the packaged items pulled directly from the serialized items?

No, they are pulled from the Sitecore database because the Sitecore packaging APIs work in Items. So make sure to sync before you generate a package. Unless you’re using Transparent Sync in which case the items will already be up to date.

Should I use this to deploy my site?

No. As mentioned above, packages are a slower and more dangerous method to deploy item updates to your site.

Does this mean modules will start requiring Unicorn? 🐘

No. Unicorn would only be used in the development of the module, and the build process used to generate plain old Sitecore Packages for module releases. The module itself would need depend on neither Unicorn, Rainbow, or SPE.

Unicorn 4 Preview, Part 2: SPE Support

Unicorn 4 will feature full support for Sitecore PowerShell Extensions (SPE) to perform Unicorn actions. If you’ve ever wanted deep programmatic control over Unicorn (for example “I want to sync Foundation.*”), or if you’ve got an existing deployment process that’s already using SPE Remoting to perform deployment tasks - this is for you.

To use Unicorn cmdlets in SPE, all that is necessary is to install the SPE package along with Unicorn. Unicorn 4 comes with configuration that remains quiescent until SPE is installed that will automatically enable the Unicorn cmdlets. In case that’s not clear enough: SPE is an optional addition and will not be required to use Unicorn 4.

So what can we do with Unicorn cmdlets for SPE?

Configurations

1
2
3
4
5
6
7
8
# Get one
Get-UnicornConfiguration "Foundation.Foo"

# Get by filter
Get-UnicornConfiguration "Foundation.*"

# Get all
Get-UnicornConfiguration

The result of Get-UnicornConfiguration is an array of IConfiguration objects, which you can spelunk (e.g. with their Name property) or pass to other cmdlets. Configurations are read only.

Syncing

Sync cmdlets make use of Write-Progress to provide a similar progress bar experience to the Control Panel, albeit a bit less responsive.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Sync one
Sync-UnicornConfiguration "Foundation.Foo"

# Sync multiple by name
Sync-UnicornConfiguration @("Foundation.Foo", "Foundation.Bar")

# Sync multiple from pipeline
Get-UnicornConfiguration "Foundation.*" | Sync-UnicornConfiguration

# Sync all, except transparent sync-enabled configurations
Get-UnicornConfiguration | Sync-UnicornConfiguration -SkipTransparent

# Optionally set log output level (Debug, Info, Warn, Error)
Sync-UnicornConfiguration -LogLevel Warn

For example:

Partial Syncing

Sometimes you want to only sync a portion of a configuration. You can do that with PowerShell using Sync-UnicornItem.

1
2
3
4
5
6
7
8
# Sync a single item (note: must be under Unicorn control)
Get-Item "/sitecore/content" | Sync-UnicornItem

# Sync multiple single items (note: all must be under Unicorn control)
Get-ChildItem "/sitecore/content" | Sync-UnicornItem

# Sync an entire item tree, show only warnings and errors
Get-Item "/sitecore/content" | Sync-UnicornItem -Recurse -LogLevel Warn

Reserializing

The cmdlet to reserialize is called Export-UnicornConfiguration because Reserialize is not an approved verb for a cmdlet :)

1
2
3
4
5
6
7
8
# Reserialize one
Export-UnicornConfiguration "Foundation.Foo"

# Reserialize multiple by name
Export-UnicornConfiguration @("Foundation.Foo", "Foundation.Bar")

# Reserialize from pipeline
Get-UnicornConfiguration "Foundation.*" | Export-UnicornConfiguration

Partial Reserializing

Sometimes you want to only reserialize a portion of a configuration. You can do that with PowerShell using Export-UnicornItem.

1
2
3
4
5
6
7
8
# Reserialize a single item (note: must be under Unicorn control)
Get-Item "/sitecore/content" | Export-UnicornItem

# Reserialize multiple single items (note: all must be under Unicorn control)
Get-ChildItem "/sitecore/content" | Export-UnicornItem

# Reserialize an entire item tree
Get-Item "/sitecore/content" | Export-UnicornItem -Recurse

Converting to Raw YAML

You can also dump out the raw YAML for an item - or items. The output of ConvertTo-RainbowYaml is either a string or array of strings depending on how many items were passed to it. Note that unless -Raw is specified, the default field formatters and excluded fields Unicorn ships with are used. These are non-customizable and do not follow Unicorn defaults if changed.

This capability enables casual use of YAML serialization without having to use Unicorn or set up a configuration. It’s not a good solution for general purpose synchronization though simply because the nuances of storing trees of items in files are many. Very many. But I’m curious what uses people will find for this :)

1
2
3
4
5
6
7
8
9
10
# Convert an item to YAML format (always uses default excludes and field formatters)
Get-Item "/sitecore/content" | ConvertTo-RainbowYaml

# Convert many items to YAML strings
Get-ChildItem "/sitecore/content" | ConvertTo-RainbowYaml

# Disable all field formats and field filtering
# (e.g. disable XML pretty printing,
# and don't ignore the Revision and Modified fields, etc)
Get-Item "/sitecore/content" | ConvertTo-RainbowYaml -Raw

Converting from Raw YAML

In Rainbow the IItemData interface is the internal unit of an Item. The ConvertFrom-RainbowYaml cmdlet converts raw YAML string(s) into IItemData which you can then spelunk as objects or deserialize as needed.

1
2
3
4
5
6
# Get IItemDatas from YAML variable
$rawYaml | ConvertFrom-RainbowYaml

# Get IItemData and disable all field filters
# (use this if you ran ConvertTo-RainbowYaml with -Raw)
$yaml | ConvertFrom-RainbowYaml -Raw

Deserialization

To deserialize items, use Import-RainbowItem which takes IItemData items in and deserializes them into the Sitecore database. No comparison is done before deserialization, which makes this a bit slower than a full Unicorn sync.

As a shorthand Import-RainbowItem also accepts YAML strings, however as IItemData can represent any sort of item, it is not limited only to deserializing YAML-sourced items.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Deserialize IItemDatas from ConvertFrom-RainbowYaml
$rawYaml | ConvertFrom-RainbowYaml | Import-RainbowItem

# Deserialize raw YAML from pipeline into Sitecore
# Shortcut bypassing ConvertFrom-RainbowYaml
$yaml | Import-RainbowItem

# Deserialize and disable all field filters
# (use this if you ran ConvertTo-RainbowYaml with -Raw)
$yaml | Import-RainbowItem -Raw

# Deserialize multiple at once
$yamlStringArray | Import-RainbowItem

# Complete example that does nothing but eat CPU
Get-ChildItem "/sitecore/content" | ConvertTo-RainbowYaml | Import-RainbowItem

Questions?

Does this mean the existing PowerShell Remote API is obsolete?

No. The existing PowerShell API uses Windows PowerShell to provide remote syncing capability and does not require installing Sitecore PowerShell. They serve different parallel purposes, and both are here to stay.

You have no gifs or memes in this post. Is something wrong?

you mad?

Can I have a beta yet?

I’ll be releasing a beta once I finish the features I have planned. Yep, there’s at least one more ;)

Unicorn 4 Preview, Part 1: Project Dilithium

Not too long ago I was considering the future of Unicorn. Rather naïvely, I couldn’t think of all that much that needed fixing. I wondered if there would ever be a Unicorn 4.

Now since you’ve read the title of this blog post, something tells me you don’t think that attitude lasted very long. It didn’t. Because there’s a pretty simple truth about Unicorn, even with all the optimizations I put in to Unicorn 3:

There were two primary causes of this:

  1. The Sitecore API is slow and not suited to dealing with large quantities of items
  2. Unicorn was originally designed to optimize for a few configurations, and Helix was blasting the number of active configurations to large numbers

I found that the sync time was becoming annoyingly long. Long enough that it was a distraction, long enough that a piddly 20% faster from more profiling just wasn’t good enough.

What to do, what to do.

Introducing Project Dilithium

I started on a crazy quest to speed up syncing performance. Sitecore’s recently added Publishing Service gained drastic speed gains in publishing by leveraging direct database access. Publishing, like syncing, is a batch-oriented operation. The Sitecore API is a single-item-oriented API. It does no optimizations, aside from caching, when you want more than one item. I resolved to try out SQL and see how it went. It went. See for yourself:

(benchmarks on i5-3570k@4GHz, SSDs, freshly restarted IIS + SQL, average of 3 runs each)

So what is Dilithium?

There are two pieces of the Dilithium project: SQL and Serialized. Both operate in a similar fashion: before a sync operation begins, all items that will be involved in that sync are read in a single big batch either from SQL or disk. The actual sync then occurs against the pre-batched items. In case you’re wondering, the graph above includes the batching times for Dilithium - no cheating :)

The SQL version takes advantage of the Descendants table to grab all predicate-included items for every configuration in a single huge SQL query (this does mean that FastQueryDescendantsDisabled must be turned off). These items are then indexed and prepared for fast access. In this way, the entire data contents of the stock core database can be read in about 1500ms, compared to the Sitecore API’s 8 seconds.

The serialized version skips all the checks that usually are needed to traverse a Serialization File System tree - expensive operations that guarantee that the children you’re getting all belong to the same item ID, for example. It’s essentially reading all files sequentially and caching them for sync just like the SQL version.

When both DiSerialized and DiSQL are used together, they enable parallel loading of both sources at once. This reduces the load time quite significantly, as you can see on the shortest bar on the chart above.

Questions?

Direct SQL, are you nuts?

Yes. Yes I am. Directly querying the Sitecore database is a terrible idea in almost every situation…except this. The schema for content has also been very stable for several major revisions of the product. And Stephen Pope said it was ok :)

Is it any faster adding or modifying items?

No. I’m not crazy enough to also do direct SQL writes, so all modifications go through normal Sitecore item saving APIs and perform pretty much the same as Unicorn 3. This also means that they’re mature, well-tested code - Unicorn 4 under the hood is not much different from v3, except for Dilithium.

Can I turn it off?

Heck yes. You can enable or disable Dilithium on a per-configuration basis. It also works just fine if you sync Dilithium and standard configurations together.

Serialized items and formats are identical between Dilithium and normal, so you can also enable and disable at will with no migration process.

Will this eat up all my RAM?

If you’re syncing gigabytes of media items…yes, definitely. For more normal developer and light content items, nope. DiSQL does not read blob data, so if you become RAM constrained turn off DiSerialized first.

Note that the precaches are disposed of immediately after the sync completes, so memory spikes are temporary.

Let’s address the 🐘 in the room, how is Unicorn 4 60% faster than 3 without Dilithium?

Actually pretty simple: eliminating a cache bombing that was done between syncing configurations that was needed for Transparent Sync. Well, it wasn’t needed if you’re syncing a non-Transparent configuration. Doing that meant that sites like Habitat with a ton of small non-Transparent configurations perform a lot faster.

Any other performance tricks?

In my daily development I use a mix of Transparent Sync for developer items (templates, renderings), and standard sync for content items. Since Transparent Sync is disk-based anyway there is little point to running a sync on Transparent Sync configurations when you’re just doing local development. As such, there’s now an option to skip Transparent Sync-enabled configurations when you sync all. This is an option in the PowerShell API in Unicorn 4, as well as an option in the control panel (same place as log level).

Depending on how many Transparent Sync configurations you have, your time savings can be significant from enabling this. You should not however do this when deploying transparent configurations to a shared server.

Can I have it?

Not yet, it still needs some additional testing and vetting before I call it alpha. If you’re interested in being a brave early tester, get ahold of me on Sitecore Slack :)

Is this the only new thing in Unicorn 4?

Nope. I’ve got some more tricks up my sleeve, so stay tuned for some more preview blogs coming soon™

Unicorn 3.3 Released

I’m happy to announce the release of Unicorn 3.3 and Rainbow 1.4. This is primarily a maintenance and bug fix release and is recommended for all users of Unicorn 3.x.

As there was no official announcement for Unicorn 3.2, this will also note what was new in 3.2. Items from 3.2 will be noted as such.

What’s new?

Improved Configuration Dependencies

Dependencies between configurations have been improved to include implicit dependencies that are created when one configuration includes a path that is a child of another configuration. This simplifies the need to define explicit dependencies in many cases. This config file documents how this works and how to disable it if you need to. #165

Improved Debugging with PowerShell API

You can now enable diagnostic logging of the CHAP base signature on the client and server side, so if you have issues getting the tool authentication working you can diagnose why. See this and this for configuration information. #182

Full Sitecore 8.2 Compatibility

Unicorn and Rainbow now build against and require .NET 4.5.2 for Sitecore 8.2 compatibility. This means that your project referencing Unicorn must also target framework 4.5.2 or later, even if not using Sitecore 8.2. Unicorn should work with Sitecore 7 and later. #175

(3.2) Improved Exclusion Grammar

You can now exclude children of a child of an included path. #138

(3.2) User and Role Serialization

Unicorn 3.2 introduced YAML-based user and role serialization via the Unicorn.Users and Unicorn.Roles NuGet packages. User and role serialization works similarly to item serialization. To add this capability to your Unicorn installation you simply install the correct NuGet package and the readme will guide you, or you can read the documentation using the links below.

Bug fixes

  • When deserializing an item that contains a field missing in Sitecore, the field hint is shown instead of just its ID. This aids in tracking down the offending field, since it does not exist to find in the DB. #169
  • The Unicorn.SharedSecret.config.example has been renamed to include a z so that simply renaming it to use does not cause a config load order issue #181
  • Fixed an issue with null vs empty field values that could cause an exception in Rainbow #16
  • Fixed an issue that would cause missing language data when you removed a version from a tracked item using the Sitecore API within an EventDisabler block #168
  • Fixed an error that occurred when an item changed template, but the old template was already deleted as part of the same sync process and thus no longer existed #164
  • An issue that could cause spurious “Item changed, do you want to overwrite?” popups when saving an item in Experience Editor and using Transparent Sync has been fixed #163
  • Fixed an issue that could cause a spurious Unicorn conflict warning when saving items with checkboxes in certain cases #151
  • Serialized items with field values that equaled the standard values defined in the database are no longer left as standard values, and are correctly explicitly set to their serialized value. This was a common cause of items that ‘synced every time’. #162
  • Moving an item between Unicorn configurations will correctly respect exclusions that exclude some children of the target location #161
  • Unicorn is now free of uses of EditContext, which is a 10-year-old antipattern #160
  • Pathing that escaped from the webroot using ~/ no longer breaks when set as the serialization path #157
  • Errors about being unable to send headers after they have already been sent in Sitecore 8.1 Update-3 and later have been resolved #155
  • Unicorn Auto Publish now works correctly if it is set to publish to more than one publishing target #152
  • Renaming items whose filenames have been truncated on disk (very long db item names) will no longer result in the creation of duplicate files #146
  • Items whose only change between db and disk is the BranchID are now synced correctly #144
  • YAML values containing \ and " are now correctly treated as multiline values so that serialized files are valid YAML. This is a non-breaking change, but you may notice values such as __Created by being altered as items are saved. All Unicorn 3.x versions can parse either format without issue. #143
  • You can now explicitly add a wildcard item (an item named *) to a Unicorn configuration without it being interpreted as a wildcard character. See test cases #142
  • The Unicorn data provider now returns null for ChildIDs if no children exist to enhance compatibility with the Sitecore data provider’s behaviour #141
  • A race condition when using multithreaded sync and auto publishing has been fixed #183
  • Rapidly clicking a checkbox in the control panel will no longer potentially corrupt the page state #158
  • Unicorn and Rainbow both now reference Sitecore from the Official NuGet Feed, so if you need to build from source copying Sitecore assemblies is no longer required. Just clone and build!
  • 3.2 Setting exclusions on a configuration that uses Transparent Sync is now an exception (this has never been a supported configuration, just now it’s an error too)
  • 3.2 Configuration predicates with no include nodes are now an error instead of including all items

Upgrading

To upgrade to Unicorn 3.3, you need to make sure the project hosting Unicorn and Rainbow is targeting .NET 4.5.2 or later. Then simply upgrade the NuGet package, if you have Unicorn 3.1 or later.

For earlier versions, follow the upgrade instructions on the appropriate launch blog post (e.g. for 3.0 or 3.1) before you upgrade to 3.3.

Installing

Install the Unicorn package from NuGet. A readme will be shown to help you get started after install. You must be using packages.config (NuGet 2.x style) package management for Unicorn and Rainbow, because they install content items to the project. If you wish to use project.json (NuGet 3.x style), you must manually install the configuration files for Unicorn and Rainbow because of limitations in NuGet 3.x.

Thanks

As usual Unicorn is not just me, it’s a community project. Thank you all for your pull requests, issue reports, and for using my work.

I’d like to specially thank Mark Cassidy for spending a lot of time tracking down - and fixing - some Unicorn issues he was having. All contributors to this release include:

Unicorn 3.1.5 Released

Unicorn 3.1.5 and Rainbow 1.3 are released to NuGet right now. This is a maintenance release. Upgrading is recommended, particularly if you are using Transparent Sync.

New Features

The new <syncConfiguration> dependency (see Unicorn.config) enables configurations to opt in to having synced items be updated into the link database and/or the search indexes after being changed by a sync.

This improves Unicorn’s compatibility with content syncing scenarios that are less developer focused, at the expense of reduced sync performance.

Transparent Sync also supports these flags, and items changed while Sitecore is alive to see them will have their links and indexes taken care of if set to do so. However if the IIS App Pool is not active during the file changes, nothing is updated.

Per-configuration concurrency

Unicorn 3 has always supported multithreaded sync and reserialize, but it has been disabled since release due to a concurrency bug in the Sitecore Template Engine that results in sync deadlocks when it is used to sync template item changes.

3.1.5 brings the ability to set the max thread count per-configuration (using the same <syncConfiguration> as indexing support) so that for configurations that are not syncing templates (or on Sitecore prior to 8.0U2) you may again enable concurrency. Threaded syncing can be 30-50% faster than non threaded.

Control panel UX improvements

Unicorn 3.1.5 brings a ‘Sync all’ button to the control panel, and a few minor UI tweaks such as relegating the log verbosity to an options modal.

Bug Fixes

  • Checkboxes inherited through several layers of standard values will no longer result in the derived template being reset to base standard values
  • Auto-publish no longer will miss publishing unversioned fields in certain cases
  • Renaming or moving items with transparent sync enabled and data cache turned on no longer results in a corrupted data cache
  • Change Template now works correctly with transparent sync
  • Unicorn now understands content items that are using Sitecore 8.1 language fallback without creating extra versions on the serialized copy
  • The log verbosity feature has had several issues fixed that would result in logs incorrectly not being shown at the selected level
  • Unicorn’s Micro IoC container now understands int constructor parameters
  • Adding or changing serialized template items when using transparent sync will now correctly clear the template engine, preventing an error
  • Automated build powershell examples are now more flexible when invoked in unusual path locations
  • In the control panel, the batch action buttons will no longer ‘overlay’ configuration checkboxes in certain situations
  • Logging of unversioned field updates will no longer incorrectly show a version number alongside the update in the logs
  • Certain unusual situations when syncing templates and items of that template will no longer throw a null reference due to stale caches

Upgrading

Upgrading from 3.1.4 is just a NuGet upgrade away. There are some config changes, so overwrite files and merge anything custom back in. Or even better use config patches and leave the default config alone :)

Note that Unicorn 3.1.5 changes the storage format for checkbox fields so that unchecked checkboxes are stored as ‘0’ instead of Sitecore’s standard blank value. This makes sync more reliable, however it does means that the first sync after serializing a new item with an unchecked checkbox will result in a ‘change’ as the zero value is pushed back into the Sitecore database. Checkboxes that are already serialized with blank values will continue to work just fine as they are.

Credits

Unicorn is a team effort of the Sitecore community. I’d like to thank the following folks who contributed to this release:

Dana Hanna
Kasper Gadensgaard
Mark Cassidy
Mike Carlisle
Søren Kruse
WizX20

And extra special thanks to Robin Hermanussen, Richard Seal, Alex Washtell, Mark Cassidy, and everyone else I missed for tirelessly answering questions in #unicorn on Sitecore Slack when I’m busy or not around!

Unicorn 3.1.4 Released

Unicorn 3.1.4 and Rainbow 1.2 are released to NuGet. This is a maintenance release which contains a few UX improvements and minor bug fixes and improvements.

What’s new?

Logging Verbosity

The most obvious change is that you can now select a logging verbosity in the control panel. If you’ve ever run a sync where many, many changes were synced you may have run into the issue where browsers start to crawl when fed that many log messages. Now you can choose to log only warnings or errors for that kind of situation. The Sitecore log files always get full verbosity, so you can see what exactly happened even if it was not written to the browser. Your verbosity setting is stored in a persistent cookie, so your selection is remembered.

Verbosity does not affect automated tools that invoke Unicorn. They always receive full verbosity. (#112)

XML Field Formatting Improved

XML fields are now stored by default with attributes on new lines. This further improves mergeability of Rainbow serialized items by allowing attribute merges. If you hate this idea, you can turn it off in Rainbow.config. (#12)

Rainbow 1.1:

<r xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <d id="{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}" l="{0DAC6578-BC11-4B41-960A-E95F21A78D1F}" />
</r>

Rainbow 1.2 (Unicorn 3.1.3+):

<r xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <d
    id="{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}"
    l="{0DAC6578-BC11-4B41-960A-E95F21A78D1F}" />
</r>

This formatting change is completely backwards compatible and no reserialization is required. As items are saved, they will be rewritten to use attributes on newlines.

Fixes & Minor Improvements

  • The UX of the control panel has been streamlined to result in less vertical space for each configuration, a boon for setups like Habitat with lots of configurations.
  • Unicorn auto publishing now functions correctly on items with versions in several languages (#114)
  • Using the “Copy to” function will no longer result in an incorrect serialized parent ID on the copied item. Other forms of duplication were not affected. (#119)
  • Fixed a case where NullReferenceException was incorrectly thrown when using transparent sync (#117)
  • Fixed an error when duplicating items that did not exist in the database when using transparent sync (#116)
  • Rainbow YAML formatting is more friendly to third party YAML parsers with the way it escapes values. This does mean that some values that did not previously get wrapped in double quotes (like GUIDs), will now have it going forward. Format is backwards compatible, no reserialize required. Items will upgrade to the new version on save. (#11)

Upgrading

To upgrade simply update your NuGet packages. There are no other steps required for this release unless coming from Unicorn 3.0.x.

If you wish to completely normalize your items to the latest Rainbow 1.2 format, you may reserialize them. But it is not a requirement, as Rainbow 1.2 can parse 1.1 serialized items.

Credits

As usual, Unicorn updates are a community effort. Big thanks to the folks below for reporting bugs, sending pull requests, and general encouragement.

Unicorn 3.1 Released

Unicorn 3.1 and Rainbow 1.1 are released to NuGet. These updated versions bring several new features and a bunch of fixes. Upgrading is recommended for all users of Unicorn 3.0 due to the fixing of some data issues.

What’s new?

Configuration Dependencies

Unicorn configurations may now declare dependency relationships between each other. This is useful for example when one configuration may contain templates and another configuration might contain items based on those templates. With a dependency relationship the dependent configurations are guaranteed to always go first when configurations are synced. Transitive relationships (e.g. A -> B -> C) are supported.

Dependencies are defined using a dependencies attribute on the configuration.

Dependencies do not force dependent configurations to sync. If config A depends on config B and you choose to sync only config A, B will not be synced. However if you sync both A and B, then B will always sync first due to the dependency.

Include/Exclude Predicate Enhancements

The predicate system has been significantly overhauled in Unicorn 3.1 to allow more advanced include and exclude scenarios.

  • Exclude entries may now use paths relative to the parent <include> by omitting a leading slash. This would exclude /foo/bar:

      <include path="/foo">
          <exclude path="bar" />
      </include>
    
  • A new explicit grammar for excluding all children of an include has been introduced. This replaces the previous implicit grammar of adding a trailing slash (which still works but is deprecated).

      <include path="/foo">
          <exclude children="true" />
      </include>
    
  • The explicit children grammar supports <except> syntax to exclude all children except for specific ones.

      <include path="/somechildren">
          <exclude children="true">
              <except name="tests" />
              <except name="fests" />
          </exclude>
      </include>
    
  • A new explicit grammar for excluding only children of a specific sub-path has been added. This grammar also supports <except> syntax and relative paths.

      <include path="/foo">
          <exclude childrenOfPath="/foo" />
          <exclude childrenOfPath="bar"> <!-- /foo/bar/* -->
              <except name="quux" /> <!-- include /foo/bar/quux -->
          </exclude>
      </include>
    

Take a look at the test predicate configuration for examples of all available grammars.

Control Panel Tool Authentication

The method used to authenticate automated tools to Unicorn has been updated to be significantly more secure using a CHAP implementation. This also comes with a handy PowerShell module that you can use to simply invoke the handshake from a build script.

Should you require the legacy shared secret method, that is still available but it is not the default. Security is also now pluggable, if you wanted to use your own implementation.

The remote API ships disabled by default and you must activate it by generating a random shared secret and adding it to the config. This same secret must be known to invoke the API, and unlike the legacy API the secret is never transmitted directly.

Breaking Changes

Unicorn 3.1 has some breaking changes compared to Unicorn 3.0, so upgrading requires reserializing your items to get them in a consistent format with the latest version. Unicorn 3.1 does understand Unicorn 3.0-formatted items, but Unicorn 3.0 does not understand Unicorn 3.1-formatted items. There will also be some repeated changes in the logs if you sync 3.0 items with 3.1, so usage with 3.0 items is not recommended.

Introducing breaking changes was a hard decision to make, but I think the saner defaults and new capabilities will justify the slightly more difficult upgrade process. So what’s breaking?

  • The item’s database name is now stored in the serialized data (#7). This enables parsing serialized items in contexts where the database cannot easily be inferred, such as Courier.
  • Unversioned fields have official support and are nested under the Language instead of under a Version within the language which was incorrect. Prior to reserializing, syncing a 3.0-formatted item with 3.1 may result in messages that the field has been reset to standard values and then set again. IItemData now has an UnversionedFields property to store unversioned fields by language.
  • The name on serialized fields has been converted from a comment in the YAML into an actual parsed value. Note that this value is just a hint and if a template field is renamed it is NOT updated automatically.
  • The default setting of Rainbow.SFS.ExtraInvalidFilenameCharacters has been changed to "$" to make Rainbow compatible by default with TFS, and the TFS config example that previously set this has been removed.
  • The default setting for Rainbow.SFS.SerializationFolderPathMaxLength has been increased from 90 to 110 based on real world experience that 90 is too short.
  • The default setting for Rainbow.SFS.MaxItemNameLengthBeforeTruncation has been reduced from 100 to 30 based on real world experience that 100 was long enough to break the path length limitation for items with near 100 length names.
  • Automated deployment tools calling Unicorn have a new security API. See below for details on this, but it’s now more secure.

Other Improvements:

  • Syncing dictionary items will now cause the dictionary cache to be cleared at the end of the sync, forcing the synced entries to be loaded.
  • The Unicorn Control Panel is now pipeline-based and extensible so that modules may define their own verbs or alter the UI if needed.
  • Error messages when serializing and deserializing XML-based fields (layout, rules) have been improved.
  • Unicorn and Rainbow both now utilize the C# 6.0 syntax and require the C# 6.0 compiler to build. They still target .NET 4.5.
  • Config file comments and documentation have been updated to cover the new features of Unicorn 3.1.

Bug fixes:

  • Renaming or moving items which are serialized on loopback paths will no longer cause the serialized children to be deleted.
  • Missing field errors on the root item of a tree now result in a retry and non fatal error as expected.
  • A non fatal warning when syncing multiple configurations will no longer result in the sync stopping at the end of the configuration with the non fatal warning.
  • When using the New Items Only evaluator, you will no longer receive sync conflict warnings when saving items created by the evaluator.
  • Copying items when Transparent Sync is enabled and the source item is not in the Sitecore database now works correctly.
  • Empty field values will no longer cause problems when Transparent Sync is enabled.
  • XML-based fields (layout, rules) with empty values will no longer cause exceptions. #6, #90
  • Items installed by Sitecore packages or update packages into Unicorn configuration areas will no longer lose some field values in the serialized items. #92
  • Predicate include entries and SFS trees are no longer confused by paths which are different but start with similar bases, such as /sitecore/content and /sitecore/content monkey
  • Plain text Unicorn logs returned to automated tool calls will no longer include HTML-encoded values such as &gt;
  • Naming items a Windows reserved file name such as CON or COM1 will no longer result in an error. #8
  • Fields with a blank value in Sitecore that were not included in the serialized item were not being correctly reset to standard values. This has been fixed.

Upgrading

Ok so there are breaking changes, how do I upgrade? Thankfully it’s not too hard.

  1. Before you upgrade, run a full sync of all configurations. If some have Transparent Sync enabled, turn that off during the upgrade. (you can reenable after step 4)
  2. Upgrade your NuGet packages to Unicorn 3.1. (which will also install Rainbow 1.1)
  3. Run a build of your solution, and if you develop out of webroot publish your changes.
  4. Using the Unicorn control panel, Reserialize all configurations to flush them to disk as 3.1-format items.
  5. If you use the automated tool API to do automated execution of Unicorn, migrate to the PowerShell module to invoke the new API.
  6. Have fun!

Contributors

Unicorn 3.1 has seen contributions from many community members. Thank you all for your support, testing, bug reports, and contributions!

Unicorn 3.0.2 Released

Unicorn 3.0.2 (and Rainbow 1.0.1) have been released to NuGet.

This release includes some critical fixes to the serialization system, and upgrading is highly recommended for all users of Unicorn 3.

Improvements

  • Unicorn auto-publish now reports its activities to the sync console, and operates synchronously to guarantee published state for automated builds.
  • Guidance on some Transparent Sync pitfalls has been added to the wiki.
  • The new Rainbow.SFS.ExtraInvalidFilenameCharacters setting enables TFS users to successfully serialize branch templates which begin in $, a reserved character by TFS. The Rainbow.TFS.config.example file is now shipped with Rainbow to illustrate its usage for TFS.
  • An example configuration file to demonstrate how to move your serialized items elsewhere has been added.

Fixes

  • CRITICAL Saving item versions that were not in the default Sitecore language was not updating the serialized item. This has been fixed. #76
  • CRITICAL Serialization of significant empty field values, such as unchecked checkboxes when the standard value is checked, has been fixed. #74
  • Items that are restored from the Sitecore recycle bin or archive to Unicorn-controlled locations now result in the item being serialized on restore. #72
  • Resetting a field to standard values now correctly results in the field value being removed from the serialized item. #74
  • Query string order is always respected when syncing multiple configurations at once using the control panel.
  • The serialization conflict dialog (if disk doesn’t match base item data on save) now formats correctly using HTML for Sitecore 8+
  • Using the Dump Item and Dump Tree commands in Sitecore to reserialize partial item trees now correctly serializes an item to all configurations that it may belong to, instead of just the first one.

Upgrading

Upgrading from Unicorn 3.x is as simple as updating your NuGet packages. If it prompts to overwrite any changed config files, I’d suggest doing so and using a diff with your source control to resolve differences.

If you suspect that you have serialized items that may be missing significant empty values, you should reserialize those after upgrading to get their values normalized.

The Credits

Unicorn isn’t just my project. Thanks a lot to all the community members who reported bugs, sent pull requests, and diagnosed issues to help make Unicorn better.

Unicorn 3.0.2 is brought to you by…

Robin Hermanussen
Thomas Eldblom
Nathanael Mann
@GeorgeKarbyn
Kevin Williams
@amitthakur2014
@WizX20
@kamsar
…and the rest of #unicorn on Sitecore Slack

As usual, if you run into trouble you can submit an issue on GitHub, hit me up on Sitecore Slack, or tweet at me.