Branch Datasource Presets

Ever stored rendering component data source items under a page? For example:

/sitecore/content/Page
/sitecore/content/Page/Datasources/ComponentDataSourceItem

It’s a good practice and it seems to work quite well for page specific components. I use it all the time.

But have you ever wished branch templates understood that pattern in a logical fashion? What if this:

/sitecore/templates/branches/Foo/$name
/sitecore/templates/branches/Foo/$name/Datasources/ComponentDataSourceItem

…expanded out to have the branch create the hierarchy and re-link the layout details on the instiantiated branch item to point to the right relative child data source item, instead of the child item under the branch template?

Doing this lets you use branch templates to create preset rendering hierarchies, including page specific data source items.

Sound good? Well you’ve found the right place to get the code to do just that. This requires Sitecore 8 or above with the pipeline-based item provider to operate.

Unicorn 3.1.4 Released

Unicorn 3.1.4 and Rainbow 1.2 are released to NuGet. This is a maintenance release which contains a few UX improvements and minor bug fixes and improvements.

What’s new?

Logging Verbosity

The most obvious change is that you can now select a logging verbosity in the control panel. If you’ve ever run a sync where many, many changes were synced you may have run into the issue where browsers start to crawl when fed that many log messages. Now you can choose to log only warnings or errors for that kind of situation. The Sitecore log files always get full verbosity, so you can see what exactly happened even if it was not written to the browser. Your verbosity setting is stored in a persistent cookie, so your selection is remembered.

Verbosity does not affect automated tools that invoke Unicorn. They always receive full verbosity. (#112)

XML Field Formatting Improved

XML fields are now stored by default with attributes on new lines. This further improves mergeability of Rainbow serialized items by allowing attribute merges. If you hate this idea, you can turn it off in Rainbow.config. (#12)

Rainbow 1.1:

<r xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <d id="{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}" l="{0DAC6578-BC11-4B41-960A-E95F21A78D1F}" />
</r>

Rainbow 1.2 (Unicorn 3.1.3+):

<r xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <d
    id="{FE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3}"
    l="{0DAC6578-BC11-4B41-960A-E95F21A78D1F}" />
</r>

This formatting change is completely backwards compatible and no reserialization is required. As items are saved, they will be rewritten to use attributes on newlines.

Fixes & Minor Improvements

  • The UX of the control panel has been streamlined to result in less vertical space for each configuration, a boon for setups like Habitat with lots of configurations.
  • Unicorn auto publishing now functions correctly on items with versions in several languages (#114)
  • Using the “Copy to” function will no longer result in an incorrect serialized parent ID on the copied item. Other forms of duplication were not affected. (#119)
  • Fixed a case where NullReferenceException was incorrectly thrown when using transparent sync (#117)
  • Fixed an error when duplicating items that did not exist in the database when using transparent sync (#116)
  • Rainbow YAML formatting is more friendly to third party YAML parsers with the way it escapes values. This does mean that some values that did not previously get wrapped in double quotes (like GUIDs), will now have it going forward. Format is backwards compatible, no reserialize required. Items will upgrade to the new version on save. (#11)

Upgrading

To upgrade simply update your NuGet packages. There are no other steps required for this release unless coming from Unicorn 3.0.x.

If you wish to completely normalize your items to the latest Rainbow 1.2 format, you may reserialize them. But it is not a requirement, as Rainbow 1.2 can parse 1.1 serialized items.

Credits

As usual, Unicorn updates are a community effort. Big thanks to the folks below for reporting bugs, sending pull requests, and general encouragement.

Unicorn 3.1 Released

Unicorn 3.1 and Rainbow 1.1 are released to NuGet. These updated versions bring several new features and a bunch of fixes. Upgrading is recommended for all users of Unicorn 3.0 due to the fixing of some data issues.

What’s new?

Configuration Dependencies

Unicorn configurations may now declare dependency relationships between each other. This is useful for example when one configuration may contain templates and another configuration might contain items based on those templates. With a dependency relationship the dependent configurations are guaranteed to always go first when configurations are synced. Transitive relationships (e.g. A -> B -> C) are supported.

Dependencies are defined using a dependencies attribute on the configuration.

Dependencies do not force dependent configurations to sync. If config A depends on config B and you choose to sync only config A, B will not be synced. However if you sync both A and B, then B will always sync first due to the dependency.

Include/Exclude Predicate Enhancements

The predicate system has been significantly overhauled in Unicorn 3.1 to allow more advanced include and exclude scenarios.

  • Exclude entries may now use paths relative to the parent <include> by omitting a leading slash. This would exclude /foo/bar:

    <include path="/foo">
        <exclude path="bar" />
    </include>
    
  • A new explicit grammar for excluding all children of an include has been introduced. This replaces the previous implicit grammar of adding a trailing slash (which still works but is deprecated).

    <include path="/foo">
        <exclude children="true" />
    </include>
    
  • The explicit children grammar supports <except> syntax to exclude all children except for specific ones.

    <include path="/somechildren">
        <exclude children="true">
            <except name="tests" />
            <except name="fests" />
        </exclude>
    </include>
    
  • A new explicit grammar for excluding only children of a specific sub-path has been added. This grammar also supports <except> syntax and relative paths.

    <include path="/foo">
        <exclude childrenOfPath="/foo" />
        <exclude childrenOfPath="bar"> <!-- /foo/bar/* -->
            <except name="quux" /> <!-- include /foo/bar/quux -->
        </exclude>
    </include>
    

Take a look at the test predicate configuration for examples of all available grammars.

Control Panel Tool Authentication

The method used to authenticate automated tools to Unicorn has been updated to be significantly more secure using a CHAP implementation. This also comes with a handy PowerShell module that you can use to simply invoke the handshake from a build script.

Should you require the legacy shared secret method, that is still available but it is not the default. Security is also now pluggable, if you wanted to use your own implementation.

The remote API ships disabled by default and you must activate it by generating a random shared secret and adding it to the config. This same secret must be known to invoke the API, and unlike the legacy API the secret is never transmitted directly.

Breaking Changes

Unicorn 3.1 has some breaking changes compared to Unicorn 3.0, so upgrading requires reserializing your items to get them in a consistent format with the latest version. Unicorn 3.1 does understand Unicorn 3.0-formatted items, but Unicorn 3.0 does not understand Unicorn 3.1-formatted items. There will also be some repeated changes in the logs if you sync 3.0 items with 3.1, so usage with 3.0 items is not recommended.

Introducing breaking changes was a hard decision to make, but I think the saner defaults and new capabilities will justify the slightly more difficult upgrade process. So what’s breaking?

  • The item’s database name is now stored in the serialized data (#7). This enables parsing serialized items in contexts where the database cannot easily be inferred, such as Courier.
  • Unversioned fields have official support and are nested under the Language instead of under a Version within the language which was incorrect. Prior to reserializing, syncing a 3.0-formatted item with 3.1 may result in messages that the field has been reset to standard values and then set again. IItemData now has an UnversionedFields property to store unversioned fields by language.
  • The name on serialized fields has been converted from a comment in the YAML into an actual parsed value. Note that this value is just a hint and if a template field is renamed it is NOT updated automatically.
  • The default setting of Rainbow.SFS.ExtraInvalidFilenameCharacters has been changed to "$" to make Rainbow compatible by default with TFS, and the TFS config example that previously set this has been removed.
  • The default setting for Rainbow.SFS.SerializationFolderPathMaxLength has been increased from 90 to 110 based on real world experience that 90 is too short.
  • The default setting for Rainbow.SFS.MaxItemNameLengthBeforeTruncation has been reduced from 100 to 30 based on real world experience that 100 was long enough to break the path length limitation for items with near 100 length names.
  • Automated deployment tools calling Unicorn have a new security API. See below for details on this, but it’s now more secure.

Other Improvements:

  • Syncing dictionary items will now cause the dictionary cache to be cleared at the end of the sync, forcing the synced entries to be loaded.
  • The Unicorn Control Panel is now pipeline-based and extensible so that modules may define their own verbs or alter the UI if needed.
  • Error messages when serializing and deserializing XML-based fields (layout, rules) have been improved.
  • Unicorn and Rainbow both now utilize the C# 6.0 syntax and require the C# 6.0 compiler to build. They still target .NET 4.5.
  • Config file comments and documentation have been updated to cover the new features of Unicorn 3.1.

Bug fixes:

  • Renaming or moving items which are serialized on loopback paths will no longer cause the serialized children to be deleted.
  • Missing field errors on the root item of a tree now result in a retry and non fatal error as expected.
  • A non fatal warning when syncing multiple configurations will no longer result in the sync stopping at the end of the configuration with the non fatal warning.
  • When using the New Items Only evaluator, you will no longer receive sync conflict warnings when saving items created by the evaluator.
  • Copying items when Transparent Sync is enabled and the source item is not in the Sitecore database now works correctly.
  • Empty field values will no longer cause problems when Transparent Sync is enabled.
  • XML-based fields (layout, rules) with empty values will no longer cause exceptions. #6, #90
  • Items installed by Sitecore packages or update packages into Unicorn configuration areas will no longer lose some field values in the serialized items. #92
  • Predicate include entries and SFS trees are no longer confused by paths which are different but start with similar bases, such as /sitecore/content and /sitecore/content monkey
  • Plain text Unicorn logs returned to automated tool calls will no longer include HTML-encoded values such as &gt;
  • Naming items a Windows reserved file name.aspx) such as CON or COM1 will no longer result in an error. #8
  • Fields with a blank value in Sitecore that were not included in the serialized item were not being correctly reset to standard values. This has been fixed.

Upgrading

Ok so there are breaking changes, how do I upgrade? Thankfully it’s not too hard.

  1. Before you upgrade, run a full sync of all configurations. If some have Transparent Sync enabled, turn that off during the upgrade. (you can reenable after step 4)
  2. Upgrade your NuGet packages to Unicorn 3.1. (which will also install Rainbow 1.1)
  3. Run a build of your solution, and if you develop out of webroot publish your changes.
  4. Using the Unicorn control panel, Reserialize all configurations to flush them to disk as 3.1-format items.
  5. If you use the automated tool API to do automated execution of Unicorn, migrate to the PowerShell module to invoke the new API.
  6. Have fun!

Contributors

Unicorn 3.1 has seen contributions from many community members. Thank you all for your support, testing, bug reports, and contributions!

The Benchmark Show: Sitecore Precompilation

I’m mildly obsessed with Sitecore performance, so when I found a new trick I had to see what it could do: aspnet_compiler.exe. This is a really old tool that was designed to compile your .aspx pages into an assembly - and then later your cshtml as well. This improves startup time because no dynamic compilation needs to take place at runtime.

Getting aspnet_compiler to run against Sitecore, by which I mean executing it against an entire deployment ready web root including both your code and the sitecore folder, is a bit of a challenge. Because you cannot tell aspnet_compiler to ignore anything, and because one of the cshtml files in Sitecore 8.1 Update-1 apparently has a compiler error (for the curious, add @using Sitecore.ListManagement.Client.Web.UI.Controls.ImportMapTo to /sitecore/shell/client/Applications/ListManager/Controls/ContactFields/ContactFields.cshtml). You also have to comment out the @Html.Sitecore()s in /sitecore/shell/Templates/layout.cshtml (don’t worry it’s just a template used when you add a new MVC layout).

Once you get the fixes made, running aspnet_compiler is as simple as:

"c:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe" -v / -p "c:\path\to\deploy\ready\files" "c:\output\path"

It will take a while - probably at least a minute - while it compiles every aspx and cshtml in the whole place into assemblies.

Cool trick. Does it work?

Let’s look at some hard data. I took my base Sitecore site (which runs Performance.config) and executed my deployment scripts over it, then precompiled that to a new folder with aspnet_compiler. Then I devised several tests to tease out both the effects of precompilation and whether Sitecore’s built in SPEAK precompilation has improved since its 8.0 days - where it could make startup take 140 seconds. The tests were in most cases run both with a cold ASP.NET Temporary Files folder (where .NET puts compiled Razor caches) and with a warm one for comparison. Cold would be the first startup on a new machine, whereas warm would be after an app pool recycle thereafter.

Test 1: Home Page

This is a simple test of loading the home page, which is a fairly simple Sitecore MVC layout with about 4 view and controller renderings and several partials. All tests begin with the app pool stopped and no w3wp executing.

Test 2: SPEAK Media Library Dialog

The SPEAK media browser (what you get when browsing from an image field, for example) has been historically a pain point performance wise. For this test I loaded the dialog iframe URL into a Chrome tab and measured the time to finish page load with the Chrome developer tools.

Results

Home Page (cold)

Baseline (Performance.config): 22.25s
SC Precompiler: 24.8s
Precompiled: 21.5s

As expected the precompiled is marginally fastest. Sitecore 8.1’s precompilation function, as we’ll see throughout the tests, seems to add 2-4 seconds to startup time. Without preexisting compiled temp files, startup is extremely slow in all cases - unexpectedly so for the supposedly ‘precompiled’ site. My guess is that it copies the precompiled files to the temp folder, which expends I/O, but that’s a guess.

Home Page (warm)

Baseline: 9.71s
SC Precompiler: 13.97s
Precompiled: 9.51s

With the temporary files warm, startup gets much faster. Once again the Sitecore precompiler is in last place.

SPEAK Media (cold)

Baseline: 29.6s
SC Precompiler: 33.8s
Precompiled: 22.93s

The effects of precompilation are strong when loading the very complex, partial-heavy SPEAK dialog - coming in 47% faster than the Sitecore default.

SPEAK Media (warm)

Baseline: 12.2s
SC Precompiler: 16s
Precompiled: 12.82s
SC Precompiler (warmed): 13.23s

Performance is nearly identical once the temporary compiled files are cached the first time for SPEAK. In this case there is also a test for having warmed the Sitecore precompiler and allowed it to finish precompiling in the background, then restarted the cold app pool.

Analysis (TL;DR)

Overall the effects of using aspnet_compiler prior to deployment are relatively minor in most real world cases. Initial startup is ideally a rare case, and once their caches are heated all options perform quite similarly (as an example, the second hit to the media dialog is ~250ms to finish). For more elastic deployments, such as cloud setups, or for larger clusters it is probably worth the effort since you burn the CPU cycles once and it does significantly improve first hit to SPEAK dialogs as well as marginal overall performance improvements. Rendering heavy Sitecore MVC implementations would also likely see similar gains to SPEAK. In addition to performance, using the aspnet_compiler validates the C# in your Razor views - which is a nice plus in the event of a Razor syntax error which would normally be discovered at runtime.

The big downside of precompiling is that it’s fairly slow (several minutes on fast hardware, probably much slower otherwise or without SSDs), and that it has to re-copy the whole web root to another output location (lots of I/O). So overall build time on CI may increase significantly.

Another option: Use the Sitecore precompiler

If you look at the results, startup time with the Sitecore precompiler is always a bit longer. But once the Sitecore precompiler completes its asynchronous run - usually within a couple minutes after startup - the results are very close to a natively precompiled site (see the warm media dialog results). This is overall a much simpler option. It does mean every instance has to compile for itself, but it does not require adaptation of the build process nor long times spent copying files.

And here’s the best part: you can use it for your own views, too. It’s just a Razor precompiler pipeline processor, and you can add one - or more - for your sites’ MVC views. For example:

<processor type="Sitecore.Pipelines.Initialize.PrecompileSpeakViews, Sitecore.Speak.Client">
    <Paths>/Areas/MySite/Views</Paths> <!-- virtual path to your views -->
</processor>

This will cause your views to all get precompiled asynchronously on startup - if they are not already - which writes their assemblies into the Temporary ASP.NET Files folder. So you trade a couple seconds of startup time, and an initial spike of CPU cycles, for consistent performance thereafter. However, you lose the protection against syntax errors provided by build-time precompilation.

Next time on the benchmark show: Common Sitecore development tasks, benchmarked on a variety of hardware.

Unicorn 3.0.2 Released

Unicorn 3.0.2 (and Rainbow 1.0.1) have been released to NuGet.

This release includes some critical fixes to the serialization system, and upgrading is highly recommended for all users of Unicorn 3.

Improvements

  • Unicorn auto-publish now reports its activities to the sync console, and operates synchronously to guarantee published state for automated builds.
  • Guidance on some Transparent Sync pitfalls has been added to the wiki.
  • The new Rainbow.SFS.ExtraInvalidFilenameCharacters setting enables TFS users to successfully serialize branch templates which begin in $, a reserved character by TFS. The Rainbow.TFS.config.example file is now shipped with Rainbow to illustrate its usage for TFS.
  • An example configuration file to demonstrate how to move your serialized items elsewhere has been added.

Fixes

  • CRITICAL Saving item versions that were not in the default Sitecore language was not updating the serialized item. This has been fixed. #76
  • CRITICAL Serialization of significant empty field values, such as unchecked checkboxes when the standard value is checked, has been fixed. #74
  • Items that are restored from the Sitecore recycle bin or archive to Unicorn-controlled locations now result in the item being serialized on restore. #72
  • Resetting a field to standard values now correctly results in the field value being removed from the serialized item. #74
  • Query string order is always respected when syncing multiple configurations at once using the control panel.
  • The serialization conflict dialog (if disk doesn’t match base item data on save) now formats correctly using HTML for Sitecore 8+
  • Using the Dump Item and Dump Tree commands in Sitecore to reserialize partial item trees now correctly serializes an item to all configurations that it may belong to, instead of just the first one.

Upgrading

Upgrading from Unicorn 3.x is as simple as updating your NuGet packages. If it prompts to overwrite any changed config files, I’d suggest doing so and using a diff with your source control to resolve differences.

If you suspect that you have serialized items that may be missing significant empty values, you should reserialize those after upgrading to get their values normalized.

The Credits

Unicorn isn’t just my project. Thanks a lot to all the community members who reported bugs, sent pull requests, and diagnosed issues to help make Unicorn better.

Unicorn 3.0.2 is brought to you by…

Robin Hermanussen
Thomas Eldblom
Nathanael Mann
@GeorgeKarbyn
Kevin Williams
@amitthakur2014
@WizX20
@kamsar
…and the rest of #unicorn on Sitecore Slack

As usual, if you run into trouble you can submit an issue on GitHub, hit me up on Sitecore Slack, or tweet at me.

Unicorn 3.0.1 Released

The first maintenance release for Unicorn 3.x has been released to NuGet. Unicorn 3.0.1 brings with it a small set of bug fixes and improvements:

  • Data provider config patch is compatible with Sitecore Commerce and other modules that add data providers (thanks @jonnekats!)
  • Fixed a bug in Transparent Sync where syncing an item based on a template that only existed in transparent sync (and not the database) would act like the template did not exist. Ironically this was fixed in 3.0.0 but some n00b forgot to merge the config to activate it! (thanks Kevin Williams!)
  • Fixed a race condition that could cause SFS trees to incorrectly reinitialize during a reserialize operation and cause an error (thanks @akshaysura13!)
  • Content editor warnings have been improved:
    • The name of the Unicorn configuration that includes the item is now shown so it’s easy to trace back why something is included.
    • For transparent sync items, there is an indication of whether the item is on disk only or both on disk and in the database as well. Note that items in both places may well have different field values, there is no equality check.

Due to the fix to Transparent Sync, this update is recommended for anyone using that feature.

Have fun!

Unicorn 3.0 Released

patrick you're a bloody genius

It’s been over a year since the last major release of Unicorn, but I haven’t been bored. Today I’m happy to announce the release of Unicorn 3.0 and Rainbow 1.0.

Unicorn 101

Sitecore development artifacts are both code and database items, such as rendering code and rendering items. As developers, we use serialization to write our database artifacts into source control along with our code so that we have a record of all our development artifacts. Unicorn is a tool to make serializing items and sharing them across teams and deployed environments easy and fun.

Unicorn takes advantage of keeping serialized items always up to date on the disk, which allows all merging to take place using your source control tool. Modern source control tools such as Git or SVN are capable of automatically merging the vast majority of item changes, which saves you time and reduces the chance of human error when merging.

Unicorn 3 is fresh off the compiler and brings with it a huge raft of improvements. This version brings its own serialization format and filesystem hierarchy that are far more friendly to source control and merge conflicts than the default Sitecore format. It’s also ridiculously fast - about 50% faster overall than Unicorn 2 or Sitecore serialization APIs.

What else is new?

In fact there are lots more new features added to Unicorn 3 since the what’s new blog post was written. Not just minor details either! Read on…

Transparent Sync

This is a feature that demanded its own blog post! Transparent Sync enables Unicorn to sync the serialized items in real time, completely automatically. It does this by using its data provider to directly read the serialized items and inject them into the Sitecore content tree. The items on disk are the items in Sitecore: it bypasses the database entirely for transparently synced items.

Seriously, it’s magic. Transparent Sync might be the best feature of Unicorn 3. You should give it a try!

New Configuration Architecture

Previously Unicorn was distributed with its default example configuration directly in Unicorn.config. This was suboptimal because it encouraged modifying the stock config file and making future upgrades more of a merging challenge than they needed to be. In the new order, Unicorn ships without any configurations defined. There are two example configuration-adding patch files distributed with the NuGet package which you can duplicate and edit to your desires, and the README distributed with the NuGet package details what you need to do to get started.

Control Panel UX Improvements

The control panel has been redesigned to have an improved UX when there are many configurations defined. In the old UI, once more than a couple configurations were defined the page would need to scroll. The new UI reduces the vertical space for each configuration significantly:

new control panel

The new control panel also has an experience for selecting only the configurations you wish to sync in a single batch:

new control panel

For automated builds, you can also copy the URL on the sync button when you’ve got multiple configurations selected in the control panel to run the same batch at build time. This is great if you’ve got some configurations that you may not wish to sync during a CI build.

New Items Only Evaluator

This is an evaluator that changes the behavior of syncing to only write new items from serialization into Sitecore. Existing items with the same ID or items that are not serialized are left alone. This can be useful for example if you’re wanting to push some developer-initiated content items up, like metadata or lookup source items, that you want to always exist but once they have been created they become the content editor’s to do with as they will. A sample configuration that enables the NIO evaluator ships with the NuGet package. Thanks to Nathanael Mann for the feature request.

Exclude all children with the predicate

To exclude all children of an item simply add a trailing slash to the <exclude> like so:

<include database="master" path="/sitecore/content">
    <exclude path="/sitecore/content/" />
</include>

Visual Studio Control Panel Support

Recently a plugin for Visual Studio was released that enables you to run Unicorn syncs directly within Visual Studio. Unicorn 3 supports this tool out of the box, by simply enabling the Unicorn.Remote.config.disabled patch file.

new control panel

The Unicorn Control Panel for Visual Studio requires VS2013 or VS2015. It is developed by Andrii Snihyr - thank you for the community support!

TFS Support

Fellow Sitecore MVP and Connective DX’er Dave Peterson has written up a plugin for Rainbow/Unicorn that integrates it with TFS. TFS 2010 (or server workspaces in 2012 and later) have the unfortunate limitation that all files are locked by default. This interferes with Unicorn’s operation as it writes files as items are changed in Sitecore, resulting in errors.

The TFS integration hooks the TFS API to Rainbow’s SFS data store, causing it to actually check out the file Unicorn is about to write or delete before acting. You must use a 32-bit IIS app pool when using the plugin, as the TFS API is 32-bit only.

The TFS plugin is available on NuGet; for installation instructions and the latest info see the README on GitHub

Documentation

The documentation in the README, comments in the stock config files, and verbiage in the actual control panel and logs have all been reviewed and improved. If you get confused by anything, let me know and I’ll make the docs better or the error message clearer in the next release.

Thank you

Unicorn is a project by and for the Sitecore community. I’d like to thank everyone who’s contributed to the project in a major or minor way: without you, we wouldn’t have this tool today. Thank you!

Introducing Transparent Sync in Unicorn 3

A couple years ago Alex Shyba told me about an idea he had: what if we used a data provider to map serialized items directly into the content tree, bypassing the database and eliminating the need to sync? I was like

ermahgerd!

I went nuts and wrote a prototype that did exactly that.

The prototype, nicknamed Rhino because it has a prominent horn like certain other mythical beasts, actually worked fairly well. Unfortunately there are two hard problems in computer science: naming, cache invalidation, and off by one errors. Cache invalidation, specifically using the FileSystemWatcher to observe file changes by source control, was unreliable. Because of how core serialization is to Sitecore development practice, unreliability is not acceptable. Reluctantly, I shelved Rhino and worked on Unicorn 2 instead.

unicorn logo

The idea of Rhino stuck around. The improvements brought around in Rainbow, such as partial item reading and tighter control around storage, enabled working around the limitations that had precluded Rhino from being useful in production. Thus the idea returns as Transparent Sync, which might just be the best part of Unicorn 3.

Transparent Sync enables Unicorn to sync serialized items in real time, completely automatically. It does this by using its data provider to directly read the serialized items and inject them into the Sitecore content tree. The items on disk are the items in Sitecore: it bypasses the database entirely for transparently synced items. Changes made on disk update nearly instantly in the Sitecore editing interfaces.

not sure if witchcraft, or transparent sync

Imagine a scenario where a development team is working with a feature-branch driven workflow, like GitHub Flow. In order to perform a code review when using Transparent Sync, you merely checkout the feature branch under review and your Sitecore is immediately configured with the items that the feature includes.

I don't always do code reviews

When should we use Transparent Sync?

Transparent Sync is turned off by default because you should understand how it works before enabling it.

Transparent Sync is excellent for development artifacts like templates and rendering items, but it’s inappropriate if you’ve got hundreds of thousands of content items. At startup Unicorn must build an index of metadata, which involves reading the headers of each serialized file. If you have a SSD this penalty is pretty minimal, but a traditional hard drive not so much. In testing I enabled transparent sync for the whole default core and master database (19,228 items) using a SSD and noted about 100ms increase in startup times on average.

Because Transparent Sync bypasses the normal sync process, transparent sync also bypasses anything that is hooked to sync. This would include things like custom evaluators (like NewItemsOnlyEvaluator) and the sync event pipelines. If you are relying on these customizations, turn transparent sync off for the configurations that use them.

How do we use Transparent Sync?

Note: you must perform an initial serialization of a configuration with Transparent Sync off before you enable it. Otherwise the items in the configuration will seem to disappear as transparent sync shows all zero items that are on disk!

Turning transparent sync on is really easy: take the <configuration> you want to add transparent sync to and put this line in it:

<dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" />

You can also change the setting in the global <defaults> if you want to change the default setting of transparent sync for all configurations.

Once Transparent Sync is enabled all you have to do is change items on disk and the updates will immediately appear within Sitecore.

Going to production

In production we usually remove the Unicorn data provider as it’s not normally required. However without the Unicorn data provider, transparently synced items disappear: they aren’t really in the database at all, they’re on disk. There are two approaches to solve this problem:

  • Transparent Synced configurations can sync in the traditional way. This will persist the disk-based items permanently in the database.
  • Keep the data provider enabled in production. This is appealing because it means you can just deploy files to production and be done with it - and rollback in the same way. Be aware that if the IIS app pool identity cannot write to the serialized items you will be unable to make any ‘emergency’ changes to the items in Sitecore.

What happens if I reserialize a Transparent Sync configuration?

The database is used for all reserialize operations. In a Transparent Sync configuration, it is common for items to NOT reside in the database at all. If you reserialize this configuration it will be reset to what is in the database, thus deleting any Transparent Sync items that are not already in the database. Similarly if you use ‘Dump Item’ to do a partial reserialize on a Transparent Sync item it will be reverted to what is in the database, which may well be ‘nothing’.

There are warnings in the control panel if you reserialize a Transparent Sync configuration.

Anything else?

I hope you have as much fun with Transparent Sync as I had making it!

Unicorn 3: What's new?

Now that I’ve finished being confusing with my last three posts about Rainbow, let’s talk about Unicorn 3.

new logo!

Unicorn 3 is the product of a year of thought and implementation. The design goal was nothing less than fixing every annoyance or issue that we ran across in daily usage of Unicorn 2. So what’s new?

New serialization format

High on the list of annoyances with daily Unicorn usage was the difficulty of resolving merge conflicts in the Sitecore serialization format.

The only fix for this was to make a better format: human readable, easily mergeable, and fixing the daily annoyances of the Sitecore format. Because no tool is an island, the new serialization tools Unicorn 3 uses have been split off into their own library: Rainbow. Rainbow enables others to use the new serialization tools developed for Unicorn 3, without depending on Unicorn.

The details of the new format and why it exists can be found in part 1 of my Rainbow blog series.

New file organization

Alongside designing the new serialization format, another problem was to resolve the longstanding limitations and bugs in the way Sitecore’s built in serialization stores its file hierarchy. This was worth a whole blog post in and of itself, but in summary I think those limitations have been eliminated.

The new hierarchy also alters the way the storage tree operates. Whereas Sitecore’s model represents an entire database, Unicorn 3 represents a set of subtrees of the database. Depending on the depth of the root paths, this can result in much shorter filesystem paths and fewer over-length path loops.

Improved user experience

Improved console messaging

Unicorn 3 has greatly improved messaging, both reducing extra output that isn’t necessary as well as vastly better error formatting. This hilarity was a constant annoyance if something went wrong in Unicorn 2 (in this case, invalid serialization format):

seriously? who shipped this.

Unicorn 3’s version of the same error:

woohoo

Editor warnings for Unicorn-controlled items

Another common issue is having someone edit an item that Unicorn controls on a deployed server, and having their changes overwritten by the next automated deployment. People asked for a way they could know if an item was “Unicorned” or not. Well, now there is one:

warning will robinson

The message shown changes depending on the environment as well; this is how it looks when “production mode” is on:

warning will robinson

And also, if production mode is enabled if you attempt to save the item:

warning will robinson

Unicorn-enabled serialize commands

Ever used these handy tools on the by-default-hidden Developer tab of the Sitecore ribbon?

controls

Guess what happens if you use these commands on an item Unicorn 3 controls? You guessed it: you can do partial syncing and partial reserialization using these commands. In Unicorn terms, there is no difference between “update” and “revert”: both just mean “sync.”

Note: Using the commands on non-Unicorn items results in their default behaviour. Also “update database” and “revert database” do not interface with Unicorn at all, and perform their default actions.

Performance: 50% more of it

Unicorn 3 has had significant performance profiling applied to it, and is about 10-20% faster than Unicorn 2 or stock Sitecore serialization. On top of that, multithreading is utilized to further increase sync and reserialize speed. Between threading and optimizations, it’s generally about 50% faster than Unicorn 2. For maximum performance, using SSD storage is highly recommended as it has a major impact on sync speed.

Auto-publish synced Items

Unicorn 3 enables efficient auto-publishing of items that a sync has modified, using manual publish queue injection. This option is enabled by default by the Unicorn.AutoPublish.config file. You can remove this file to disable the feature if you don’t like it.

Deleted template field handling

In Unicorn 2, deleting template fields could cause havoc. If any other serialized item contained a value for the deleted field, and you failed to reserialize that item after deleting the field, your next sync would be greeted with a big ugly error and you’d have to go manually remove the offending field from the file.

Unicorn 3 fixes this issue, by making this error a warning instead. (We can’t just ignore it, because ignoring it would cause syncs that created template fields as well as values for that field to not load the values the first time)

woohoo

Versioned to Shared field conversion

Converting a field between shared, versioned, and unversioned by syncing a template was not supported in Unicorn 2. The field itself would change, but the existing values for the field would not move to the correct database table in Sitecore. Unicorn 3 includes the necessary adaptation to migrate stored values when field sharing is changed by a sync.

Improved configuration scheme

Unicorn 2 used a single configuration file: Serialization.config. This was a bit confusing as parts of it were ideally removed when deployed to a CE or CD environment and that was not always obvious.

Version 3 addresses this situation by creating its own App_Config\Unicorn folder and splitting the configuration into files that can simply be deleted instead of modified. This folder contains:

  • Unicorn.config: Contains core configuration, e.g. predicates and dependency setup, can be left intact anywhere.
  • Unicorn.DataProvider.config: Attaches the Unicorn Data Provider, which handles automatic serialization of changed items to disk. Can be removed for any environment not collecting changes. (usually, any non-dev environment)
  • Unicorn.UI.config: Adds the Unicorn Control Panel, Partial Sync, editor warnings, and other end-user UI elements. Remove for CD environments.
  • Unicorn.AutoPublish.config: Causes Unicorn to automatically publish items that it changes during a sync. Remove if you don’t want this feature, or on CD environments.
  • Unicorn.Deployed.config.disabled: If this config is enabled, typically on a deployed CE instance, saving an item added to Unicorn results in a warning confirmation.

Each of these config files also has comments explaining the above as well as what the settings within do.

Technical Improvements

Unicorn 3 also includes numerous miscellaneous technical improvements.

  • No third party dependencies. (that means you, Ninject)
  • Improved default predicate configuration settings account for Sitecore 8 additions and common modules.
  • Added SerializationHelper, a handy class that lets you more easily invoke Unicorn operations such as Sync without a lot of setup, if you want to programmatically use Unicorn.
  • Added pipelines to hook to sync events. unicornSyncBegin occurs when a configuration sync begins. unicornSyncComplete occurs when a configuration sync ends. unicornSyncEnd occurs at the end of all syncs. (e.g. after all configurations have synced if batching > 1)
  • Thanks to Rainbow, the object model is unified (ALL items are IItemData both from Sitecore and serialized). Previously each had their own parallel models.

System Requirements

Unicorn 3 has been developed on Sitecore 7.2 Update-5, as well as Sitecore 8 Update-5. It should be compatible with all version of Sitecore 7 and 8.

Unicorn 3’s code should be easy to adapt to Sitecore 6.2-6.6, however there is no official support for it so you’d want to compile from source.

Ok, ok. Is it released yet?

No. But it is in an actively updated beta on NuGet that is fairly stable :)

Rainbow Part 3: What is Rainbow?

It may seem a bit weird that in part 3 of a series, we’re talking about what something is.

Nobody can tell you what it is, you have to see it for yourself

After all, shouldn’t that be part 1? Nope, this is iterative development :)

So, what is Rainbow? Rainbow is designed to be a complete replacement for the Sitecore serialization format and filesystem organization, as well as enabling cross-source item comparison. It is a pure code library that comes with no UI of any kind, that is designed to be used with other libraries that use its serialization services with their own UIs. Libraries that consume Rainbow - such as Unicorn - gain the ability to abstract themselves from serialization details. Libraries that extend Rainbow can add new serialization formats, new places to store serialized items, and new ways to organize them.

taste it

Rainbow Features

Universal Item Data and Data Stores

Rainbow implements a set of interfaces that wrap the structure of a Sitecore item - item, version, and field. These interfaces provide a universal language that all Rainbow data stores can implement against. You could get an IItemData from Sitecore and write it out to disk as a YAML formatted item. You could get an IItemData from a web service, and deserialize it into a Sitecore database. You could construct an IItemData programmatically, and serialize it to a Sitecore database. Implementations of IDataStore provide places to store item data. It’s completely universal, and everything Rainbow does revolves around these abstractions.

YAML-based serialization formatter improves the storage format for serialized items

  • This post goes into more detail about the hows and whys of the YAML serializer
  • The format is valid YAML. YAML is a language that is designed to store object graphs in a human readable fashion. It uses significant whitespace and indentation to denote data boundaries. Note: only a subset of the YAML spec is allowed for performance reasons.
  • Any type of endline support. Yes, even \r because one of the default Sitecore database items uses that in its text! No more .gitattributes needed.
  • No more Content-Length on fields that requires manual recalculation after merge conflicts
  • Multilists are stored multi-line so fewer conflicts can occur
  • XML fields (layout, rules) are stored pretty-printed so that fewer conflicts can occur
  • Customize how fields are stored when serialized yourself with Field Formatters

Serialization File System (SFS) storage hierarchy

  • This post goes into more detail about SFS
  • Human readable file hierarchy
  • Extremely long item name support
  • Unlimited path length support
  • Supports non-unique paths (two items under the same parent with the same name), while keeping it human-readable
  • Stores each included subtree in its own hierarchy, reducing file name length
  • You can plug in the serialization formatter you desire - such as the YAML provider - to format items how you want in the tree

Deserialize abstract items into Sitecore with the Sitecore storage provider

  • Turn Sitecore items into IItemData with the ItemData class
  • Deserialize an IItemData instance into a Sitecore item with DefaultDeserializer (used via SitecoreDataStore)
  • Query the Sitecore tree in abstract with SitecoreDataStore, as if it were any other serialization store

Item comparison APIs

  • Compare any two IItemData instances regardless of source
  • Customize comparison for field types or specific fields with Field Comparers
  • Get a complete readout of changes as an object model

Improvements

Improvements are in comparison to Sitecore serialization and the functionality in Unicorn 2.

  • Deleting template fields will no longer cause errors on deserialization for items that are serialized with a value for the deleted field (e.g. standard values)
  • Deserialization knows how to properly change field sharing or field versioning and port existing data to the new sharing setting

Rainbow Organization

Rainbow consists of several projects:

  • The core Rainbow project contains core interfaces and components that most all serialization components might need, for example SFS, item comparison tools, field formatters, and item filtering
  • Rainbow.Storage.Yaml implements serializing and deserializing items using a YAML-based format. This format is ridiculously easier to read and merge than standard Sitecore serialization format, lacking any content length attributes, supporting any type of newline characters, and supporting pretty-printing field values (e.g. multilists, layout) for simpler merging when conflicts occur.
  • Rainbow.Storage.Sc implements a data store using the Sitecore database. This can be used to read and write items to Sitecore using the IDataStore interface.

Extending Rainbow

Rainbow is designed to be loosely coupled. It’s recommended that you employ a Dependency Injection framework (e.g. SimpleInjector) to make your life constructing Rainbow objects easier. Of course you can also construct objects without DI.

Rainbow has extensive unit and integration tests and hopefully easy to understand code, which serve as its living documentation. As of now, Rainbow has 90% test coverage and 220 tests. Unicorn, which uses Rainbow as a service, can also be a useful source of examples.

If you can’t find what you’re looking for you can find me on Twitter (@kamsar) or on Sitecore Community Slack.

Next time, we’ll start talking about what’s new in Unicorn 3 - other than all the Rainbow things that come along because Unicorn uses Rainbow.