Synthesis 6.0.0 has been released to the NuGet gallery. Despite the lofty sounding version bump, this is just a case of SemVer because Sitecore 7 Update 1 moved a class (IndexField) between assemblies. This means binary distribution of Synthesis for Sitecore 7 RTM will not work with Update 1, and distributions for Update 1 will not work with RTM. Joy!

Only upgrade if you’re using Sitecore 7 Update 1!

New things in 6.0.0:

• Fixed an issue where unpublished fields could cause NullReferenceExceptions when accessing HasValue in weird ways
• Added the ability to customize the interface suffix on generated interfaces
• Compiled for NuGet against Sitecore 7 Update 1

I’ve released Unicorn 1.0.4 to NuGet. This release fixes bugs that would cause serialization inconsistencies to occur if items having children under Unicorn’s control were renamed or moved (the paths stored in the moved children would still be the original path).

The solution is to re-serialize children after moving or renaming, which means Unicorn will not scale very well if you put a giant number of items under it and then move the parent of them all. But then it’s not designed for that in the first place - a template with 30-40 subitems should be tolerably quick for a relatively unusual operation.

This fix has been verified using Rhino.Fsck, a library for verifying the consistency of a serialized tree of items.

Rhino solves this issue slightly more elegantly by simply reading the items from disk, changing the path, and re-writing the SyncItem, but it’s also lower level. Eventually I’d like to deprecate Unicorn in favor of Rhino.

I’ve pushed Unicorn 1.0.3 to NuGet. This release fixes a number of bugs, improves logging, and removes some annoyances:

• Deleting a whole tree could cause the app pool to be killed due to a bug in Sitecore’s serialization event handler (380479) - this fix expands on a previous workaround to cover recursive deletes
• Fixed an issue where if you changed an item too rapidly after saving or moving it, you could get file read errors or spurious “conflicts” reported due to the async ShadowWriter still being in process.
• “Inconsequential” item saves are now ignored. For example, the Template Editor gleefully changes the revision and last updated date of every template field when it saves, not merely changed ones. Those, as well as rename events where the name does not actually change (again, Template Editor ftw), are ignored and updated files are not written to disk. These fields are also ignored when calculating serialization conflicts on item save.
• Items deleted by Unicorn are now placed in the recycle bin instead of completely removed, so they can be restored if mistakenly deleted.
• Unicorn sync logs are automatically written to the Sitecore log as well (requires dependency upgrade to Kamsar.WebConsole 1.2.2 for the new TeeProgressStatus)

Here’s a fun little issue that you might eventually run across. Suppose you’re writing some code that needs to retrieve the extension from a URL. For example, a URL rewriter or Sitecore pipeline that acts only on certain file types.

You might think, as I would have, “oh, that’s built in to .NET - we’ll just use System.IO.Path.GetExtension()!”

And that would work, almost all of the time. The only issue comes from an internal implementation detail of IO.Path: it checks that the path does not contain characters that are invalid in a filesystem path. Specifically, “, <, >, |, and ASCII unprintables (< 32). Well, those characters (except perhaps the unprintables) are valid in a URL - so trying to get the extension of a URL path containing these characters will throw a nice fat exception:

System.ArgumentException: Illegal characters in path.
   at System.IO.Path.CheckInvalidPathChars(String path)
   at System.IO.Path.GetExtension(String path)

Unfortunately there is no way to disable this behavior - which is logical, given the purpose of IO.Path as a processor for file system paths, not URLs. Not even first parsing the URL using the System.Uri class will fix this, as this StackOverflow question suggests. The LocalPath property still includes the invalid characters that break Path.GetFileName() or Path.GetExtension().

There are a couple of ways I could see solving this problem. The first, and simplest - though possibly prone to security issues, would be to replicate what Path.GetExtension() does but omitting the invalid characters check. Reference:

int length = path.Length;
for (int i = length; --i >= 0; )
{
    char ch = path[i];
    if (ch == '.')
    {
        if (i != length - 1)
            return path.Substring(i, length - i);
        else
            return String.Empty;
    }
    if (ch == DirectorySeparatorChar || ch == AltDirectorySeparatorChar || ch == VolumeSeparatorChar)
        break;
}
return String.Empty;

The second would be to remove any invalid characters prior to calling Path.GetExtension():

string url = "\"http://mo\"nkeys/foo/bar/\"";
            
var invalidChars = Path.GetInvalidPathChars().ToHashSet();
for (int i = url.Length - 1; i >= 0; i--)
{
    // technically you could use a stringbuilder, but since 99% of the time this won't ever be used, this seems optimized enough
    if (invalidChars.Contains(url[i])) url = url.Remove(i, 1);
}

var ext = Path.GetExtension(url);

Personally I like the second solution, since you’d benefit from any upstream fixes in Path.GetExtension() in future framework releases.

For years at ISITE Design we've been inventing ways of making Sitecore development easier and more fun. Today I'm happy to announce that some of those tools we've been using internally for ages are now publicly available for everyone to use.

Synthesis

Synthesis is an object mapping framework that automatically generates strongly typed C# classes to represent your Sitecore templates. While this sort of library is nothing new in Sitecore circles, Synthesis brings some unique features to the table:

• A universal object interface for Sitecore 7. The same generated objects that work in database queries also work great in LINQ against an index. If you access data that isn't in the index, they'll even transparently load the item from the database for you.
• Automatic LINQ query filtering. You won't have to remember to filter by template, language, and version when you query LINQ with Synthesis.
• Generation of an interface hierarchy to represent template inheritance. This brings a whole world of power to you by enabling you to write polymorphic renderings. Nearly all of your interaction can be done by interface.
• Automatic checking to see if your generated classes match what's defined in Sitecore
• Extensible field API allows you to automatically map custom field types onto a strongly typed version of themselves. Ever dealt with custom XML fields before? This makes them awesome.

All of the other expected features of an object mapper are also available, such as:

• Full support for Page Editor
• Support for standard field types such as images, multilists, and rich text fields

In Action

There are some examples of using Synthesis on the GitHub page.

Getting Synthesis

Synthesis is available now on NuGet for a one-click install. You can also see the source on GitHub, or review the documentation.

Which NuGet package should I download? For installing into a web project, use the Synthesis package - it comes with configuration files. If you're referencing it from a library, use the Synthesis.Core package which is binary only.

Blade

While Synthesis handles the mapping side, Blade handles making renderings awesome and has some nice integration points with Synthesis.

Blade uses the MVP pattern to allow you to create very DRY, single-responsibility renderings, allowing you to focus on presentation code and not data access. With Blade, your renderings only request the type of model they expect. For example:

public partial class SampleSublayout : UserControlView<MyViewModelClass>

Then Blade steps in and maps the model type onto a Presenter class, which defines how to supply the model type. For example:

public class SamplePresenter : SitecorePresenter<MyViewModelClass>
{
    protected override MyViewModelClass GetModel(IView view, Item dataSource)
    {
        var model = new MyViewModelClass();
        model.Example = dataSource.DisplayName;

        return model;
    }
}

This enables simple isolation of both views and presenters for writing testable frontends, and looser coupling even if you aren't testing. Don't worry, not every rendering needs a presenter. If you're just using a model type from a Sitecore object mapper such as Synthesis (via the Synthesis.Blade package), Glass, or Compiled Domain Model, you can make that automatically map to the data source item without defining an explicit presenter.

Ubiquitous Razor 2 Support

Blade also brings first-class support of the Razor 2 templating language into Sitecore, without using the Sitecore MVC functionality. This is advantageous when developing sites that use modules that are not compatible with the Sitecore MVC rendering technology - you can reap the benefits of MVC-style renderings without the hassle of implementing renderings twice or resorting to XSLT. A Razor rendering with Blade works exactly like a sublayout - only the path is to a cshtml file instead of an ascx. There are also many useful helpers included that simplify things like rendering dates, images, and controlling page editor functionality. For example:

@inherits RazorRendering<MyProject.MyViewModelClass>
<div>
<h1>@Html.Raw(Model.Example)</h1>

@Html.TextFor(model => model.SitecoreDataField)
</div>

Blade does its best to feel as much as possible like Sitecore as well as ASP.NET MVC, so it doesn't require digesting lots of unfamiliar ways of doing things. Sitecore features such as Page Editor are fully supported. MVC conventions like automatic HTML encoding, partial views, anonymous types defining HTML attributes, and relative view referencing are all there.

Awesome Rendering Diagnostics

Ever spent an hour figuring out why a rendering was busted, only to realize it was output cached without you realizing it? What about having to find a rendering file for a HTML/CSS dev so they can make markup changes? You'll love Blade then. Blade's rendering types emit HTML comments that indicate where a specific rendering begins and ends, as well as if it was output cached, when, and what the cache criteria were. The comments disappear when dynamic debug compilation is disabled, leaving production sites pristine.

Check this out:

<!-- Rendering was output cached at 5/16/2013 9:31:42 PM, ClearOnIndexUpdate, VaryByData, VaryByQueryString -->

<h1>Hello, world!</h1>
<!– End Rendering ~/layouts/Sample Inner Sublayout.ascx, render took 0ms –>

Full Sitecore 7 Support

Blade supports the enhancements made to renderings in Sitecore 7. It supports renderings that decache on index update, have multiple items as data sources, index-query based data sources, and even defining your own data source resolution logic. Even better, it removes boilerplate code by automatically normalizing all kinds of data source specifications back into simple Item (or Item[]) instances.

Getting Blade

Blade is available now on NuGet (and its Synthesis integration package) for a one-click install. Make sure you have Sitecore Rocks configured and connected before installing Blade as it uses Sitecore NuGet to install some supporting items. You can also see the source on GitHub, or review the documentation.

Which NuGet package should I download? For installing into a web project, use the Blade (and/or Synthesis.Blade) package - it comes with configuration files. If you're referencing it from a library, use the Blade.Core (and/or Synthesis.Blade.Core) package which is binary only.

Feedback

I'd love to hear what you think about these libraries or answer questions about them. Feel free to comment below, submit issues on the GitHub repositories (bonus points if there's a PR attached!), tweet @kamsar, or yell "hey you!" over the back fence.

Unicorn, a library I wrote to handle automatically serializing Sitecore content items into a source control system, is now available on NuGet.

Be sure to review the post-installation steps in the GitHub README file - if used incorrectly, Unicorn can be dangerous to your items.

In other news, I began a major refactoring of Unicorn this weekend that will bring it a ton of new features mainly in the form of extensibility. Don't want it to delete orphaned items? Custom comparison to determine updating? Implement your own Evaluator. Want to serialize to SQL? Use a custom serialization format that ignores versions? Sweet - make a SerializationProvider. Want to filter serialization based on anything you can imagine (like the rules engine? an Excel spreadsheet? presence of a version in a specific language?)? That's where a Predicate comes in.

At the moment "Unicorn 2.0" is not yet fully stable. I plan to add a few more things to it such as detecting duplicate item IDs during processing and possibly a "what-if" mode. I'll be working on it time permitting :)

My handy WebConsole library is now available from NuGet for easier installation.


This release also fixes an issue where disposal was working in a rather silly fashion on sub-tasks.


If you’re unaware of this library, it’s a very useful thing to use if you find yourself writing “hack pages,” “processor pages,” “rebuild pages” or other things of the like on a regular basis. Any long running or complex task you’d run from a page can easily get a realtime progress bar, console-style logging, exception handling support, and the ability to break the task up into ‘subtasks’ with their own tracked progress.


In short, you can make your processing pages more informative and even vaguely pretty if you like the console aesthetic, in practically no extra time over simply writing them in the first place. I wrote this for myself, and I use it in a bunch of things :)

As a followup to my previous post about gotchas with Sitecore 7’s LINQ provider, here’s another thing to consider.


The indexing providers are very greedy about indexing. This means that unlike with traditional querying with Sitecore.Data.Item, where your results are automatically filtered by the context language and latest item version, no such filtering occurs with LINQ. You will receive all versions and all languages unless you specify otherwise.


As you might imagine, this can result in unexpectedly large quantities of search results in some cases. It can also be extra sneaky since during development you might only have one version in one language - so you wouldn’t even notice the issue.


So how do you fix the issue? First, let’s talk about versions. The default indexing configuration includes a field called _latestversion.
This is a boolean field that is only set to true for items that are the latest version in their language. We can take advantage of this by implementing a property on our mapped objects and mapping it to this index field like so:

[IndexField("_latestversion")]
public bool IsLatestVersion { get; set; }

Then, when we write a query we want to limit, we simply add a clause to the query:

.Where(x => x.IsLatestVersion)

// alternatively if you don’t want to be strongly typed and have an indexer, you can use
.Where(x=> x[“_latestversion”] == “1”)

Now you’ll only get the latest version. Now for languages, which are also pretty simple. If you’re inheriting from the SearchResultItem class, you already have a Language property. Otherwise you can add one like so:

[IndexField("_language")]
public string Language { get; set; }

Then, we add the following clause to the query:

.Where(x => x.Language == Sitecore.Context.Language.Name)

Now we get results like regular queries. If you're like me, the next question you're asking is "how can I just write this once and forget about it?" For example something like:

public static IQueryable<T> GetFilteredQueryable<T>(this IProviderSearchContext context)
    where T : MyItemBaseClass
{
    return context.GetQueryable<T>().Where(x => x.Language == Sitecore.Context.Language.Name && x.IsLatestVersion);
}

Unfortunately, this seems to be nigh impossible with the current revision of Sitecore 7. The issue has to do with how LINQ resolves expressions involving generic types that are not resolved at compile time. Effectively the expression in the example above converts to:

.Where(x=> ((T)x).Language == Sitecore.Context.Language.Name)

Notice the cast to T? That throws the expression parser for a loop. I've been told this will be fixed in a later release of Sitecore 7, but will not be part of the RTM release, so for the moment it looks like we're writing filtering on each query.

The upcoming Sitecore 7 release brings with it a new “LINQ-to-provider” architecture that effectively allows you to query search indexes using standard LINQ syntax. Mark Stiles wrote a pretty good synopsis of the feature that you should probably read first if you’re unfamiliar with the basics of how it works. This post won’t cover the basics.


I’ve been diving in to the underpinnings of the LINQ architecture and have discovered a number of things that may well cause confusion when people start using this technology. Be warned that this post is based on a non-final release of Sitecore 7, and may well contain technical inaccuracies compared to the final release.

You have to enable field storage to get output

By default, the values of fields are not stored in the index. If the values are not stored, you can query against the index using custom object mapping (e.g. filters work), but you will not see any field values mapped into results objects. You can define field storage parameters either on a per-field basis, or a per-field-type (e.g. Single-Line Text) in the default index configuration file (in App_Config/Include).

Changes to the storage type require an index rebuild before the storage is reflected.

LINQ is not C Sharp

Yeah, you heard me right. LINQ may look exactly like C#, but it is not parsed the same way. The lambda expressions you may use to construct queries against Sitecore are compiled into an Abstract Syntax Tree (AST) - sort of a programmatic representation of the code forming the lambda - and that is in turn parsed by the Sitecore LINQ provider.

The code you write into lambdas is not executed as normal C# code is. This is important to remember, because effectively the LINQ provider is simply mapping your query in as simple terms as possible to a key-value query inside of Lucene (or SOLR, etc). For example:

// we'll use this as a complex property type to map into a lucene object<br>
public class FieldType {
    public string Value { get; set; }
    public string SomeOtherValue { get; set; }
}
// this will be the class we'll query on in LINQ
public class MappedClass {
    [IndexField("field1")]
    public FieldType Field1 { get; set; }
}

// example queries (abbreviated code)
var query1 = context.GetQueryable<mappedclass>().Where(x=>x.Field1.Value == "foo");
var query2 = context.GetQueryable<mappedclass>().Where(x=>x.Field1.SomeOtherValue == "foo");

You’d expect query1 and query2 to be different wouldn’t you? NOPE. You’re not writing C# here, you’re writing an AST in C# syntax. The Sitecore LINQ engine takes the shortest path to a key-value query pair. What this really means is that it:

1. Resolves the Lucene field name of the field you're querying on (in this case, "field1")
2. Evaluates the operator you used, and the constant value you've assigned to compare to
3. Constructs a Lucene query expression based on that

In effect, you can only ever have one value queried for each property. In the example above both examples are in effect x.Field1 == "foo". The query would be that way even if you did a crazy query like x.Field1.Foo.Bar.Baz.Boink.StartsWith("foo") - that would boil down to x.Field1.StartsWith("foo").

There is a facility you can tie into that controls how Sitecore converts types in queries (TypeConverters). Unfortunately, that does not solve the problem of disambiguating properties - the conversion only informs the engine how to convert the constant portion of the query (in this case, the string.

Sitecore LINQ does not care if the return entity type is mapped to a valid object for it

If you execute a query against a type, say the MappedClass type in the previous example, the LINQ layer will map all results of the query against the MappedClass type. Sounds great, but be careful - it will also map results that may not have the expected template to map to the MappedClass type onto it.

For example, suppose I made a model for the Sample Item template that comes with Sitecore. Then I queried the whole database as SampleItem. Out of my results, probably only two are really designed to be mapped to my SampleItem - the rest will have nulls everywhere. This is potentially problematic if you forget to be specific in your queries to limit the template ID to the expected one.

You must enumerate the results before the SearchContext is disposed

If you’ve ever dealt with NHibernate or other lazy-loaded ORMs, this might make perfect sense to you. A typical search query method might look something like this:

public IEnumerable<MappedClass> Foo()
{
    var index = SearchManager.GetIndex("sitecore_master_index");
    using (var context = index.CreateSearchContext(SearchSecurityOptions.EnableSecurityCheck)) {
        return context.GetQueryable<mappedclass>().Where(x=>x.Name == "foo");
    }
}

Can you guess the problem? IEnumerable doesn’t actually execute until you enumerate the values (e.g. run it through foreach, .ToArray(), etc). If you return the IEnumerable<mappedclass>, it cannot be enumerated until the SearchContext has already been disposed of. Which means that will throw an exception!

To avoid this problem you need to either:

• Return a pre-enumerated object. Usually the simplest form of this is simply to return the query with .ToArray() at the end, thus enumerating the query into an array before the context is out of scope. Warning: This also means that you should filter as far as possible within the query, including paging, especially with large result sets.
• Execute all the code within the context's scope. This is probably less desirable as it either means spaghetti code in your codebehind, or a request-level context manager like NHibernate sometimes does.

The object mapper uses cached reflection to set each mapped property

Yes, it uses “the R word.” Reflection is great, but it’s not all that fast. This is fine if you’re running optimized code: you’ll want to perform pagination and filtering within Lucene, and avoid returning large quantities of mapped objects. The performance is dependent on both the number of objects and the number of properties you have on each object as each property results in a reflection property set.

I would suggest trying to keep the number of mapped objects returned under 100 in most cases, and/or using output caching to minimize the effect of reflection.

It’s also possible to patch the way the mapping code works (I have a reflection-less proof of concept that in relatively unscientific tests is about 50-100x faster than the reflection method. It enforces some code generation requirements however and is not as general purpose or as simple to use as the reflection method.

It’s still awesome

While this post has largely focused on unexpected gotchas around the Sitecore LINQ provider, it’s actually pretty darn nice once you get used to its quirks. It’s certainly loads easier to use than any previous Lucene integration, and the backend extensibility allows for all sorts of interesting extensions and value storage.

It’s been announced that I was selected as a Sitecore MVP for 2013! Some of the folks at the office whipped up a neat infographic showing where all the MVPs this year are from - check it out.