Steve Sanderson’s blog

Wow, they’re really taking this operating system to the next level.

… and perhaps even more surprising:

Is there anything Vista can’t do?

In a recent post, I covered a few strategies for using LINQ to SQL in a traditional multi-tier architecture. The winning strategy for me, so far anyway, is DataContext-per-unit-of-work - it’s both convenient and safe. Or so I thought.

By default, child collections in LINQ to SQL are lazy-loaded. That is, they’re not loaded until they’re required, then the child entities are populated on a one-by-one basis. In the worst case this causes the “Select N+1” problem. If we’re not going to use the child entities, great, we win, but if we know we are, it’s better to load them all in advance in a single query.

RIP : “Including” operator

Before the March 2007 CTP of LINQ to SQL, we saw plenty of demos and code snippets using the “Including” operator. It looked like this:

var query = (from customer in MyDataContext.Customers select customer)
                .Including(customer => customer.Orders);

That was a nice, concise syntax to invoke eager loading selectively (i.e. on a per-query basis). It’s quite similar to NHibernate’s ICriteria.SetFetchMode(”Orders”, FetchMode.Eager) mechanism for controlling per-query loading strategies.

Unfortunately, LINQ to SQL’s “Including” syntax was killed in the March CTP. Random forum comments and blog posts imply it was removed because it was either too buggy or just didn’t fit with their general architecture. Unfortunately we’re now a bit stuck if we want eager-loading on a per-query basis.

The official solution: DataLoadOptions

First it was called DataShape, then it was renamed to DataLoadOptions. You use it like this:

MyDataContext context = new MyDataContext();
DataLoadOptions options = new DataLoadOptions();
options.LoadWith<Customer>(c => c.Orders);
context.LoadOptions = options;
var query = from c in context.Customers // ... etc

Not exactly elegant, but it can be a powerful way to control the “shape” of the data that’s returned. Note the shape.AssociateWith() method which lets you control the ordering and filtering of related entities too.

Unfortunately, changes to DataContext.LoadOptions affect the entire DataContext (obviously), and once assigned, cannot be changed during the lifetime of the DataContext (for consistency). This is completely at odds with the DataContext-per-unit-of-work strategy, as different subtasks within the same unit of work now have unwanted dependencies on each other.

Trying to use DataLoadOptions safely, and failing

I’m not yet convinced that there is a good way to control fetching strategies in LINQ to SQL on a per-query basis, since DataLoadOptions is the only control we have and that only works on a per-DataContext basis. If you don’t need deferred execution in a particular case, you can do this:

public static IList<Customer> CustomersWithOrders
{
    get
    {
        return Customers
                .Select(c => new { Customer = c, Orders = c.Orders })
		.ToList()
                .Select(pair => pair.Customer)
		.ToList();
    }
}

Here we load pairs (Customer, Orders), then from that project back to simple Customer objects. Since the relevant Order records are already loaded, no further queries are needed to evaluate customer.Orders for these customers.

If anybody can help me out with a better solution, i.e. that retains the deferred IQueryable semantics, I would be really glad to hear it. Until then, this is pretty frustrating. It may be that I’m on completely the wrong lines, but the LINQ-to-SQL multi-tier story doesn’t yet appear to have a happy ending.

SQL Server in the future

One limitation when eagerly loading object graphs is the dimensionality of the result set. To get customers and their orders, we have to JOIN the two tables together, and waste a lot of space to represent the parent-child relationship in two dimensions, like this:

The other problem, besides wasting space, is that it’s hard to page over data like this. How many rows must you retrieve to ensure you get the first 10 complete Customer records? LINQ to SQL, like other ORMs, does some clever trickery to hide you from this problem, but it’s still wasteful.

What I’d really like to see from SQL Server is multi-dimensional result sets. You can only have 2-dimensional results sets today, but why stop there? Of course, it’s hard to visualise 6-dimensional results in Query Analyzer, but I don’t really care about that. If aggregation operations weren’t forced to collapse lists onto scalars, we could GROUP BY customer, and get the list of Orders as a child table.

This would map trivially onto an object graph and is very easy to page over. I guess there are lots of reasons why SQL Server doesn’t already do this. Perhaps someone with more knowledge of its architecture can explain.

By the way, this is exactly what LINQ-to-objects does when you use its “groupby” keyword, and it’s very slick.

According to ScottGu,

VS 2008 and .NET 3.5 include a ton of new features for ASP.NET development. We are planning to deliver even more ASP.NET functionality next year with a “ASP.NET 3.5 Extensions” release. The first public preview of this will be available for download next week on the web.

Next week’s ASP.NET 3.5 Extensions preview release will include:

• ASP.NET MVC: This model view controller (MVC) framework for ASP.NET provides a structured model that enables a clear separation of concerns within web applications, and makes it easier to unit test your code and support a TDD workflow. It also helps provide more control over the URLs you publish in your applications, and more control over the HTML that is emitted from them.

If your web application is even slightly bigger than the typical MSDN example, you’re almost certainly using some sort of multi-tier architecture. Unfortunately, as Dinesh Kulkari (MS LINQ Program Manager) explains on his blog, LINQ-to-SQL version 1 has no out-of-the-box multi-tier story.

So, if you’re going to use it, you’re going to have to get imaginative and write your own story.

What we already do

The typical multi-tier web application handles requests like this:

It’s a simplified view but covers the basic point, which is that to fulfil a request, the top layer makes multiple calls into the lower layers.

Using LINQ-to-SQL

Simple example: the user wants to rename a “Customer” entity. You want to do something like this:

void RenameButton_Click(object sender, EventArgs e)
{
   Customer myCustomer = ServiceLayer.CustomerEngine.GetCustomer(_customerID);
   myCustomer.Name = NameTextbox.Text;
   ServiceLayer.CustomerEngine.SaveCustomer(myCustomer);
}

Easy - you load the record, update it, then send it back to be saved. We made two calls to the service layer.

And that brings us to… the problem

Where do DataContexts come in? With LINQ to SQL, you’re always querying or saving changes on a DataContext, which is responsible for two things:

• Managing connections to the database
• Tracking changes on entities it created, so it knows which ones to write back to the database

Who creates the DataContext, and what is its lifespan?

You certainly can’t create and destroy a new DataContext within each service layer call, even though most trivial examples you’ll see do exactly that. You can’t do that because our Customer object above needs to remain attached to the same DataContext during its entire lifetime. If we created a new DataContext during the SaveCustomer() call, the new context wouldn’t know what changes had been made to the customer, so wouldn’t write anything to the database.

Somehow, we have to manage the lifespan of a DataContext across multiple service layer calls. This is the “multi-tier story” we’ve been missing. There are three most obvious mechanisms I can think of.

Story one: DataContext creation in UI layer

You can, if you want, create and destroy the DataContext objects in your ASPX-code-behind event handlers, passing the context into each service layer call. This allows the DataContext to span multiple calls, as follows:

void RenameButton_Click(object sender, EventArgs e)
{
   MyDomainDataContext dataContext = new MyDomainDataContext(CONNECTION_STRING);
   Customer myCustomer = ServiceLayer.CustomerEngine.GetCustomer(dataContext, _customerID);
   myCustomer.Name = NameTextbox.Text;
   ServiceLayer.CustomerEngine.SaveCustomer(dataContext, myCustomer);
}

Advantages of this approach:

• Fine-grained control over DataContext lifespan and reuse

Disadvantages of this approach:

• DataContext is exposed to UI layer (unless you wrap it in some sort of handle object), which encourages lazy non-separating-concerns coding
• Code smell throughout your application - any chain of calls through the service layers involves a “context” parameter passed down through each layer
• Tedious work of creating DataContexts all the time

Story two: DataContext per application

The opposite extreme is having a single, static DataContext shared across your entire application. You may instantiate it either on application start, or lazily, and your service classes may simply access it whenever they need database access.

Do not do this, because:

• DataContext isn’t thread-safe, as far as I know
• You lose isolation and cannot control when SubmitChanges() is called - concurrent requests will interfere with one another
• Memory leaks are pretty likely

Story three: DataContext per unit-of-work (i.e. per request)

The classic solution to the problem is slightly more tricky that the others, but achieves - almost - the best of all worlds. This is the typical solution advocated by many when using other object-relational mappers, like NHibernate.

If each HTTP request has access to its own private DataContext object which lives for the duration of the request, you can expose it to the whole data access layer (and not to the UI layer), knowing that related calls will use the same DataContext object and thus keep track of object changes properly. Also, you don’t get unwanted interaction between concurrent requests.

But where would you store such an object? The natural place is in the IDictionary HttpContext.Current.Items. That’s a storage area whose lifespan equals the lifespan of the request, and is private to that request. You could, therefore, set up a static helper method available to your whole service layer:

internal static class MyDomainDataContextHelper
{
    public static MyDomainDataContext CurrentContext
    {
        get
        {
            if (HttpContext.Current.Items["MyDomainDataContext"] == null)
            {
                HttpContext.Current.Items["MyDomainDataContext"]
                    = new MyDomainDataContext(ConfigurationSettings.AppSettings["connectionString"]);
            }
            return (MyDomainDataContext)HttpContext.Current.Items["MyDomainDataContext"];
        }
    }
}

Now, querying the database is as simple as:

var query = from c in MyDomainDataContextHelper.CurrentContext.Customers where // ...etc

Since the DataContext is available statically, there’s no need to construct new instances or pass them around from the UI layer downwards. The UI layer doesn’t even need to know that such a thing exists. How nice!

Stop! That’s still not good enough

The simple implementation of method three leads to a serious limitation. The service layers are now coupled to HttpContext.Current, which is not a good idea - firstly because you ought to be striving to make your service layers independent of the UI platform (in this case, the web), and secondly because it’s going to break in some cases. HttpContext.Current won’t be available to your unit test runner, for instance.

Fortunately we can fix this with a trivial implementation of Inversion of Control. Let’s define an abstract notion of a “unit of work datastore” in our service layer.

namespace ServiceLayer
{
    public interface IUnitOfWorkDataStore
    {
        object this[string key] { get; set; }
    }
    public static class UnitOfWorkHelper
    {
        public static IUnitOfWorkDataStore CurrentDataStore;
    }
}

Now, when our application starts, we can map this datastore to HttpContext.Current.Items, by adding code to Global.asax:

private class HttpContextDataStore : IUnitOfWorkDataStore
{
    public object this[string key]
    {
        get { return HttpContext.Current.Items[key]; }
        set { HttpContext.Current.Items[key] = value; }
    }
}
 
protected void Application_Start(object sender, EventArgs e)
{
    ServiceLayer.UnitOfWorkHelper.CurrentDataStore = new HttpContextDataStore();
}

… and then the references to System.Web.HttpContext.Current.Item in the service layer can be replaced with UnitOfWorkHelper.CurrentDataStore:

internal static class MyDomainDataContextHelper
{
    public static MyDomainDataContext CurrentContext
    {
        get
        {
            if (UnitOfWorkHelper.CurrentDataStore["MyDomainDataContext"] == null)
            {
                UnitOfWorkHelper.CurrentDataStore["MyDomainDataContext"]
                    = new MyDomainDataContext(ConfigurationSettings.AppSettings["connectionString"]);
            }
            return (MyDomainDataContext)UnitOfWorkHelper.CurrentDataStore["MyDomainDataContext"];
        }
    }
}

And we’re done! The service layer no longer has any dependency on or awareness of System.Web, and our unit tests can supply any IUnitOfWorkDataStore of their own creation by assigning it to ServiceLayer.UnitOfWorkHelper.CurrentDataStore.If you’re using a full-fledged Inversion of Control container, like Castle Windsor or Spring, you will no doubt register your IUnitOfWorkDataStore with it and access it that way. For the rest of us, this simple implementation works nicely.

So that’s all perfect then

Yes, I thought so for a little while. In the next post, I’ll explain some issues you’re likely to run into when using LINQ to SQL in this way. Problems that don’t happen with NHibernate…

Let’s imagine your web development team is going to move out of “classic” ASP.NET and into MonoRail/ASP.NET MVC/etc. We assume you have a good reason (”It’s new and exciting” is usually good enough for us devs).

What kind of issues are you going to face? What questions are going to keep coming up?

“… but how do I do postbacks?”

The loss of the postback model is going to hit a lot of web developers hard - there’s a big mindset shift to make. So how did we get here, and where are we going?

In the history of web development so far, there have basically been two major different approaches/philosophies to web UI.

1. Event-driven

Just like with native GUI apps, we imagine that when the user clicks a button, we can change the text on some label. When they change a value in a list, the “total” is updated. The fact that this stateful UI has to be transmitted over HTTP is just a technical obstacle to be overcome.

2. RESTful

Following the principles of REST and SOA, we understand that the cleanest web apps are stateless. We think in terms of requests and responses, working with HTTP rather than against it.

While RESTful apps are the fashion today, don’t forget that it’s the older technology. In 1997, the age of Perl, we had no choice. The first Event-driven platforms came later, achieving their statefulness through complicated abstraction layers (ASP.NET), fiddly scripting (AJAX), or abandoning HTML entirely (Flash).

Here’s another way of comparing the two mindsets:

* I know classic ASP.NET actually replaces the whole page on a postback, but the mental model is that it’s doing partial page updates. It just technically uses full page updates to achieve that.

Note I’m trying to keep AJAX out of this, because it’s just a way of strapping extra event-driven semantics on top of whatever other platform you’re using. I totally accept that it’s useful today, but in 3-5 years it will be gone and we won’t miss it.

So which should I use?

Another way to think about the progression of these approaches is like this:

The most effective web frameworks today are undoubtedly the highly-streamlined, designed-for-HTTP, MVC derivatives. I call ASP.NET “ineffective” because anyone who’s used it for more than a few years knows that the abstraction doesn’t really work, it tends to be too heavyweight, and the “page lifecycle” is the software equivalent of cholera.

What’s the future?

The future is not Ruby on Rails. It’s not ASP.NET MVC either, though that will hopefully improve our industry for the next 2-4 years and you should probably use it.

The future is, inevitably, obviously, undoubtedly, the question mark on the table above. It has to be an event-driven architecture since that matches both the end-user and developer’s instincts about UIs. But unlike ASP.NET, it will be thick-client, not a load of leaky abstractions. It may be so well integrated into the OS that users don’t think of it as being different to their local apps.

In other words, the future is client-side. Well, it’s going to be interesting anyway.