(w)asp.net - extending the world

The resultor pattern

Thu, 21 Mar 2013 05:00:24 GMT

In this short post belonging to the series about extending the world of Linq, I'll talk about an interesting way to leverage anonymous types through a pattern we use at work. We all know what they are, and we all know their most important limitation: once defined, an anonymous type cannot be passed to other functions or returned to the caller of the function which defined it (we are talking about language features, and we don't consider any technique based on reflection or other knowledge about the runtime internal structure of such types). It is quite common to write functions which return complex values, and if we want to make them reusable we need at some point to define some structure representing the result of the computation. We could use tuples, but they are not that good at communicating information, so we must surrender and define small DTOs just to represent those answers, just because anonymous type cannot cross the boundaries of a function. But is that completely true?

Let's suppose we need a function which parses string, and for each one we want to return a set of values. We could write something like this:

public static IEnumerable<DTO> Parse(
    this IEnumerable<string> source, 
    int someParam, string someOtherParam)
{
    return from item in source
           select new DTO
           {
               First  = item,
	       Second = item.Length,
	       Third  = new DateTime(2012, 1, 1),
	       Fourth = 42
           };
}

public class DTO
{
    public string   First   { get; set; }    
    public int      Second  { get; set; }    
    public DateTime Third   { get; set; }    
    public decimal  Fourth  { get; set; }    
}

The body of the Parse function is doing some stupid calculations, but that's not the point, what I want to show here is that for even for simple cases like this one we need to define a special class (DTO) just for the sake of returning four values for each parsed string. It feels like a "waste" of power. We need to define some static type to return our answers. And even more disturbing, it often happens that the caller does not really need all the values that we are returning, but just some of them.

But there is a way out. Type inference helps us to put a 'trick' in place and make anonymous types cross the function boundary. Let's look at our Parse function: each time it parses a string it 'calculates' 4 values and it packs them in our DTO to return them to the caller, but we could slightly change the point of view and say that the Parse function calculates those values and 'passes' them to the caller. This subtle change makes us think that we could do what we always do in such cases: call a function.

Let's change our Parse function signature like this:

public static IEnumerable<T> Parse<T>(
    this IEnumerable<string> source, 
    int someParam, string someOtherParam, 
    Func<string, int, DateTime, decimal, T> resultor)

In this new version we changed 2 things:

the return values is not an enumeration of DTOs (or whatever well-defined type), but IEnumerable<T>
the same T appears as part of the type of a new parameter we added to the already existing ones: this new parameter (a callback) is declared as a Func<...> with N+1 type parameters, where the first N types are the same types of the properties in the original DTO, and the last type parameter is T

This 'strange' signature is in fact very handy, and thanks to type inference we can modify our test call like this:

var parsed = from p in items.Parse(0, "wasp", 
                 (first, second, third, fourth) => new
                 {
                     First  = first,
                     Second = second,
                     Third  = third,
                     Fourth = fourth
                 })
             where p.First
                    .Equals("a", 
                            StringComparison.OrdinalIgnoreCase)
             select p;

The lambda expression we pass as the last parameter in the call defines an anonymous type, which thanks to type inference becomes the T of the generic Parse function. The complete body of Parse function could therefore be something like this:

public static IEnumerable<T> Parse<T>(
    this IEnumerable<string> source, 
    int someParam, string someOtherParam, 
    Func<string, int, DateTime, decimal, T> resultor)
{
    return from item in source 
	   select resultor 
           ( 
               item, 
               item.Length,
               new DateTime(2012, 1, 1), 
               42 
           ); 
}

It's very similar to the original version, but the new DTO() part has been replaced to a function call to the supplied Func<...> delegate, which is called resultor. So we now have moved the responsibility of creating a representation of our 'answer' outside of the Parse function, which now just calls someone capable of building such a representation. Thanks to type inference we can define the builder on the fly through a lambda, and use an anonymous type to contain the representation. We don't need the DTO class anymore, and one more advantage that we gain now is that we can easily handle cases where we are not interested in all the components of an answer. If for instance we just want to deal with the first and the fourth value, we can call Parse like this:

var parsed = from p in items.Parse(0, "wasp", 
                 (p1, _, ___, p2) => new
                 {
                     This  = p1,
                     That = p2
                 })
             where p.This
                    .Equals("a",
                            StringComparison.OrdinalIgnoreCase)
             select p;

I learned about this at work, and there we name it the resultor pattern. So far I did not find anything around describing this technique explicily, Bill Wagner in one of his books talks about something similar but in a more basic way. It adds one more piece to our toolbox which helps us move our coding more 'into Linq'. It could be used in other contexts too, but it really shines inside Linq queries.

Extending the world, related material

Thu, 01 Mar 2012 13:25:01 GMT

As promised, here you can find the source code relative to the discussions we made so far:

Source code

Please remember that this is not production code at all, it has the only purpose to give you the chance to easily try what I've been trying myself so far while writing this series.

I can also give you a few links where you can fine more formal and deeper material about the topics I'm blogging about in this series. I read a lot of stuff around functional programming and its influence over imperative ones, and especially over C#, and I think that Bart De Smet and Wes Dyer wrote some of the most interesting pieces I had the chance to find. It's a pity that both have not been blogging about such topics in the last years, but their writings are still actual and very interesting. From there it is easy to naviggate and find more stuff, but here you have a couple of links more which I think deserve some consideration in our context: functional programming for ~~dummies~~ imperative programmers :)

Enjoy :)

Extending the world, step 4

Tue, 17 Jan 2012 22:14:18 GMT

Last time we introduced a new simple exercise, where we have a computation which might fails and we want to to try to remove all the noise that the error handling introduces all around the algorithm. We talked about Monads (in a very simplified way) and we defined a way we could write our computation inside a Linq expression, using a Monad to isolate the unwanted error handling code. What we still have to do is actually implement this new Monad and see how we can leverage this concepts and strategies to reach our goal.

Following a few simple steps, we had arrived to this way to express our computation:

var r = from x in M.Unit(fi)
        from y in M.Unit(fj)
        from x in M.Unit(fk)
        select x + y + k;

We already said that this "would" compile as long as we provide the right methods to enable Linq to host and execute our code on our Monad. Let's first give a better definition about our Monad, so that we will be able to figure out how we should implement all the details. Basically our Monad will have to isolate the error handling code from the actual computation, which we should be able to express in a compositional way. The idea is that the Monad will be the only actor aware of the error handling, and it will be in charge of keeping track of computation executions and of their actual success or failure. We already said that a Monad by definition has to supply unit and bind, and it is actually bind that will takes care of the specific behavior of our Monad.

We still have to give this Monad a name, and given its description I think that ValueOrError is a good one. Let's write a (very) basic implementation of a class that is able to hold a value or an error condition:

public class ValueOrError<T>
{
	private readonly Exception error;
	private readonly T value;
	private readonly Func<T> eval;
	internal ValueOrError(Func<T> eval)
	{
		this.eval = eval;
	}
	internal ValueOrError(T value)
	{
		this.value = value;
	}
	internal ValueOrError(Exception error)
	{
		this.error = error;
	}
	public T Value
	{
		get { return eval != null ? eval() : value; }
	}
	public Exception Error
	{
		get { return error; }
	}
	public bool HasError
	{
		get { return error != null; }
	}
	public bool HasValue
	{
		get { return !HasError; }
	}
}

We wrote a generic type, and we specified a costructor to hold an error, and 2 constructor to store respectively a value of type T, or a function which is able to produce a value of type T. We could make this code more interesting, but it is enough for our exercise. We also did not manage any specific execution behavior yet, we will do that later. We already know that, to be a Monad, a type must supply 2 special functions, unit and bind. Let's write those as extension methods of ValueOrError<T> inside some static class (ValueOrErrorExtensions for example), and let's start with Unit:

public static ValueOrError<T> Unit<T>(this T source)
{
	return new ValueOrError<T>(source);
}
public static ValueOrError<T> Unit<T>(this Func<T> source)
{
	return new ValueOrError<T>(source);
}

For Unit we have 2 overloads which are mapping the 2 constructors dealing with "values" we have in ValueOrError<T>, and these overloads fulfill the unit mission: put a value inside the Monad.

Let's move on to bind, here we need an implementation which has to have the signature we described in the previous episode:

M<TR> Bind<TS, TR>(this M<TS> source, Func<TS, M<TR>> selector)

A bind is supposed to accomplish these steps:

extract the value(s) of type TS from the input ValueOrError<TS>
apply the supplied function, which transforms the value(s) of type TS in value(s) of type TR and then puts it (or them) in ValueOrError<TR>
return the just created instance of ValueOrError<TR>

This is what we expect from a bind, but while doing this job this method can actually do more things, and in our scenario we know that we need someone who handles errors in a proper way. Let's define what "proper" is:

Bind should check if the source ValueOrError<TS> already contains an error, if yes we know that we should not go on with further processing, so we just create an instance of ValueOrError<TR> (because we have to go from TS to TR), using the constructor which receives an Exception and supplying the error condition of the source ValueOrError<TR> as its input; this way we are stopping the execution chain and passing along the already generated error condition, although wrapped in a new Monad
on the other hand, if the source is a valid value, we can go on with the computation, so we just need to extract the source value and pass it to the selector function which puts the value inside a Monad of type ValueOrError<TR>; if we are in a case where the Monad has been supplied a Func<T> as a constructor parameter, reading its value will actually execute the supplied function, so it is enough to do that inside a try...catch block and, in case of error, return an instance of ValueOrError<TR> containing the thrown exception as the error condition

This is definitely easier to read in code:

public static ValueOrError<TR> Bind<TS, TR>(
        this ValueOrError<TS> source, 
        Func<TS, ValueOrError<TR>> selector)
{
	if (source.HasError)
		return new ValueOrError<TR>(source.Error);
	try
	{
		return selector(source.Value);
	}
	catch (Exception ex)
	{
		return new ValueOrError<TR>(ex);
	}
}

It's very simple, but it's clean and isolated in one point. We could have also moved the try...catch block directly inside the ValueOrError<T> class, but this is a detail and it is not so relevant for our discussion. What's relevant is how to trigger this mechanism, and we already saw that we are going to have this code run inside Linq expressions. Let's write the simplest Linq expression we can, as we did the last time:

var r = from i in ValueOrError.Unit(fi) select i;

which is equivalent to this:

var r = ValueOrError.Unit(fi).Select(i => i);

We need to implement Select, and we can take advantage of Bind and Unit to do it in a simple way:

public static ValueOrError<TR> Select<TS, TR>(
        this ValueOrError<TS> source, 
        Func<TS, TR> selector)
{
	return source.Bind(s => selector(s).Unit());
}

Again very simple, and our knowledge about how to go from T to ValueOrError<T>, about how to go from ValueOrError<TS> to ValueOrError<TR>, and most important about how to deal with errors, is hidden inside Unit and Bind. The result of our computation will be an instance of ValueOrError<T> containing either the computed value fi(), or the error condition encountered during the execution. This was very trivial and we might have done it without all the Monad theory and the Linq support, but as soon as we try to implement more complex computations we start appreciating the added value of these mechanisms. Let's go back to our "complex" algorithm:

var r = from x in ValueOrError.Unit(fi)
        from y in ValueOrError.Unit(fj)
        from x in ValueOrError.Unit(fk)
        select x + y + k;

If we'd try to write the expression as a composition of method calls, we should come up with something like this:

var q = ValueOrError.Unit(fi)
                    .SelectMany( i      => ValueOrError.Unit(fj), 
                                (i,  j) => new {i, j})
                    .SelectMany( @t     => ValueOrError.Unit(fk), 
                                (@t, k) => @t.i + @t.j + k);

Now it is clear that we need to implement SelectMany for our Monad, and at the same time we can appreciate that the query syntax is much more readable than the methods chain, which by the way makes explicit a temporary range variable @t that the query syntax manages to keep hidden. Let's implement SelectMany, in terms of Unit and Select (which is based on Unit and Bind):

public static ValueOrError<TR> SelectMany<TS, TC, TR>(
        this ValueOrError<TS> source, 
        Func<TS, ValueOrError<TC>> matcher, 
        Func<TS, TC, TR> selector)
{
	return source.Bind(s => matcher(s).Select(t => selector(s, t)));
}

If we analyze its signature and we forget for a moment about the last parameter, we can see that it matches the Bind signature, and the last parameter just adds an additional level of indirection, which brings us some more flexibility and actually enables the nice query syntax we are trying to use. SelectMany itself is just using Bind and Select, and there are several amazing things around its implementation:

these very short and neat lines of codes inside our extension methods enable such a clear and coincise Linq expression describing our computation
if you try to follow every single method call to get to the final result, you will immediately notice how complex the call sequence can get, and all this complexity is handled by 4 method, where 3 of them are just one line of code, and the more complex one is just an if statement followed by a function call inside a try...catch block
we already said that we wrote Select and SelectMany in terms of Bind and Unit, but what we did not said is that, following this pattern, whatever Monad you will be writing you will just need to reimplement Bind and Unit, whereas Select and SelectMany will stay exactly the same!
if you think that methods like SelectMany are hard to understand, you should just use the trick to "follow the types" and you will be able to write that line of code by yourself; let's try together:
- SelectMany and Bind have the same first argument and the first return type, so we can start writing this: return source.Bind(...);
- Bind expects a second argument of type Func<TS, ValueOrError<TR>>, and we have matcher and selector with signatures which do not help, but we also have Select available, which goes from ValueOrError<T1> to ValueOrError<T2>, so we can call matcher on TS, obtaining a ValueOrError<TC>, and then call Select on it to go to ValueOrError<TR> thanks to selector, which can be called from "inside" Select supplying it the instances of TS and TC that at that point would be available

So far we wrote nice and clean code, but we didn't really analyze what this code does when dealing with real scenarios. Let's try to follow a chain of computations considering a case where our 3 functions succeed:

fi is put inside a ValueOrError<...> (let's call it VOE^fi, and we'll implicitly use this notation for similar actions in further steps)
SelectMany is called over VOE^fi
inside SelectMany, Bind is called over VOE^fi
VOE^fi is checked for errors, it does not have any therefore we go on
Value is extracted from VOE^fi => fi() is called, let's call its return value fi! (we'll implicitly use this notation for similar actions)
fi! is passed to Bind's selector, which at this point was matcher(s).Select(t => selector(s, t)), which via substitution becomes ValueOrError.Unit(fj).Select(t => selector(s, t)), which eventually becomes (VOE^fj).Select(t => selector(s, t))
Select is called over VOE^fj, which means (VOE^fj).Bind(s => selector(s).Unit())
Bind is called over VOE^fj, we do not detect error conditions so we call fj(), obtaining fj!, and we pass it to s => selector(s).Unit(), which by substitution is ((i, j) => new {i, j}).Unit(), where i = fi! and j = fj!; the anonymous type contains the first 2 results and it is put back in the ValueOrError<...> via Unit()
we repeat the same steps with the second SelectMany, this will let our anonymous type pass through, fk() will be called, and at the end of the same chain we just described we will get to (@t, k) => @t.i + @t.j + k, where @t is our previous anonymous type and k = fk!: the actual sum will be performed

It is not exactly easy to explain, I hope I did not loose you here, if you try to follow such a code with a debugger you will be able to track down what's happening. But we built all this infrastructure to deal with errors, so we should show an example of such a case, and let's suppose the very first computation fails:

fi is put inside a ValueOrError<...>
SelectMany is called over VOE^fi
inside SelectMany, Bind is called over VOE^fi
VOE^fi is checked for errors, it does not have any so we go on (it's not been computed yet)
Value is extracted from VOE^fi => fi() is called and this time it fails, so an exception is raised and trapped, and a VOE^ERROR is returned
because fi() failed, selector is never called, so Bind returns VOE^ERROR to SelectMany, which returns it to its caller
the second SelectMany is called, and we soon reach again a Bind call, but this time we detect an error condition on the source (VOE^ERROR) and therefore we immediately return a new VOE^ERROR containing the same error as the source one, and we skip again the selector call, so SelectMany will return VOE^ERR as a result of the whole computation

Easy, isn't it? :) In both cases we will receive a ValueOrError<T> instance, and we will just have to examine if we have met an error condition or not. What also interesting is the way we actually calculated our computation, we used the term "substitution" several times, and this is another clue that what we are doing is quite functional: our computations can be treated as mathematical expressions where you calculate intermediate results and then just use them to replace parts of expressions (lambda calculus, I guess...).

If you are still with me, I am sure you would be curious to try something like this, and you might want to have some real code to test. Well, my next post will be dedicated to deliver you a sample project that I'm preparing, along with some reference material you can use to investigate more the topics of this series.

Extending the world, step 3

Tue, 17 Jan 2012 22:14:07 GMT

So far in our series we have seen how Linq is a world that can be extended if we craft our types the right way. We learned that Linq is definitely not just for IEnumerable or IQueryable, and I'm more and more convinced that what's most interesting about Linq is the way it guides you to write more functional code. I'm not a functional programmer (yet), so please do not expect too much knowledge about that from my posts, but I'm slowly gaining a more functional view on things, and I'm feeling that it's a good thing to have. There are a few aspects of functional programming that Linq tries to enforce:

no side effects: the code we write should not deal with the production of anything which is not strictly related to our result, and which in some way modifies a "global state"
immutability: what's defined inside a Linq expression should be immutable, and this is a good thing because it helps avoiding side effects (see previous point)
declarativity: inside Linq we do not think in terms of state and statements, but in terms of declarations: we tend to describe more what we want from a computation, and less how we want any result to be computed
compositionality: within a Linq expression we can compose computations, chaining hen and mixing them thanks to the operators we have available (of course the types we are using in our expression must support them), resulting in better readability of the intents we have (see previous point)

There are more, of course, but let's stay with these and let's try to expand them. To me it is easier to do it with samples, so here we go.

Let's imagine that we have to compute 3 values and to sum them, but let's say that our 3 inputs are calculated or retrieved by someone who may fail (a db connection? a web service? you name it). Those are precious resources, and we do not want to waste them, so in a sequential world (too early to talk about parallelism or async, sorry... :) ) we would prefer to avoid the subsequent calls if a previous one failed, invalidating the whole computation. At the same time, if something went wrong, we do not want the whole process to fail, but we want to collect information about the error that occurred. How would you write such a code? Maybe something like this:

int r = 0;
Exception error;
try
{
	r = fi();
}
catch(Exception ex)
{
	error = ex;
}
if (error == null)
{
	try
	{
		r = r + fj();
	}
	catch(Exception ex)
	{
		error = ex;
	}
}
if (error == null)
{
	try
	{
		r = r + fk();
	}
	catch(Exception ex)
	{
		error = ex;
	}
}
if (error == null)
{
	...
}
else
{
	...
}

Not so good looking... I will not explain the details, the code is trivial, and actually it could be improved a little, but still you would have something where you would not easily spot the true essence of the algorithm, which indeed is:

int r = fi() + fj() + fk();

What's really noisy here is the error handling, which is distracting you from the heart of the computation, but you cannot avoid it and it makes your code less readable. You would like to clean it up, moving the error handling "somewhere else". Something like the well known AOP approach, but you still want to check if something went wrong in the context of your code. And if you go back to our list of functional characteristics, you can also look at the errors as "side effects", which should really be avoided, or at least isolated from our computation.

A functional programmer at this point would say the magic world: you need a Monad! He might also say that you need a Maybe Monad, or something similar. What a Monad really is, it is still not 100% clear to me, and not being a functional programmer (yet) I try to bring that idea in the space I know better and with the concept I'm used to deal with. So, to me a Monad is a "container", or a "decorator", which is able to "augment" some other instance, or set of instances, of some type T. Let's call out Monad M<T>. A Monad should also, in some way, supply 2 special "static" functions:

return, or unit: such a function, given a type T, returns an M<T>; its job is to take something and "put it" into the Monad
bind: such a function should receive an instance of M<T>, and a function that goes from T to M<R>, and returns an instance of M<R>; in other words, it's job is to "extract" the value(s) of type T from M<T>, then to apply the function that transform the value(s) of type T in value(s) of type R and puts it (them) in M<R>, finally returning the instance of M<R> created

Let's write the signatures:

unit: Func<T, M<T>>
bind: Func<M<T>, Func<T, M<R>>, M<R>>

What's important is that both know the internals and the mechanics of the Monad, and they should be the only ones to know such details. We'll get back to them later, for now let's supposed we have them, and let's use unit like this:

int r = M.Unit(fi) + M.Unit(fj) + M.Unit(fk);

What did we do here? We simply put each of our functions (in a Monad we can put everything, so functions are welcome to) inside an instance of our Monad, and then we sum those instances as if we were still dealing with functions return values. Lovely... but this does not work. It does not even compile, of course, we do not have the + operator defined on our Monad, and we cannot define one because M<T> is generic, and we don't know how to implement the sum. And then, what about implementing all the other operators (-, *, /, ...)? We do not want to do that, of course, the goal of our Monad is to deal with whatever is NOT the computation. So what?

Enter Linq. As the title of this series says, let's extend the world (Linq) to understand our Monad. We already learned how to extend Linq, we just have to supply the right functions to implement the right operators. But first, why do we need Linq? Because Linq has a nice syntax that let us declare computations in a clean way, and because incidentally (well, probably not so incidentally) the select operator would help us a lot. Let's see it's signature:

Func<M<T>, Func<T, R>, M<R>>

It's similar to "bind", but it's not the same: select receives an "augmented" value (or values), extracts it (them) from the "augmenting" Monad, applies to the value(s) a function which transform it (them) to something of type R, and finally puts it (them) in M again, returning it to then following piece of computation. We have 3 interesting points here:

the function that goes from T to R is supplied by us, we are the ones knowing how to do that
select will just extract things from Monad, then will call us to transform them in something else, and finally will put the result back in the Monad
select will use bind to do part of its job, and this will keep the deep knowledge of the internals of the Monad inside bind (and unit): we will say that select is written in terms of bind

Let's simplify for a moment our algorithm to this:

int r = fi();

This seems stupid, but later you will understand why I did it. To do this calculation "inside the Monad", before we tried this:

int r = M.From(fi);

but this would not compile. Let's suppose we have the right implementation of the extension method Select for our M<T>, then we can try this:

var r = from x in M.From(fi) select x;

and this works! Actually, this expression is equivalent to this:

calling unit: put fi() inside M
calling select: extracting back fi() from M, "do something" with fi(), give the result back to us to "transform it" in something else (the select x part, which in this case is just leaving the value as it is), and putting back the result in the Monad, which is returned to the left

You will have also noticed that now the lefthand side of the computation is not int r anymore, but var r, where var actually stands for M<int>. So now we are holding an instance of a Monad over our original type int, and this instance of the Monad will contain the result of our computation... well, maybe :) We'll dig more about this next time.

The final piece to get to a meaningful computation is SelectMany, whose signature is exactly what we need to join different pieces. As long as every part is contained in the same type of Monad M<T>, SelectMany enhances select and enables us to write this:

var r = from x in M.From(fi) 
		from y in M.From(fj) 
		from x in M.From(fk) 
		select x + y + k;

I'm sure that you can see how readable this is, the algorithm (x + y + z) is very clear and the noise introduced by M.From and by the query syntax is not too bad, and actually gets almost invisible when the computations get more complex. Probably still too noisy for someone using F# or Haskell, but for us in C# it's not bad at all.

This episode it's been more about the concepts, and actually I did not say anything concrete about how to solve the error handling problem, this will be the topic of the next episode, where we will implement a specific Monad to solve this issue. Its implementation will also allow us to appreciate how unit, bind and select are implemented to accomplish our goals.

Extending the world, step 2

Tue, 17 Jan 2012 22:13:56 GMT

In the previous episode of this short series, I showed how you can define a type in a way that enables it to participate to Linq queries. Linq query syntax defines a set of operators, which are mapped to methods that types involved in a Linq query must provide. If they natively do not, we can still make them work because extension methods come to the rescue. How those methods should be named and shaped is clearly documented, and it would be too much to be explained here, so I will skip these details. I'll go back to our exercise and I'll use it to give you some more details about a couple of interesting points.

So we have our Twitter client:

public static class NiceTwitterClient 
{ 
    public static Searcher Search(string from) 
    { 
        return new Searcher(from); 
    }
}

returning an instance of a Searcher type, and then we have our query:

var twits =  from t in NiceTwitterClient.Search("@wasp_twit")
             where t.Contains("#csharp")
             where t.Contains("cats")
             orderby t.Date descending
             select t;

When we write:

var twits =  from t in NiceTwitterClient.Search("@wasp_twit") ...

we retrieve an instance of Searcher through our call to Search and we "put it into" Linq. With the first part of the expression from t we also define a range variable t which Linq will use when building the subsequent calls. Because we must be compliant to the query syntax, Linq will try to map its operators to corresponding Searcher methods (or extension methods), therefore they must have appropriate definitions. In our sample query, we go on with:

... where t.Contains("#csharp") ...

which Linq translates into this:

... NiceTwitterClient.Search("@wasp_twit")
                     .Where(t => t.Contains("#csharp")) ...

Here you can see how the range variable t is used to build a lambda expression. So, when we compile our code, the compiler will look for an available method named Where on type Searcher. C# syntax allows us to use lambda expressions to represent both delegates of certain types and code expressions built on the same types, so thanks to C# type inference the code above would match both the following definitions (I wrapped in square brackets the code you would add in case of extension methods):

public [static] Searcher Where([this Searcher source, ] 
    Func<SearcherClause, bool> filterClause) 
    { ... }

public [static] Searcher Where([this Searcher source, ] 
    Expression<Func<SearcherClause, bool>> filterClause) 
    { ... }

Our Searcher happens to be implementing the second version of the Where method, so the compiler will happily match that one and we will get called there. Our method will return a (possibly new) instance of Searcher, and we will be ready for a new pass, looking again for the right match. In our sample query we go on with another where clause:

... where t.Contains("cats") ...

which will be resolved as before. You probably already noticed that I specified 2 consecutive where clauses, instead of doing it just once and joining the 2 predicates in a logical and, and you might be wondering why. Well, simply because I'm lazy :) Let's see how our Where method is implemented and everything will be clearer:

public Searcher Where(Expression<Func<SearcherClause, bool>> filterClause)
{
    var mce = filterClause.Body 
              as MethodCallExpression;
    if (mce != null)
    {
        var a = mce.Arguments[0];
        if (a is ConstantExpression)
            return new Searcher(((ConstantExpression)a).Value.ToString(), this);
        if (a is MemberExpression)
        {
            var me = a as MemberExpression;
            if (me.Expression is ConstantExpression)
            {
                var value = ((ConstantExpression)me.Expression).Value;
                if (value is string)
                    return new Searcher(value.ToString(), this);
                var type = value.GetType();
                var fi = type.GetFields()[0];
                return new Searcher(fi.GetValue(value).ToString(), this);
            }
        }
    }
    throw new NotSupportedException(
        "When doing Where I understand calls to Contains method of SearcherClause.");
}

The implementation deals with expression trees, and you already know that those can be arbitrarily complex, and they easily get very hard to analyze. I built a naive implementation which just handles a couple of simple cases, and operation like logical and/or are not managed at all, so I just added an implicit semantic to the Where methods which states that composing Where call has the same effect as building a logical and on the involved expressions.

There are a few more interesting things to say here. The first one is about the return value of the Where method, which is an instance of the Searcher type which receives a reference to the current Searcher (the this parameter), holding it along with the specified filter values extracted from the expression. This way we build a "chain" of Searcher instances. This is just one of the possible ways to deal with the problem, probably the easier and therefore the most suited to an exercise like this. That chain will be used at the end of the computation to determine the true call to the "real" Twitter client.

The second interesting thing is a peculiar view we might adopt to analyze the code. Our Where method might be seen as a way to navigate through types, and in this case we may say that Where "goes from Searcher to Searcher", so it actually keeps us in the same context. If we move forward to the next line in our query:

... orderby t.Date descending ...

we are asking Linq to perform an orderby descending, which it will translate into this:

... searcher_from_previous_where.OrderByDescending(t => t.Date) ...

This will match to this implementation:

public Sorter OrderByDescending(Expression<Func<SorterClause, object>> sortClause)
{
    return order(sortClause, false);
}

private Sorter order(Expression<Func<SorterClause, object>> sortClause, 
    bool ascending)
{
    var ue = sortClause.Body as UnaryExpression;
    if (ue != null && 
        ue.Operand is MemberExpression)
        return new Sorter(((MemberExpression)ue.Operand).Member.Name, ascending, this);
    throw new NotSupportedException(
        "When doing OrderBy I understand UnaryExpression only, accessing members of SorterClause.");
}

Here we can see that we are again doing some expression tree analysis (even more naive that before), and that our OrderByDescending "goes from Searcher to Sorter", changing our context. Our Sorter type is quite similar to Searcher, but it actually defines a new context, letting us know that we moved from the "filter" phase to the "sort" phase. Sorter does not offer Where methods anymore, but only Select methods, which will be analyzed later, and this way it guides us to the right path while building our query, letting us take advantage of features like Visual Studio Intellisense which will suggest us only appropriate methods, and leveraging the compiler support which will emit errors in case we try to use unsupported Linq query operators.

When we are done with sorting, our Linq query is dealing with a Sorter instance, and our implementation offers us just one "way out": using the select operator. A Linq query must end with a call to a select operator (or equivalent, like for grouping), so we should just add appropriate implementations of Select returning an output which is meaningful for us, and in this case an IEnumerable<T> will be just fine. The actual T will be left generic in an implementation, whereas in another available overload it will be closed to a simple Twit type we expose to make things simpler. Under the hood, the actual implementation of IEnumerable<T> will just be an internal type aware of all the parameters we've been collecting along the chained calls, and on the act of "enumerating" it these parameters will be used all together to call the original ugly client, to receive its response and eventually to convert it in an enumeration of Twit instances (possibly transformed via the correspondant Select overload).

I guess this naive exercise should demonstrate how Linq could be extended, in fact we shaped our types in order to have them nicely integrated with Linq query syntax. But there are many more different things we can do in this context to reinforce our initial concept of Linq as a "world" where we can put our things and have them play nicely. We'll see some more of them in the next episodes :)

Extending the world, step 1

Tue, 17 Jan 2012 22:13:42 GMT

After the general introduction to this short series of post, I would like to set up a sort of roadmap, defining the goals that I have while writing this stuff. But first, a sort of disclaimer: I'm not a big expert of these things at all, and I'm quite new to this kind of approach, but that's exactly the reason why I want to write about it, to illustrate things from the point of view of someone who's really learning to use this new mindset and the related techniques, because I know there are a lot of guys like me out there :) The code I will show will be written by myself in a sort of "learn by trial" process, but it will be strongly inspired by several sources, which I will try to mention as most precisely as possible. If someone reading these posts will find errors or omissions his comments will be welcomed.

The points I would like to demonstrate are:

how you can extend Linq to understand new types
how you can extend your types to make them live "better" inside Linq
how you can apply both strategies to write more functional code

The first 2 point will be mostly treated together because they are deeply related, I will try to make clear which side of the medal I will be looking at. And, by the way, I will not write at all about the process of writing a Query Provider, there is a lot of material about it and at the same time it's a hard problem which indeed I had to solve already, I would never be able to talk about it in a useful form in just a few posts.

Let's start. First important statement: Linq is not for IEnumerable and IQueryable only. Linq is there for any type you want, and its query syntax is just "syntactic sugar" around the composition of chained calls over available methods, which might be extension methods too. As long as Linq is able to find (extension) methods with appropriate signatures for the types of the objects you are supplying as "sources", it will be able to understand and execute the query you are writing. You already know that the query syntax understands a limited sets of methods, but the approach is still valid even on methods that are not among the ones Linq handles, as long as those methods are written with "composition" in mind. What that's mean? Well, to me it's simply that they should be "functions": they receive inputs and always produce 1 output (not out or ref parameters here). If the output is of a type Linq is able to understand, you will be able to keep on chaining things as you like, writing "computations", as they are called. Your code starts moving from an imperative style to a declarative one, and the advantage here is that the concern is not on the "how" of things, but on "what" you want from a computation. All the details get hidden inside the functions, and you just declare how you want to compose them, supplying "things" to customize their behavior. This classical imperative code:

foreach(var i in getNumbers)
    Console.WriteLine(i);

could be rewritten like this:

getNumbers.ForEach(Console.WriteLine);

where ForEach is an extension method for IEnumerable<T>. You might be thinking: there's no such method on IEnumerable<T>, but only on List<T>. Right, but that's where the magic begin: you can write it easily in a static class, and then add the right using directive to have it available:

public static IEnumerable<T> ForEach<T>(
    this IEnumerable<T> source, 
    Action<T> action) 
    {
        foreach (T o in source) 
        { 
            action(o); 
            yield return o; 
        }
    }

What you are doing is defining a function in a composable way, extracting a common task (enumerating elements to do something on each one of them) and make it available for future uses. This is a quite stupid case, and I don't mean you should really use this kind of ForEach, in future I will try to show you more complex scenarios where you will hopefully get to appreciate the approach better. Such a function allows us to seamlessly integrate "lateral" actions in our computations:

var q = getNumbers()
        .ForEach(Console.WriteLine)
        .Where(i => i%2 == 0)
        .OrderByDescending(i => i)
        .Select(i => 3*i);

I guess you can already see some advantage of this approach: such a query would be much more "noisy" to write in the old plain imperative style, and our "insertion" of a cross cutting concern, like outputting an intermediate result to the console, has been easy and clean thanks to our extension method. I have more good news about it: we don't have to start writing dozens of extension methods for every little operation like this one, because out there we can find several open source libraries full of useful little functions we can use in our computations. I can recommend MoreLinq, but it's not the only one for sure, and sometimes it's just more fun to try to write your own extension methods.

So far I've been cheating, you are right, I said I would have talked about types which are not IEnumerable<T>. I just wanted to start easy, now let's do another baby step towards our goal. Imagine you have this Twitter client library you found somewhere, which works just fine but exposes an ugly API, and when you have to use it your code starts look just bad. This library is as stupid to use as in the next example:

var client = new UglyTwitterClient();
var twits = client.GetTwitsFromSomeone(
    "@wasp_twit", 
    "date desc", 
    "#csharp", "cats");

Well, maybe it's not so ugly, I don't know, but let's say it is. For sure there are aspects, like the properties of a twit, which could be more strongly checked than just putting them in strings. And we are querying a system without using Linq, no way! We would prefer to write something like this:

var twits = from t in NiceTwitterClient
                      .Search("@wasp_twit")
            where t.Contains("#csharp")
            where t.Contains("cats")
            orderby t.Date descending
            select t;

Ok, it's easy, you have this method Search which returns an IEnumerable<string> so I can call Contains(), and then... wait, we have a Date property on the range variable, so it must be some type, maybe a Twit type we can enumerate in order... enumerate where? On the client? No, I cannot simply download all the twits of this guy locally to filter them! Gosh, I have to implement IQueryable, parse the expression tree, manage all the corner cases, do the remote call, unpack the data...

As I said, here we don't do IQueryable. Here we extend Linq with our types, and that's how this new Linq-enable client can be written. We need our NiceTwitterClient and its static method Search:

public static class NiceTwitterClient
{
    public static Searcher Search(string from)
    {
        return new Searcher(from);
    }	
    ...
}

Search creates an instance of this Searcher type, which will allow us to implement a "fluent interface" which will also be Linq-aware. To do so, our Searcher will expose the Linq methods we decide to expose: Where, OrderBy, OrderByDescending:

public Searcher Where(
    Expression<Func<SearcherClause, bool>> fc) 
{ ... } 
public Sorter OrderBy(
    Expression<Func<SorterClause, bool>> sc) 
{ ... } 
public Sorter OrderByDescending(
    Expression<Func<SorterClause, bool>> sc) 
{ ... }

Here we can notice another aspect of Linq extensibility, probably an expected one: extension methods are not the only way to join the Linq world, instance methods will do too. Those methods receive instances of Expression class, or "expression trees", and return instances of Searcher or Sorter types. The latter is very similar to Searcher, but it's meant to store info about the sorting clause we want in our query. Those types are really defining how our fluent interface evolves: Searcher exposes only these 3 methods and 2 overloads of a Select method, which will be discussed in details next time; Sorter exposes the Select overloads only. Linq query syntax is able to map them to the where, orderby, orderby descending and select operators, and environments like Visual Studio are able to help us with Intellisense. About this last point, this approach is even better than the IQueryable one, because extending through this well known interface forces us to deal with ALL the Linq operators, which will be shown as available by the Intellisense and should be handled as invalid, and that's possible at runtime only. Our approach makes Intellisense smarter, and moves the check of query correctness at compile time. Neat :)

Next time I will illustrate the implementation of this exercise, and please remember that it is an exercise indeed, the code will just illustrate the concepts and should not be considered as complete or "production ready" at all, but if you are new to this concepts and features I'm quite sure it will be stimulating.

Extending the world

Wed, 18 Jan 2012 21:36:37 GMT

After a few months I'm back to write on my blog, but to be honest I had a good excuse, since my wife and I have dealt with a move abroad (I'll talk about it separately), which is not exactly something trivial. However, we are slowly returning to normal, and today I managed to get a little time to write this post.

From late August I've been working on an interesting project for a large company, side by side with great guys who every day give me new inputs. One of the things that we most frequently discuss is the functional approach to programming. Nothing new, but unfortunately in the "mainstream" OO world functional programming is often seen as something old, just good for college professors with old PCs, CGA monitors, and weird text editors. Instead, it is a world that is rapidly changing, and it is more and more influencing the development of computer science, and its application to problems of everyday life. But probably this is not enough to decide to abandon the old and comfortable world of OOP for a difficult and perhaps dangerous magical mystery tour...

Game over, then? Not really. The hours spent talking (and especially listening) about themes such as sequences, immutability, lazyness, higher order functions, monads, compositionality, and more, are slowly influencing the way I think and do programming, because most of them are applicable in OOP languages and in languages like C#. The turning point that transformed C# in a "functional space" has been the introduction of Linq. Here you have to remove all the marketing and mainstream technical comunication, which present Linq as the new ORM, a way to make queries directly from code, or in more advanced cases as something for which we can write a query provider that enable us to implement a idiomatic query model. Those are correct views but perhaps limiting, and they do not describe the essence of Linq but only some of the most popular applications. "My" idea now is more and more this one: Linq is an environment in which you can "extend the world" :)

I'll try to write more posts in the coming days about more specific aspects of this statement, now I will focus here on a quick look about one feature: extension methods. They are normally presented as a way to add methods to existing types for which you do not have the source code. In short, a sort of "trick" Microsoft has used to enrich types and interfaces (!) with useful implementations which were not introduced before. The fact that I mentioned the interfaces too should have you wondering about it: how do you add implementations to constructs which by definition do not allow them? Strange, isn't it?

So let's try to reverse the point of view: Linq and its the query syntax are "the world", or rather a "functional" world inside C#, with its required (although not enforced) rules of immutability, absence of side effects, deferred execution, compositionality; then there are the types of the BCL, including interfaces, but also the whole .NET Framework types set, and, why not, our own types, our own interfaces, as well as types and interfaces from any library around. We are envious of IEnumerable and IQueryable, and we'd like so much that our types could join them in Linq query, right? So what can we do? We should begin to write a query provider, shouldn't we? Well, not necessarily, it is often more than enough to write a bunch of extension methods for our types, and here we go, suddenly they can participate in Linq queries! We did not extend our objects, we have extended Linq with them :) This way even the fact that you can "extend" interfaces with implementations makes more sense, because actually you are extending Linq to understand specific interfaces.

Think about it, and next time I will bring some examples and some of related linq links. Just to make you curious, we'll write code like this:

var q = from i in 20.ToMaybe()
        from l in 22.ToMaybe()
        select i + l;

        Assert.IsTrue(q.HasValue);
        Assert.AreEqual(42, q.Value);

var r = from i in ToMaybe.Empty<int>()
        from l in 22.ToMaybe()
        select i + l;

        Assert.IsFalse(r.HasValue);

Published episodes: