Just A Summary : Category ruby, everything about ruby

Get sophisticated

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 20 Aug 2008 11:25:00 -0500

Ruby’s primitives (Strings, Hashes, Arrays, Numbers – anything that has a literal syntax) are fine things. But that doesn’t mean you should use them everywhere. You’re often much better off wrapping them up in your own Value Objects.

Something I was working on at Railscamp this weekend threw up a great example of why it makes sense to replace primitives with more specific objects as soon as possible. Tom Morris asked me to take a look at Rena, an RDF/Semantic Web thingy.

The RDF spec describes two types of literals: a plain literal, which is a string with an optional language attribute and a typed literal, which is a string and an encoding (so the string might represent an integer, float, or anything else that your schema feels like expressing).

These literals can be output in one of (at least) two formats. We’ll start by looking at Literal#to_trix and see where that takes us:


def to_trix
  if @lang != nil && @lang != "" 
    out = "<plainLiteral xml:lang=\"" + @lang + "\">" 
  else
    out = "<plainLiteral>" 
  end
  out += @contents
  out += "</plainLiteral>" 
  return out
end

If we look over at TypedLiteral#to_trix we see a much more straightforward implementation:


def to_trix
  "<typedLiteral datatype=\"#{@encoding}\">#{@contents}</typedLiteral>" 
end

How do we eliminate that ugly conditional at the beginning of Literal#to_trix, and analogous conditionals in Literal#to_n3 and TypedLiteral#to_n3?

My first thought was that I wanted to be able to write something like:


def to_trix
  "<plainLiteral#{@lang.to_trix}>#{@contents}</plainLiteral>" 
end

But I didn’t want every string in the world suddenly acquiring a to_trix method. So, the solution was to intoduce a Literal::Language class and coerce our language into it, so Literal#initialize became:


def initialize(contents, lang = nil)
  @contents = contents
  @lang = Language.coerce(lang)
end

And Language would look something like:


def self.coerce(lang)
  if lang.is_a?(self)
    return lang
  end

  new(lang.to_s.downcase)
end

def initialize(lang)
  @value = lang
end

def to_trix
  if @value == ''
    ''
  else
    " xml:lang=\"#{@value}\"" 
  end
end

That ugly conditional’s still there though, so we introduced the Null Object pattern, and things started to look a good deal cleaner:


class Language
  class Null
    include Singleton

    def to_trix
      ''
    end
  end

  def self.coerce(lang)
    case lang
    when self
      return lang
    when nil, ''
      return Null.instance
    else
      return new(lang.to_s.downcase)
    end
  end

  ...

  def to_trix
    " xml:lang=\"#{@value}\"" 
  end
end

At this point, we’re still just pushing code around. If anything, we’ve got more lines of code now than when we started, but we’re starting to move behaviour nearer to the data it relates to, and our objects are starting to look like objects rather than data structures. So, we press on and make a TypedLiteral::Encoding class and, at this point things start to look interesting. TypedLiteral is starting to look almost exactly the same as Literal, but with an Encoding rather than a language.

That strange leading space in Language#to_trix is starting bug me. Let’s rewrite like so:


class Literal 
  class Language
    def format_as_trix(literal)
      "<plainLiteral xml:lang=\"#{@value}\">#{literal}</plainLiteral>" 
    end

    class Null
      def format_as_trix(literal)
        "<plainLiteral>#{literal}</plainLiteral>" 
      end
    end
  end

  def to_trix
    @lang.format_as_trix(@contents)
  end
end

If we make analogous change to TypedLiteral and TypedLiteral::Encoding it’s obvious that TypedLiteral and Literal were essentially the same class. Renaming @lang and @encoding to @language_or_encoding makes this blindingly obvious, so we’ll remove all of TypedLiteral’s methods except initialize. All that remains is to introduce Literal.untyped and Literal.typed factory methods to Literal, and make Literal.new into a private method and we can remove TypedLiteral in its entireity. So we change the specs to reflect the new API (wrong way round I know). Now we have a chunk of shorter, clearer code that will hopefully be easier to extend to cope with outputting literals in other formats.

Retrospective

I realise that patterns aren’t the goal of development, but by the end of the process we have a Strategy (Language/Encoding), a couple of Factory Methods (Literal.typed, Literal.untyped) and a couple of factoryish methods (Language.coerce, Encoding.coerce).

The most important aspect of the change was the introduction of the two new value object classes. Once they were introduced, they became the obvious places in which to put the varying behaviour and eliminate the repeatition of conditional code from the to_* methods. If there were to be a third output style, I would look at introducing classes like N3Stream, TrixStream and WhateverStream and have a scheme like:


def to_n3
  print_on( N3Stream.new )
end

def print_on(stream)
  language_or_encoding.print_on(stream, value)
end

but that’s almost certainly over complicating things right now.

The other thing I like about this kind of refactoring is that it drives the code towards methods and classes which obey the single responsibility principle and, at the end of the process, not only do we have fewer lines of code in total, but the individual methods involved are all substantially shorter and closer to the left hand margin.

I really should start doing this kind of thing more in my Rails practice – I keep being put off by the fact that the composed_of helper is so annoyingly not quite right and, rather than submitting a patch or making a plugin I go “Ah well… I can live with a string for a bit longer…” and I know. From hard won experience at that, that it’s going to come and bite me. It’s already bitten Rails recently when Ruby got a new String#to_chars which doesn’t work like the ActiveSupport version.

Notes

If you want to see the gory details of how the change got made, Tom has merged this weekends changes into the github repository. It didn’t happen in quite the order I’ve described it in this post, but neither is this post a complete fabrication.

Changes

Corrected a stupid typo in the first block of code. Ugly condition is actually if @lang != nil && @lang != ''

Baby's first screencast

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 14 Mar 2008 14:29:00 -0500

If you follow the Ruby blogs, you will probably have seen a bunch of programmers attempting to do something akin to Haskell’s maybe, or the ObjectiveC style, message eating null.

Generally, by about the 3rd time you’ve written

if foo.nil? ? nil : foo.bar 
  ...
end

you’re getting pretty tired of it. Especially when foo is a variable you’ve had to introduce solely to hold a value while you check that it’s not nil. The pain really kicks in when you really want to call foo.bar.baz. You can end up writing monstrosities like (tmp = foo.nil? ? nil : foo.bar).nil? ? nil : tmp.baz (actually, if you were to write that in production code, you probably have bigger problems). One option is to just define NilClass#method_missing to behave like its Objective C equivalent, but I’ve never quite had the nerve to find out how that might work. I wanted to write

if maybe { foo.bar.baz }
  ...
end

and have nil behave like an Objective C nil for the duration of the block, but no longer. So I wrote it. Then I thought about how to present it. I wrote the thing test first using rspec and the whole thing just flowed, but writing up a test first development process for a blog entry is painful, so I’ve made a very rough (but blessedly short) screencast of the process instead.

That’s a slightly reduced thumbnail, the movie is substantially more readable. The bottom pane of the window is the output of autotest rerunning the spec every time either the spec or the implementation changes. The top pane alternates between the specs and the implementation. Generally, every time I edit the specs, a test starts failing and every time I edit the implementation it starts passing again. In the (any) real coding run, there were of course false starts, but generally the specs kept me pretty straight.

A word or two of warning: This is a completely unedited, silent, screen cast, there are typos, backtrackings and other embarrassments. I stopped recording once I’d got 4 tests passing, but this is far from release quality (it’s perfectly usable if you know its limitations, but it’s not entirely robust).

Please let me know what you think of this. I’m aiming to make a more polished version, complete with voice over and it would be good to know which bits are confusing and need addressing in more detail in the voice over.

I am not a rock star

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 23 Feb 2008 00:13:00 -0600

I am not a rock star. I am a computer programmer. I think I’m quite a good one.

You are not a rock star either.

387,000 matches to that query. Can we all just… I don’t know… grow up please?

Mutter… grumble… chunter… I’m 40 you know!

Updates

I have it on reliable authority that James O’Kelly is a Ruby on Rails Rockstar that would make a great addition to any team!

Typo 5 is out - and more on the future

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 30 Dec 2007 09:10:18 -0600

Right, we’ve cut a Typo 5 gem and it’s on rubyforge and heading to various mirrors I hope. Frédéric’s writing the release notification which will be appearing on Typosphere Real Soon Now.

It’s been a surprisingly tricky process – we’re now requiring Rails 2.0.2 because the workings of view_paths have changed in a way which means we can’t quite make themes with Rails 2.0 and 2.0.2 and working with the edge seems like the more sensible proposition. If you’re on the bleeding edge, you should find that you get the right Rails via svn:externals anyway.

Typo futures

Meanwhile, I’ve been playing with stet and I’ve come to the conclusion that, although there’s mileage to be had in a radically slimmed down approach to the way Typo works, I’m better off simply removing the misfeatures from Typo and building from there – there’s a surprising amount of stuff that needs to be done in a competent blogging engine that Typo gets right – starting again would be throwing the baby out with the bathwater I think.

However, this does mean that if you’re following the Typo SVN trunk, you’ll be seeing a reduction in features in the short term. We’ll be copying the current trunk to a 5-0-stable branch before we start with the featurectomies though, so if you’re just after bugfixes, you’ll be better off there.

Multiblogging

We’re aiming to have multiblogging in the next release, but we’re rethinking the how of it. Right now, the ‘Blog’ object adds a bunch of complexity to code that would be much happier simply assuming that it has the database to itself. So we’re going to look at switching to a database per blog approach, that way our core code can pretty much forget about the complexities of multiblogging, and (at least initially) anyone who wants multiblogging can get there by monkeying with configuration files – of course, we intend to add a web based admin interface once things settle down and we know how things are going to work.

Caching

Caching is always a bugbear in any typo installation. Because we want to be installable on the widest possible range of hosts, we can’t rely on the presence of handy tools like ‘memcached’. Also, some of our users are operating under some fairly severe memory and process constraints, so it makes sense to have the webserve serve static files as much as possible. Meanwhile, tools like Evan Weaver’s Interlock are pointing the way towards seriously effective fragment caching. I shall be looking into implementing something that conforms to the interlock interface, but which can use an arbitrary cache backing store for fragments and maintain a full page cache. It’ll be interesting to find out if this is doable…

Atom Publishing Protocol

ActionWebService is going to go away – it’s already in the ousted branch of the rails SVN repository, and including it in Typo to support the various different admin APIs is getting painful. So, we’re going to preempt it. We won’t be getting rid of the various XMLRPC APIs until the pain becomes too great, but we are going to be concentrating on implementing, and strongly favouring, the Atom Publishing Protocol.

Feeds for everything

In particular, we’ll be adding atom feeds for all sorts of administrative data as a means of enabling people to write external tools for, say, spam protection, comment moderation and notification tasks. Right now, there’s a great deal of computation happening on the server side every time someone, say, comments on a post – in the kind of resource limited environments some people are running Typo in, that’s too much work. Switching to a feed + APP approach should help enormously with resource utilization.

Speaking of resources…

Using the server to render article previews is… suboptimal. Expect to see a javascript based preview system akin to the one I use for comments here.

Rails 2.0 and the Future of Typo

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 16 Dec 2007 10:35:00 -0600

So, if you’ve been watching the Typo tree, you’ll see there’s been a fair amount of activity on it since Rails 2.0 got released. There’s a new default theme replacing the rather creaky ‘azure’, and a fair amount of work on getting our code compatible with the current state of Rails. As we work on this, it becomes apparent that Typo’s code is getting horribly brittle. I have said before that there’s been several places where we’ve zigged before Rails zagged, and we’re paying the price for that. It doesn’t help that our test coverage is distinctly ropy either – and I’m probably guiltier than most for letting things get into that state.

So, our goal is to get what we have cleaned up and working with Rails 2 before releasing Typo 5.0. Once that’s done, that line of code will go into maintenance mode – there are still plenty of bugs to fix and documentation to write, but I’m afraid that extending that base is becoming too much of a chore.

Which is why I have a new path in my local svk repository, //stet. I’m using this for experimental development of a new, slimmed down blogging engine that will be, first and foremost, a capable Atom Publishing Protocol host. Things like spam processing will be removed from the core of the application, but we’ll provide a suite of webservice clients that will consume the ‘unmoderated feedback’ webfeed and use APP to either approve or delete the feedback as appropriate.

Theming (at least initially) will probably be confined to Javascript and CSS changes, and I’m even thinking of exposing the sidebars as Atom collections – certainly I expect that, in the first cut, sidebars will be static – if you want content that looks dynamic you’ll have to do it via javascript.

My initial goal is to slim things down as far as I possibly can – I want to build a blogging engine that can cope with the tight memory constraints of shared hosting by off loading much of the heavy lifting to client boxes. After all, I have far more processing capability available to me on the laptop I’m typing this on than the slice of Site5’s hosting infrastructure that’s actually running the blog. By making things small and static, I also hope to wring good performance numbers out of the tool as well – expect aggressive page caching at the very least.

Another important goal is easy migration of Typo databases. I expect to be writing models and controllers from the ground up, but converting the database should just be a matter of running a migration.

Experimental

Of course, stet’s currently very experimental – about the only thing that’s actually written so far are a couple of routing plugins which should help radically simplify our routes.rb (expect an article’s url to change from /articles/2007/12/16/rails-20-and-the-future-of-type to /article/2007/12/16/rails-20-and-the-future-of-typo, but with a redirect in place to cater for the old style urls). I may have grandish plans for the thing, but I could equally discover that I’m off up a blind alley, in which case you can expect me to return to the current typo codebase with a few more lessons learned.

ActiveResource?

I remain unconvinced by ActiveResource as a technology. I agree with the authors of RESTful Webservices – good webservices are joined up. They take full advantage of what could be described as the defining technology of the world wide web, the URL based hyperlink to knit resources together in a discoverable fashion. An ActiveResource based webservice may well be a good HTTP citizen, but it’s still not really ‘webby’ enough for my taste. Which means the Atom Publishing Protocol will remain my friend for most of the things I hope to do with stet. It may be harder to write a good APP server, but I’m convinced that it’s a much better interface for clients, and you should always favour ease of use over ease of implementation. If nothing else, we’re aiming to have more users than developers. Many more.

Comprehensible sorting in Ruby

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 16 Dec 2007 03:00:51 -0600

Here’s a problem I first came across when I was about 13 and helping do the stock check at the family firm. The parts department kept all their various spare parts racks of parts bins. Each bin was ‘numbered’ with an alphanumeric id. We had printouts of all the bin numbers along with their expected contents and we’d go along the racks counting the bins’ contents and checking them off against the print out. What confused me at the time was the way the printouts were organized. Instead of the obvious ordering, “A1, A2, A3, ..., A99”, the lists were ordered like “A1, A10, A11, ..., A2, A20, A21, ...”. After a bit of thought I realised that the computer was sorting the numeric bits of the bin numbers as if they were just sequences of strange letters. A bit more thought made me realise why, post computerisation, people were starting to use bin numbers like “A01, A02, ...”. Computers were more important than people so, in order to make sorting things easier, just add spurious leading 0s to make the number field a fixed width and Robert’s your parent’s brother.

27 years later and computers are still crap at sorting things in a sensible fashion. Back before Moore’s Law was really kicking in, I suppose it was excusable, but surely we’ve moved past that now.

Over on the labnotes blog, there’s an example of some ruby code that attempts to do ‘human’ sorting:

module Enumerable
  def sensible_sort
    sort_by { |key| key.split(/(\d+)/).map { |v| v =~ /\d/ ? v.to_i : v } }
  end
end

It’s okay, as far as it goes. It certainly solves the parts bin problem I outlined above, but it’s not ideal. For example, you might expect ['-1', '1', '1.02', '1.1'].sensible_sort to leave the order unchanged, but what you actually get is ‘1, 1.02, 1.1, -1’. Not ideal. Let’s rewrite sensible sort as

module Enumerable
  def sensible_sort
    sort_by {|k| k.split(/([-+]?\d+(?:\.\d+)?(?:[-+]?[eE]\d+)?)/).map {|v| Float(v) rescue v}}
  end
end

That ugly regular expression should match a far wider selection of string representations of numbers. Certainly our ‘bad’ list is now sorted correctly.

But what about “a-1”, “a-2”. Using the implementation above, they’d get sorted as “a-2, a-1”, which can’t be right, can it? Let’s extend it a bit more and make sure we only worry about the ’+’ and ’-’ if they’re at the beginning of a line or preceded by whitespace.

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v}}
  end
end

And that works fine, until you find that “B” sorts before “a”. Let’s catch that as well:

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

Yay!

Oh, wait a minute, what about version numbers? How should we sort, say “perl 5.8.0” and “perl 5.10.0”? The 5.8.0 form should definitely come first… Hmm…

How about

module Enumerable
  def sensible_sort
   sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

How far down does this thing go?

I just noticed that ”.1” sorts after “1”. Time for another tweak…

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?(?:\.\d+|\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?))/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

but that doesn’t work with version numbers like ”.8.2”, ”.10.2”...

Time passes… Thorin sits down and sings about gold

I was planning on giving an extension of the regex that caught this issue as well, but I’m afraid I’ve stumped myself – I can’t do it with a single regular expression unless I can use a fixed width lookbehind assertion, but they’re only available in Perl. Of course, it’s still possible to fix it, but doing so will take more thought than I have available to me at this time on a Sunday morning. And all this is before we get onto making sure that “1/2” sorts between “0” and “1”. And phone numbers. After all, “01915551238” is ‘obviously’ the same as “0191 555 1238” and “0191 555-1238”, so they should end up next to each other in the sorted list.

It looks like this is a ‘three pipe problem’ after all. I shall probably return to this…

Rails tip: Side effect filters

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 08 Oct 2007 07:13:00 -0500

Some bugs are easy to overlook. One that has a habit of catching me out is a Rails filter that returns false occasionally when it’s being evaluated purely for its side effects. Here’s how I’ve started working round the issue:

def side_effect_filter
  return if some_conditions_not_met?
  ...
ensure
  return true
end

What happens here is that the ensure catches any return and returns true instead. The catch is that if something throws an uncaught exception anywhere, it too gets caught by the ensure and true is returned. Which may not be what you were looking for. Here’s how to fix that issue:

def side_effect_filter
  error = nil

  return if some_conditions_not_met?
  ...
rescue Exception => error
ensure
  raise error if error
  return true
end

This catches the exception in a rescue and stashes it in the error variable, then the ensure checks to see if an exception was thrown and rethrows it, otherwise, it just returns true. Which is bulletproof, but ugly. Let’s wrap the ugliness up in a method:

def self.side_effect(method, &block)
  def_method(method) do
    error = nil
    begin
       instance_eval(&block)
    rescue LocalJumpError # catches an explicit return
    rescue Exception => error
    ensure
      raise error if error
      return true
    end 
  end
end

side_effect :side_effect_filter do
  return if some_conditions_not_met?
  ...
end

Again, not pretty inside, but all we actually care about anywhere else is that the interface is good and does what it’s supposed to do. Encapsulated ugliness has its own beauty. Especially if you get the interface right.

Homework

This should pluginize quite nicely, just install the method in ActionController::Base and ActiveRecord::Base and you have a very useful tool, but I’m still not sure that the method name is right, so I’m holding off on it. If someone were to come up with a bulletproof name and release a plugin, that would be wonderful though.

Updates

Fixed a scoping issue in the encapsulated version of the code. Replaced yield with instance_eval(&block)

My head hurts

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 23 Sep 2007 14:09:24 -0500

During DHH’s keynote at RailsConf Europe it was apparent that there’s a great deal to like in edge rails, so I thought I’d have a crack at getting Typo up on it.

Ow.

I’d expected the pain points to be related to routing, but it seems that the rails routing system is approaching the level of the Excel calculation engine – nobody dares touch it for fear of breaking things, so typo’s custom routes seemed to work quite happily. There were a few things that have been deprecated, pluginized or moved out of the set of modules that’s automatically included when you do a rake rails:freeze:edge, but they were pretty easy to sort – the deprecation messages are a good deal more informative now than they were last time went deprecation squashing. There’s a surprising amount of stuff that’s been removed without any deprecation warnings though, which isn’t very sporting. DHH said there would likely be a 1.2.4 release (possibly a day before 2.0) with a bunch more deprecation warnings covering everything that’s actually going away, so if you’re thinking of moving a maturish app to Rails 2.0 it might make sense to wait for 1.2.4, install that, squash warnings, and move on up to 2.0.

The real pain comes from themes. Typo’s themes rely on Rails internals working in a particular way, but they don’t work like that any more. In theory, the internals appear to be more theme friendly, related to allowing plugins to include views. The problem is, that it’s possible to change Typo’s theme without restarting the server, and the new themish internals don’t expect anything to change until the server’s restarted.

So, I’ve been playing with plugins. The most promising approach appears to be that of the themer plugin, which gets pretty close to doing what we need, and does it in a way that seems like it should work with both 1.2.3 and Edge Rails. It does appear to be making some radically different assumptions about the structure of the themes directory, but the basic framework is good and I should be able to make things work by making our current them object conform to Themer::Base’s interface and duck type my way to the sunny uplands of Edge Rails compatibility.

Which will be nice.

I like the themer approach a lot. Instead of monkeying about in the guts of rails, it monkeys about in front of Rails. It overrides render so that you can pass it a theme/lookup object. If it sees a lookup object, it uses that to rewrite the rest of the render arguments into a form that will render the right thing using the standard implementation of render. In a work project I’ve taken a similar approach to handling polymorphic routes for things like:

map.resources :pictures do |pics|
  pics.resources.comments
end

map.resources :users do |users|
  users.resources.comments
end

I ended up with a to_params method defined on my Comment model, and stuck an extended url_for in front of the default Rails version, which looks something like:

def url_for_with_to_params(*arguments)
  if arguments[0].respond_to?(:to_params)
    with_options(arguments.shift.to_params) do |mapper|
      mapper.url_for_without_to_params(*arguments)
    end
  else
    url_for_without_to_params(*arguments)
  end
end
alias_method_chain :url_for, :to_params

Which is so much neater than the last time I attacked this particular problem (see the acts_as_resource plugin).

One of the nice things about Rails is that, although it’s opinionated and somewhat liberal with the syntactic vinegar for things the core team don’t think is the Right Way, they’re pretty good at leaving the door open for people like me who have other opinions. Both the themer plugin and my as yet unpluginized extension of url_for work by using existing capabilities in new ways and, because those capabilities are documented we can expect them to continue to work over multiple versions of Rails. Plugins that achieve similar effects by monkeying with Rails’s internal interfaces are hostages to fortune. Internal interfaces are free to change at any time, even between point releases, so a plugin can be left high and dry with surprising rapidity. Just ask the Rails Engines folk.

A tiny ruby niggle

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 09 Sep 2007 02:51:00 -0500

You know what? I’m starting to miss compulsory semicolons as statement terminators in Ruby.

“What?” I hear you say. “But not needing semicolons is one of Ruby’s cardinal virtues! Are you mad?”

I don’t think so, but maybe you’ll disagree after I explain further.

Here’s a piece of code that I might write if semicolons were the only way of terminating a statement:

Category.should_receive(:find_by_permalink)
  .with('foo')
  .and_return(mock_category);

Or how about a complex find query

def find_tags_for(tag_maker, order = 'count')
  klass = tag_maker.class
  find :all
    , :select => 'tags.*, count(tags.id) count'
    , :group => Tag.sql_grouping
    , :joins => 
        "LEFT JOIN taggings ON "
      + "      tags.id = taggings.tag_id "
      + "LEFT JOIN bookmarks ON "
      + "      bookmarks.id = taggings.taggable_id "
      + "  AND taggings.taggable_type = 'Bookmark' "
      + "LEFT JOIN #{klass.table_name} ON "
      + "      #{klass.table_name}.id = bookmarks.#{klass.to_s.underscore}_id"
    , :conditions => conditions_for(tag_holder)
    , :order => (order == 'count') 
        ? 'count(tags.id) desc, tags.name' 
        : "tags.name"
    , :readonly => true
    ;
end

I first came across the idea of the leading comma in Damian Conway’s excellent Perl Best Practices. The idea is that, by leading with the comma it’s very easy to add a new argument to an argument list or hash specification without having to remember to stick a comma on the end of the preceding line if it was at the end, and also, the leading comma makes it very plain that the line is a continuation of its predecessor in some way.

To make the examples work in Ruby, you have to add a \ to the end of each line that has a continuation, so the first example has to be written:

Category.should_receive(:find_by_permalink) \
  .with('foo')                              \
  .and_return(mock_category);

Lining up the \s helps to stop them disappearing, but it’s an awful faff.

What tends to happen (in the rails source especially) is that ruby programmers simply don’t break their lines up. A quick search of the rails source finds plenty of lines more than 160 characters long.

Of course, some will argue that it doesn’t matter, that the old 80 column limit is a silly hangover from the days of steam when the only way to interact with your code was through an 80 column, green phosphor terminal. They have a point. An arbitrary line limit is silly, and we should get over it, especially in source code. However, unless you’re going to go around with every window open to its maximum width, lines will wrap, and they won’t do it nicely, or respect the indentation conventions of your language. Long lines are murder in diffs too, finding the point of difference is so much easier when your eye doesn’t have to scan an epic line.

It’s a shame there’s no way of forcing ruby’s parser to require semicolons as statement terminators for those programmers like me who think that the restriction that a statement must end with a semicolon is worth the freedom to break lines where we like without needing to escape every line break. It’s a shame too that popular tools like Textmate are so clumsy when it comes to dealing with line breaks. I would attempt to hold Emacs up as a paragon in this respect, but its Ruby mode tends to get a wee bit bemused once you start breaking lines, so that’s no good.

Domino theory

It’s amazing how far reaching seemingly simple language design decisions can be isn’t it? Just getting rid of the need to terminate statements with a semicolon has an enormous effect on they way code in ruby looks. I’m just not sure that they look better.

Maybe Smalltalk got it really right – they chose to use the most valuable syntactic character of all, the space, to denote sending a message. That freed up the . for use as a statement (sentence?) terminator. Then that freed up ; for use in one of Smalltalk’s most distinctive patterns – the cascade. Where a Rails programmer might write:

form_for(@comment) do |f|
  f.input(:author)
  f.input(:title)
  f.input(:body)
end

A Smalltalk programmer might eliminate the need for a temporary variable by doing:

Comment>>printOn: html

  (html formFor: self)
    input: #author;
    input: #title;
    input: #body.

All those input: ... messages get sent to the result of html formFor: self. Once you get the hang of it, it’s a really sweet bit of syntax.

Incidentally, there’s been some discussion on the squeak mailing lists of a companion to the cascade, which would use a ;; as a sort of ‘pipe’. The idea is to be able to replace code like:

((self collect: [:each | each wordCount) 
    inject: 0 into: [:total :each| total + each]) 
        printOn: aStream.

with

self collect: [:each | wordCount]
    ;; inject: 0 into: [:total :each | total + each]
        ;; printOn: aStream.

(NB: Please ignore what those code snippets do, because that’s gruesome. Concentrate on how they do it).

Nobody’s quite proposed going as far as Haskell does with its Monads, which can be thought of as a magical land where the meaning of the semicolon changes according to what sort of Monad you’re in. (In an IO monad for instance, the semicolon imposes an evaluation order. In some other monad, the semicolon could just as easily denote a backtracking point). Then again, there’s nothing to stop the dedicated Smalltalker implementing something Monadish – every Smalltalk class can specify how its methods should be compiled after all…

In conclusion…

I’m not sure I’ve got a real conclusion for all this. I’m mostly musing. However, I do think it’s useful to think carefully about restrictions and what they free us to do as programmers. Lispers will wax lyrical about the way that their language’s pared down syntax lets them do amazing things with macros. Smalltalkers will defend to the death the idea that the only way to do anything is to send messages to objects. Pythonistas love their syntactic whitespace. Haskellers love their static typing (admittedly, they have an incredibly flexible notation for expressing type that leaves most other programming languages standing).

And any English speaker with ears will know that a poem like Dylan Thomas’s Do Not Go Gentle Into That Good Night gains much of its power from it’s form, the villanelle, one of the most restricted forms of poetry there is. Two lines repeating through the poem and a staggering number of rhymes to find:

Do not go gentle into that good night,
Old age should burn and rave at close of day;
Rage, rage against the dying of the light.
Though wise men at their end know dark is right,
Because their words had forked no lightning they
Do not go gentle into that good night.

Good men, the last wave by, crying how bright
Their frail deeds might have danced in a green bay,
Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,
And learn, too late, they grieved it on its way,
Do not go gentle into that good night.

Grave men, near death, who see with blinding sight
Blind eyes could blaze like meteors and be gay,
Rage, rage against the dying of the light.

And you, my father, there on the sad height,
Curse, bless me now with your fierce tears, I pray.
Do not go gentle into that good night.
Rage, rage against the dying of the light.

If that’s not making a virtue of a restriction, I don’t know what is.

Today's noun is: Reification

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 22 Aug 2007 18:22:00 -0500

Reification: The mental conversion of a person or abstract concept into a thing. Also, depersonalization, esp. such as Marx thought was due to capitalist industrialization in which the worker is considered as the quantifiable labour factor in production or as a commodity. – OED

In the sense that the OED has it, I’m not what you could call a fan of reification. At work, we have a rule that anybody who starts talking about ‘resources’ when they mean ‘people’ gets a (verbal) slap.

However, in OO circles (or maybe just in my head), reification is a good thing. It’s the process of taking something abstract and turning it into a ‘real’ object. Usually, the word gets used for big things like turning an intractable method into an object as a step on the way to refactoring that method. I tend to use it in a slightly broader sense. For me, reification is the process of turning something (a method or a data structure usually) into a full blown object with its own behaviour.

Back when I was working on Pixie (a cunning, but weird, object persistence tool written in Perl) we had a data structure which was used for keeping track of managed objects. It started life as a hash. Everything was fine at first, but over time we ended up with more and more code being repeated across the codebase that was concerned with manipulating the cache hash. So, we replaced the hash with a new object and pulled all the repeated code into methods on that object, which gave us cleaner code to extend, and a strong feeling that we should have turned the cache into an object much earlier in the game. (By leaving it so long, we had a lot more code to move about, some of it in fairly obscure places; tracking down the last bit took a while.)

Data structures like hashes and arrays are really useful in languages that have them. The catch is, they have this habit of acquiring code. When this starts to happen, it’s time to reify – to replace the hash with a task specific object. In Ruby, it’s easy enough to inherit from Hash, but Hash comes with a pile of methods that probably aren’t relevant to your particular need. Generally it’s better to delegate. The first cut doesn’t have to be that complicated, just decorate the hash with a new class and initialize an instance of the class at the point where you had just made the hash.

Once that’s done, you can go through your code and move the bits that treat the hash as a data structure onto your new class. As you gather all the common behaviour to the new class, you’ll start to see places where you can improve code quality by merging common behaviours, replacing complex conditionals with polymorphism (you’ll probably have to introduce a factory method if you do that) and pulling hash keys out into instance variables.

Stalled reification

Reifying your data structure isn’t an end in itself, it’s a step along the way as you refactor your code.

There’s an example of a stalled reification to be found in ActionController::Routing::Resources. Consider the implementations of map_resource and map_singleton_resource, which are the worker methods used whenever you do a map.resource or map.resources in your routes.rb.

def map_resource(entities, options = {}, &block)
  resource = Resource.new(entities, options)

  with_options :controller => resource.controller do |map|
    map_collection_actions(map, resource)
    map_default_collection_actions(map, resource)
    map_new_actions(map, resource)
    map_member_actions(map, resource)

    if block_given?
      with_options(:path_prefix => resource.nesting_path_prefix, &block)
    end
  end
end

def map_singleton_resource(entities, options = {}, &block)
  resource = SingletonResource.new(entities, options)

  with_options :controller => resource.controller do |map|
    map_collection_actions(map, resource)
    map_default_singleton_actions(map, resource)
    map_new_actions(map, resource)
    map_member_actions(map, resource)

    if block_given?
      with_options(:path_prefix => resource.nesting_path_prefix, &block)
    end
  end
end

There’s a lot of repetition there. The only differences are the classes of the resource object, name of the second function called in the with_options block. If we take a look at, map_collection_actions things start to look even fishier. Here’s map_collection_actions, for example:

def map_collection_actions(map, resource)
  resource.collection_methods.each do |method, actions|
    actions.each do |action|
      action_options = action_options_for(action, resource, method)
      map.named_route("#{resource.name_prefix}#{action}_#{resource.plural}", "#{resource.path};#{action}", action_options)
      map.named_route("formatted_#{resource.name_prefix}#{action}_#{resource.plural}", "#{resource.path}.:format;#{action}", action_options)
    end
  end
end

resource.collection_methods.each? Let’s see what happens if we the various map_foo_actions methods into methods on ActionController::Resources::Resource. While we’re about it, we can rename map_default_collection_actions to map_default_actions on Resource, and map_default_singleton_actions to map_default_actions on SingletonResource, which inherits from Resource. map_collection_actions becomes:

def map_collection_actions(map)
  collection_methods.each do |method, actions|
    actions.each do |action|
      map.with_options(action_options_for(action, method)) do |m|
        m.named_route("#{name_prefix}#{action}_#{plural}",
                      "#{path};#{action}")
        m.named_route("formatted_#{name_prefix}#{action}_#{plural},
                      "#{path}.:format;#{action}")
      end
    end
  end
end

(we move action_options_for onto resource as well, of course).

Once we’ve moved the various mapping helpers onto the resource classes, we can revisit map_resource and map_singleton_resource


def map_resource(entities, options={}, &block)
  resource = Resource.new(entities, options)

  with_options(:controller => resource_controller) do |map|
    resource.map_collection_actions(map)
    resource.map_default_actions(map)
    resource.map_new_actions(map)
    resource.map_member_actions(map)
  end

  if block_given?
    with_options(:path_prefix => resource.nesting_path_prefix, &block)
  end
end

def map_singleton_resource(entities, options={}, &block)
  resource = SingletonResource.new(entities, options)

  with_options(:controller => resource.controller) do |map|
    resource.map_collection_actions(map)
    resource.map_default_actions(map)
    resource.map_new_actions(map)
    resource.map_member_actions(map)
  end

  if block_given?
    with_options(:path_prefix => resource.nesting_path_prefix, &block)
  end
end

And now, we no longer have two method bodies that look very similar, apart from the resource class, we have to methods that look identical apart from the resource class. So, if we pull out the common bits and put them onto Resource, like so:


class ActionController::Resources::Resource
  def install_routes_in(map, &block)
    map.with_options(:controller => controller) do |m|
      map_collection_actions(m)
      map_default_actions(m)
      map_new_actions(m)
      map_member_actions(m)
    end

    if block_given?
      map.with_options(:path_prefix => nesting_path_prefix, &block)
    end
  end
end

Then map_resource and map_singleton_resource become


def map_resource(entities, options = {}, &block)
  Resource.new(entities, options).install_routes_in(self)
end

def map_singleton_resource(entities, options = {}, &block)
  SingletonResource.new(entities, options).install_routes_in(self)
end

Where’s the benefit?

Apart from making the active_record/lib/resources.rb a bit shorter (a laudable result in itself), where’s the benefit here?

From my own experience of implementing datestamped_resource, a routing plugin that we use in Typo, it makes the life of anyone writing a resource like routing helper for Rails a great deal easier. With datestamped_resource I ended up subclassing ActionController::Resources::Resource, doing the refactoring I’ve outlined here, but leaving the original Rails methods where they were and just implementing the ‘moved’ methods on DatestampedResource (well, not quite, map_collection_actions is pretty different from the default Resource implementation, but the other actions are pretty much the same.

In another project I’m working on, I’m trying to retain meaningful urls with (potentially) deep resource nesting, and it’d be really handy to have an inflected_resource route helper. The problem with using a meaningful to_param on your models is, avoiding permalinks that share a name with your actions. You could set up validations so that, say, ‘new’ is an illegal permalinks, but it’s clumsy.

However, if you arrange things so that your URLs are inflected, you can always tell that a URL that begins /resource/new will be a particular resource, with the permalink ‘new’, and /resources/new will be the virtual new resource.

If the resource system is factored as I outlined, this is almost trivial, you can introduce a InflectedResource subclass of Resource


class InflectedResource < Resource
  def member_path
    @new_path ||= #{path_prefix}/#{singular}/:id
  end
end

and you’re pretty much done. Admittedly, something like that (plus a small amount of copy and paste) would work with the current system, but then we’re looking at 3 substantially identical methods in ActionController::Resources and if it wasn’t time to refactor before, it’d definitely be time to refactor then.

Conclusions

Reification shouldn’t be something you do every day, but nor should it be something you do once a flood. Take a look at some of your projects and some of the places where you’re using hashes. Are those really hashes, or would they benefit from having some behaviour of their own? You can track down stalled reification by looking for anaemic classes; classes which have a lot of accessors but very little behaviour. Once you’ve found an anaemic class, look for all the places that instances of it get used. Try moving some of the client code into methods on your anaemic class. Do that a few times and you’ll end up with a real object.

If you’re fussy about never putting HTML in your models, you could end up with a mediating builder/presenter object as well, but until you start wanting to render the same structured info in different formats, I’d suggest biting the bullet and living with HTML in the model as a lesser evil than structural code. Your mileage may vary.

A cunning (evil?) trick with Ruby

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 16 Aug 2007 01:28:00 -0500

One of the handy tools that Ruby makes available to us Domain Specific Pidgin builders is instance_eval. With instance_eval you can take a block that was created in one context, with all its lovely closed over local variables, and evaluate it in the context of an instance of an arbitrary object. Which means that any references to instance variables and calls to methods are made as if your block were a 0 argument method of that class. It’s really potent, but at the same time, a little frustrating.

Frustrating? Why frustrating?

Well, it would be really cool if you could call `instance_eval` with a block that took some arguments. That way, you could inject some values into the block from still another scope. (Yes, it’s arcane, but in the places where it would be handy it would be really handy).

I just worked out how to do it:

def my_instance_eval(*args, &block)
  return instance_eval(*args,&block) unless block_given? && !block.arity.zero?
  self.class.send(:define_method, :__, &block)
  returning(__(*args)) do
    self.class.send(:remove_method, :__)
  end
end

There’s problems with that though, the most obvious one being that there’s no guarantee that __ won’t already exist as a method or that our block won’t call __. Here’s a safer, if scarier option:

def my_instance_eval(*args, &block)
  return instance_eval(*args,&block) unless block_given? && !block.arity.zero?
  old_method = (self.class.instance_method(:__) rescue nil)
  self.class.send(:define_method, :__, &block)
  block_method = self.class.instance_method(:__)
  if old_method
    self.class.send(:define_method, :__, old_method)
  else
    self.class.send(:remove_method, :__)
  end
  block_method.bind(self).call(*args)
end

This should work even in the face of being called with something along the lines of object.my_instance_eval(10) {|v| v + __}. Of course, there would be no point in calling like that. You’d only really need it when you want to do something like object.my_instance_eval(10,&a_block_param).

In the project I’m currently working, where the need for this arose, I shall probably extract the body of my_instance_eval to Proc#to_unbound_method, that way, instead of hanging onto Block objects I can hang onto UnboundMethod objects and avoid repeatedly shuffling methods.

But… how does it work?

I hope the code’s reasonably obvious. However…

The essence of it is that real methods get to take arguments, so what we need is some way of turning our block into a real method without altering the behaviour/interface of our object. Ruby does allow us create UnboundMethod objects which can be thought of as anonymous methods. To use them you have to bind them to a specific instance and then call them, which is the last thing we do in my_instance_eval. The intervening code is what turns our Proc into an UnboundMethod. First, we stash the old __ method, or a nil if there wasn’t one. Then we define a new __ using our block, and immediately use instance_method to get it back as an UnboundMethod. Then we either replace the old definition of __ or simply remove the new one, restoring the original behaviour of our class. Then we bind our new anonymous method to self and call it with our arguments. Easy.

Exercises for the interested reader

What happens if our class has method_removed or method_added implemented? Can we ‘hide’ from them? How?
What does Proc#to_unbound_method look like? Does it need to take an argument?

Doing the fixture thing

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 05 Aug 2007 02:39:00 -0500

Fixtures suck! Mocks rock! Don’t you dare let your tests touch the database!

Well… yes… I suppose. Except, mocking can be a complete pain in the arse too (made slightly less of a pain in the arse if you use the null object options) – it’s awfully easy to end up with huge long setup methods that spend all their time faking out a mock object and about two lines testing what you need.

I’m sure someone’ll be along to argue that this is evidence of lazy design on my part. Well, “Yah boo sucks!” to the lot of ‘em. Fixtures can be exceedingly useful.

Spooky action at a distance

The biggest problem I have with Rails’ current implementation of fixtures is… oh, where to start… it’s probably the action at a distance aspect of them. Your test is here, but your fixture is defined some way over there, in a random collection of yaml files.

Then there’s the problem of remembering which fixtures you need to have loaded for a particular test, and it all starts getting horrible very quickly.

One way of addressing this is to use fixture scenarios and fixture scenario builder, which works around the problem of remembering what to load and means you don’t have to write your fixtures in YAML. What it doesn’t work around is the action at a distance issue (but I’m betting it wouldn’t be too hard to repurpose the fixture scenario builder to let you declare the fixtures you need for a particular test/spec in the same place as you do the testing).

At work though, we came up with another way of making it easy to build your fixtures right in the test file. Our current approach is still a little bit clunky, and it’s nowhere near the point where I can turn it into a library, but I think it’s worth discussing anyway.

Exemplars

The question to ask is, what do you use a fixture for? Most of the time, what a test needs is a mostly generic instance of your class which will pass the validations, and which has maybe a couple of attributes set to particular values.

Let’s say you have a user class. As is common with such things, your user has a username, an email address and a password. As is so often the case, the usernames and email addresses must be unique, and the password must not be blank. Let’s say you’re working on a tagging system (isn’t everyone). Here’s the sort of specification you might write:

context "A taggable object" do
  setup do
    # makes a taggable object and a couple of users
  end

  it "Should aggregate taggings" do
    @first_user.tag(@taggable, "tag")
    @second_user.tag(@taggable, "tag")

    @taggable.save!; @taggable.reload

    @taggable.should have(1).tags
  end
end

In this context, the only thing you need from those user objects is that they behave like @users, are distinct and valid. You could simply set up a couple of users your users.yml fixture, but the approach we took at work went something like this:

class User
  class << self
    @@exemplar_count = 0
    def exemplar(overrides = {})
      @@exemplar_count += 1
      with_options(:username => "user#{@@exemplar_count}", 
                    :email => "user#{@@exemplar_count}", 
                    :password => "fredisabadpassword") do |maker|
        maker.new(overrides)
      end
    end

    def create_exemplar(overrides = {})
      returning(exemplar(overrides)) {|user| user.save}
    end

    def create_exemplar!(overrides = {})
      returning(exemplar(overrides)) {|user| user.save!}
    end
  end
end

This lets us write setup code like:

setup do
  @first_user = User.create_exemplar!
  @second_user = User.create_exemplar!
  @taggable = ...
end

In tests where you need an exemplar with a specific property, you can write Model.exemplar(:tested_attribute => specific_value) – the way the overrides work means you only have to describe the ‘interesting’ bits and the obscuring dust involved in simply building a valid object is swept under the carpet.

Homework

If you’re familiar with the Object Mother pattern, this might seem a little familiar, with the wrinkle that, instead of having a factory class, we just push the exemplar builder directly onto the model class.

If you start implementing exemplars yourself, you’ll probably spot a good deal of repetitive coding. I’ve not extracted a library yet because I’ve not quite come up with an interface that I like. Can you come up with a good way of doing it? Can you implement it?

What did I miss?

Deja vu all over again

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 29 Jul 2007 18:18:00 -0500

Back when I was still programming Perl, one of the common mistakes that you’d see when people wrote lazily initialized object accessors was code like:

sub get_content {
  my($self) = @_;
  $self->{content} ||= 'default content';
}

Code written like this would trundle along quite happily, until you had an attribute where, say, the empty string or 0 were legal values for the attribute. The problems were especially painful when the default value wasn’t something that perl treated as false. The correct way of writing that code would be:

sub get_content {
  my($self) = @_;
  unless (exists($self->{content})) {
    $self->{content} = 'default content'
  }
  $self->{content}
}

Which is substantially uglier, but safe.

Safety’s important, especially in building block code like accessor methods. An accessor method that works 99.99% of the time is like a compiler that produces correct code 99.99% of the time – useless.

Why déjà vu?

Recently, the usually spot on Jay Fields wrote up the lazy initialization pattern for Ruby. I’m not entirely sure that I agree with his motivation for the pattern, but I am concerned by his suggested code transformation. He suggests writing your lazy initialization as:

def content
  @content ||= []
end

Does that look familiar? This is subject to exactly the same potential bug as the perl code above. Admittedly, the number of possible ‘bad’ values is reduced to nil and false, but it only takes one. Here’s the fix:

def content
  unless instance_variable_defined? :@content
    @content = []
  end
  return @content
end

This code is guaranteed to work in all circumstances.

Going a little bit further…

As Mark Jason Dominus has argued persuasively elsewhere, patterns are a sign of weakness in a programming language, so how can we go about incorporating this boilerplate code into our language¹.

How about something like this (as yet untested):

class Module
  class << self
    alias_method :attr_reader_without_default_block, :attr_reader
    def attr_reader_with_default_block(*args, &default_block)
      unless block_given?
        return attr_reader_without_default_block(*args)
      end
      unless args.size == 1
        raise ArgumentError, "Expected 1 argument"
      end
      var_name = "@#{args.first.to_s}"
      self.define_method(args.first) do
        unless instance_variable_defined?(var_name)
          self.set_instance_variable(var_name, default_block.call(self))
        end
        (class << self; self; end).attr_reader_without_default_block(args.first)
        return self.send(args.first)
      end
    end
    alias_method :attr_reader, :attr_reader_with_default_block
  end
end

This code is a little more complex than the boilerplate code. When the generated method is called, possibly initializing the attribute, the (class << self; self; end).attr_reader_without_default_block(args.first) part replaces the instance’s accessor with the default attr_reader implementation and calls that instead. This is arguably premature optimization, but it’s not all that evil…

Assuming I’ve not screwed anything up, that should allow you to write.:

class Article
  attr_reader :content {'default content'}
end

and have your content lazily initialized. Extending this to let attr_writer take a block too is a reasonably obvious next step.

Extending this lazy initialization approach to work with ActiveRecord based classes is probably the next step after that. Making it work right probably involves a little bit of fossicking around in the workings of ActiveRecord, but it’s far from impossible.

¹ I tend to take the Smalltalkish view that it’s pointless to separate language from library, especially in a dynamic language. A sufficiently expressive language lets you blur the boundary between them.

How I learned to stop worrying and love aggressive mockery

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 06 Jun 2007 05:21:00 -0500

There’s something enormously liberating about writing an RSpec description that starts like:

describe ArticlesController, "feeds:" do
  before do
    @the_mock = mock('everything', :null_object => true)
    ActiveRecord::Base.stub!(:find).and_return(@the_mock)
  end

  it "/articles.atom -> atom feed" do
    get :index, :format => 'atom'
    response.should render_template('_atom_feed')
  end

  ...
end

The :null_object flag to rspec’s mock function is remarkably potent.The resulting mock will return itself from any method call that hasn’t got some other expectation set up. When I’m testing that my index methods render the appropriate views for the format, I don’t care that all the various variables have been set up correctly – I’ve already tested that in another description – I just want to get to the point where I’m about to render a template.

What this does is decouple my tests. I can change the way that the index method fetches its stuff from the database and I’m only going to have to change the innards of the specs that test that.

I tend to think of this as being analogous to the object oriented pattern of trying to write your methods at a single level of abstraction. Within a given spec, I should only be setting up expectations that are directly related to what I’m testing.

What the?

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 13 May 2007 03:42:00 -0500

Wow:

$ rake spec
...

156 examples, no failures

$ ./script/spec spec
...

156 examples, 2 failures

$ rake
[unit tests, all pass]
[functional tests, all pass]
[specs...]

156 examples, 2 failures

For extra points, the 2 failures from running the plain rake are not the same as the failures from running ./script/spec spec. And if I run ./script/spec spec after a full rake run, I get a host of extra failures.

I wonder what I’m doing to so comprehensively screw up test isolation.

Ho hum.

Things which are fun

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 08 May 2007 02:12:00 -0500

It’s definitely fun to commit a major rework of something that’s been bugging you to SVN. It’s slightly less fun to check it out in your production server and have it fall over until you remember you to retweak the environment.rb file.

Anyhow, we’re now running on Typo 4.1.1+r1438. Typo 4.1.1 got released last night and, pretty much as soon as it had cleared the gates I checked in a major rework of the Article and Feedback state mechanisms.

What state mechanisms?

When you’re modelling blog posts, and comments, it’s very easy to think of them in terms of simple status flags. An article is published, or its a draft, so we have a #published boolean and we’re done.

Except, what about an article that you want to publish at a specific time? And how do we work out when to send notifications and trackback pings? Should you send the pings again when an article is first published then withdrawn, then published again?

It’s complicated, and you can end up with a rats nest of complex conditional code.

So, you work out that an article can be in one of several states. We went with :new, :draft, :just_published, :published, :publication_pending, :just_withdrawn and :withdrawn, and we used the State Pattern to handle this. All our tangled conditionals were replaced with simple delegations to the state object. There’s still some conditional code, but there’s a great deal less of it now.

Until recently, we implemented the states using Rails’s composed_of helper, but it’s not really suited to the task – you know your code isn’t happy when you find yourself calling a class ContentState::Factory and overriding its new method. It was also really hard to divine the workings of the state machine.

So, during a discussion of Coding without ifs I found myself sketching a way of setting up a state field declaratively and thought to myself “Hey, I could implement that for Typo and make things a good deal clearer…”

So I did.

Now, if you look at the top of article.rb, you’ll find:

  has_state(:state,
            :valid_states  => [:new, :draft,
                               :publication_pending, :just_published, :published,
                               :just_withdrawn, :withdrawn],
            :initial_state =>  :new,
            :handles       => [:withdraw,
                               :post_trigger,
                               :after_save, :send_pings, :send_notifications,
                               :published_at=, :published=, :just_published?])

  include States

And it sets up the delegations (as well as a set of #new?, #draft?, ... predicates based on the names of the valid states) for you. I pulled the States module out into a separate article/states.rb file, but given a simple set of states there’s no reason not to declare them inline.

I’m really pleased with it. I’ll probably do a bit more work on the interface (I want it to be more language like), add a has_strategy declaration (which I intend to use for our feedback spam checking system) and then extract a plugin from it. If you’re interested in doing something similar in your own code right now, the thing you need to pinch is lib/stateful.rb in the Typo distribution, and the only documentation is the source. Check app/models/feedback/states.rb and app/models/article/states.rb for examples of states written to this interface.

Other things which aren't fun

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 07 May 2007 14:29:00 -0500

Last year, you would have been forgiven for thinking that Typo was pretty much dead in the water as an ongoing project. Typosphere was a placeholder, changes were few and far between, the app was a bloated monster. So, people switched, in droves, to Mephisto the new, (and excellent) kid on the block. Heck, even Tobi, the original author of Typo, has switched.

Mephisto’s a great piece of software, and I’m as sure as I can be (without taking a closer look) that its underpinning code is cleaner than typo’s. When you’re developing on a rapidly moving platform like Ruby on Rails, there could be said to be a second mover advantage – the later you start, the more likely Rails is to already do what you want, and the less likely you are to zag where Rails later zigs.

However, Rick and Justin have other projects and demands on their time, and once something is Good Enough, it’s hard to summon up the motivation to make it better until you find something that you want your application to do.

So now, as Typo comes out of hibernation, it seems that Rick and Justin have been getting it in the neck because their project is ‘stalled’. Okay, so I admit, I did smile ruefully to myself when I read that.

Then I read the comments. Wow. Some people have some serious entitlement issues.

It’s been said before, it’ll be said again I’m sure: if you don’t like something about an open source project, “Patches welcome!”. If your patches are repeatedly rejected: it’s open source, fork off. If a project is moribund: offer to help.

Complaints just put people’s backs up.

I think it’s great that there’s choices available if you want a blog built Rails. Different perspectives on a problem space are important – if nothing else, everyone gets to steal from each other, everyone’s apps get better and everybody wins.

Things which aren't fun

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 07 May 2007 09:13:00 -0500

Let’s say you’re running tests against your rails application and a test fails.

“Hmm…” you think, “I wonder what could be causing that, let’s run that test file by itself.”

The test passes.

“Okay… I wonder if it’s the rake test loader, let’s run that test file using that.”

The test passes

“Right, it’s only the third test file on the list that’s failing, let’s try running just the first 3 test files.”

The test fails.

“Okay, so it’s one of the first two test files that’s causing the problem. Let’s remove one and see what happens.”

The test passes.

“Right, because I’m paranoid, let’s try running it with the other test file instead. It should fail.”

The test passes.

“Hmm… the failure appears to be fixture related, let’s try turning of use_transactional_fixtures in test_helper.rb and running the failing set of tests again.”

The test passes.

Houston, we have a problem

Part of our problem is that Typo is so huge and old, and its test suite is somewhat spotty. Something we’re doing in the workings of a couple of our controllers is confusing transactional fixtures and that can’t be good. I suppose my first step is to get all the tests that are still failing with transactional fixtures turned off passing and see if that solves my transactional problem, but I can’t pretend it’s a fun option.

Rails, CRUD, REST and the importance of adjectives

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 18 Apr 2007 03:23:00 -0500

Consider a web request:

GET /articles/2007/04/17/adjectives-rule;edit

“What does it mean?”

If we pass it through the prism of Rails routing, it breaks apart as follows:

GET: The :method, sometimes called the verb.
/articles: The :controller, the thing that’s going to handle this request
/2007/04/17/adjectives-rule: Parameters, passed to the controller method that handles this request
;edit: The :action, some way of specifying what this request does

It’s certainly one way of looking at it, but if you do I think you’re limiting your understanding of what it means to program in a RESTful fashion. It’s an obvious fit for Rails where the dispatcher builds an appropriate controller and then does controller.send params(:action)[1] but it’s a very ‘imperative’ view and that’s not necessarily a good view. The problem is that :action at the end.

What’s happening is that Rails’s implementation is leaking into the abstraction that is the URL. If you let that happen you end up programming with CRUDdy blinkers.

Let’s try another way of breaking that request up into ‘parts of speech’. Here’s what I think the request means: “Get me the editor for the article posted on 2007-04-17 called ‘adjectives-rule’”. ;edit isn’t an action selector, it’s a view selector.

“Isn’t this just navel gazing?”

At the London Ruby Users’ Group on Monday, there was an occasionally heated discussion of RESTful programming in which several people expressed the view that REST was limiting because they “needed more than 4 verbs” for their application.

Much of the discussion of RESTful programming on Rails has been about how to break things down so that you can think of your application in CRUD terms, and that’s a terribly limiting way to view things.

Let’s look at another request and see how to interpret it.

POST /articles/new;preview
Content:
article[title]=Adjectives+Rule;
article[body]=...

If you break this request up along the lines suggested by rails, then it seems that this isn’t a RESTful request. It’s POSTing something, which means its creating a new resource, but that resource isn’t going in the datatabase. Wha?

I think this is perfectly RESTful though. What we’re saying is “Make me a new preview of the article described in the body of the request”. The POST part of the request implies ‘make something’, the rest of the request describes what to make. Our controller knows that articles created for preview are not persistent, so they don’t go in the database. The preview you create and then throw away is just as much a resource as the finished article, the only difference being that every preview gets thrown away.

Looked at like this, it’s apparent RESTful programming is much richer than the basic 4 CRUD actions. The important principles to bear in mind are:

GET must not alter any resource: This is the really, really, really big one. If you’re ever tempted to write a method that interprets a GET request and changes the state of the resource being fetched… well, think again.
Statelessness: This is the idea that the request itself contains all the information the server needs to build the response. No session cookies. There are those who argue for no cookies at all, but until browsers handle HTTP authentication better I think we’re stuck with cookie based authentication, but the cookie should really be self contained and not an opaque reference into a catch all session table.
POST, PUT and DELETE should act appropriately: POST makes new stuff, PUT changes old stuff, DELETE removes old stuff. Not that hard to understand. The only wrinkle is that browsers don’t know how to make PUT and DELETE requests. Your AJAX actions will make them happily, but if use them in a form field, the browser will be confused. Because you can’t simply use JavaScript everywhere, you’re stuck tunneling those request types through POST. Rails has mechanisms to hide the tunnelling from you.
Persistent URLs: If you GET something from /some/where/or/other today, you should be able to get it there tomorrow unless it’s been deleted. If the resource has been moved, the right thing to do is to respond with a redirect (permanent or temporary as appropriate). Serving up an entirely new resource is right out. Which doesn’t mean you can’t have a resource like /articles/at_random – the resource you’re getting isn’t the article you receive, but ‘a randomly chosen article’. What would be bad would be having /articles/at_random not expire for 10 days.

“You haven’t mentioned adjectives yet…”

This article is a write up of something that occurred to me during a conversation after Monday’s LRUG meeting. Someone (whose name I either didn’t get or forgot) asked me what was the point of using RESTful principles in the first place. During the course of my answer, it suddenly clicked that what Rails calls actions aren’t really actions; they are adjectives. So, when I sat down to write this, I was expecting that that’s what I was going to say. But the trouble with late night pub conversation insights is that they aren’t always accurate or complete. As I wrote, I realised that my basic insight that an :action isn’t really an action was okay, but an :action is actually a view (I think it might be a controller when you’re POSTing though, just not a Rails controller).

“I’m curious now, what is the point of using RESTful principles?”

RESTful programming is a discipline. It’s a set of practices and patterns that help you make good decisions about how to solve problems in your application domain in a consistent way. The hope is that, by being rigorous in your application of the principles, you’ll end up with a website that’s easy to use, easy to cache and easy to scale. Certainly many RESTful practices are just out and out good practices.

There are other disciplines. Seaside is one. As Avi has shown recently, Seaside isn’t a smalltalk application framework, it’s a state of mind, and a weird one at that.

A thought

Here’s a little something to consider:

Thesis

Rails

Antithesis

Seaside

Synthesis

I dunno, but I’m looking forward to it.

Coming up

I plan to take a closer look at some of the principles and patterns of RESTful architecture and discuss how to apply them in Rails.

¹ Almost certainly not exactly what happens, but close enough for jazz.

Mmm... parsers

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 14 Apr 2007 14:13:00 -0500

So, in my quest to get Rails routes to accept routes like:

articles/:comment[article]/comments/:comment[id]

I’ve been playing with parsers for the first time in my programming career. Quite how I’ve managed to get this far without them I leave as an exercise for the interested reader.

At the moment I have a parser that will parse

articles/2007/05/12/slug

and give back an ActionSelector that yamlizes as

--- !ruby/object:ActionSelector
  controller: articles
  action: show
  params:
    :article:
      :year: 2007
      :month: 05
      :mday: 12
      :slug: slug

Which can be easily turned into the kind of params hash that rails wants from its routing system in order to dispatch the request to the right place.

For my next trick, I need refactor the parser I have so that it’s generated via a set of parser builders. Then I need to write a parser for routing specifiers that can use those builders to build a URL parser.

It’s a simple matter of programming I tell you.

Updates

RouteBuilder works (2007041608:00ish)

I now have a RouteBuilder that generates routes and has the same interface as ActionController::Routing::RouteBuilder.

My Route doesn’t have the same interface as AC::Routing::Route yet, but it’s not far off. The fun part is working out which bits of the interface are essential and which can be safely ignored. I’m hoping that I won’t have to rewrite AC::Routing::RouteSet as well, but I’m not sure how complicit it is with the innards of AC::Route.

Routes can generate paths from options (2007041612:30)

context "Given a default route" do
  setup do
    @route = RouteBuilder.new.build('/:controller/:action/:id')
  end

  def gen(hash)
    # Rails interface is a bit surprising
    @route.generate(hash.dup, hash.dup, {})
  end

  specify "generation works" do
    gen(:controller => 'accounts', :action => 'show', :id => 1) \
      .should == '/accounts/show/1'

    gen(:controller => 'accounts', :id => 1) \
      .should == '/accounts/index/1'

    gen(:controller => 'accounts', :action => ':index') \
      .should == '/accounts'
  end
end

It’s nowhere near complete, but I’m being gung ho and starting to integrate my RouteBuilder with Rails. Then I can start pulling the Rails routing tests into my own test suite and get some confidence with its robustness before I start adding the ability to handle routes like /books/:tune[book][id]/tunes/:tune[id], which is why I started this in the first place.

With any luck and some good train hacking time on the way to London I should have something good to show at LRUG this evening.

Wanted: module/starter.rb

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 14 Apr 2007 04:17:00 -0500

I sometimes think that one of the reasons that CPAN is such a huge advantage for Perl is the ease with which you can contribute to it. It’s all very well having a tool for installing libraries from the archive, any fool can do that, but CPAN has tools for getting started with a new library too.

If I want to start a new Perl project, I do:

module-starter --module=AuthenticationFairy

and the script goes away and builds me an AuthenticationFairy directory complete with all the boilerplate stuff filled in. I know that when I look in AuthenticationFairy I’ll find a sensible directory structure, a Makefile.PL that will do the right thing, a lib/AuthenticationFairy.pm with the documentation boiler plate filled in and a test directory. And I rejoice because my yaks have been sh aved and I can get on with the interesting business of writing my first test and making sure it fails.

I’ve not found an equivalent for Ruby. Yet.

I have found sow, which is part of the Hoe package, which is definitely a start.

Are there any other tools along the lines of Module::Starter that I’ve missed?

You are in a maze of little, twisty evals, all similar.

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 12 Apr 2007 02:06:00 -0500

Rails routing is making my head hurt. It’s starting to remind me of the guts of perl’s regular expression engine, which was said to be understood by three people, plus or minus five.

I just want to write routes like /articles/:art[year]/:art[month]/:art[mday]/:art[slug].

I’m seriously considering starting again from scratch, which can’t be right.

The "Yes is No" Problem

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 04 Apr 2007 05:33:00 -0500

login: pdcawley
password: *****
Yes is no and no is yes. 
Do you want to delete all your files? [y]: _

Rereading What’s wrong with Ruby, I realised that the meat of Matthew Huntbach’s worry about dynamic classes boils down to the ‘yes is no’ problem. Given just the source code of a Ruby program, it can be tricky to work out what the eventual program ‘looks like’.

Ruby source can be thought of as a program that you run on the Ruby kernel which creates a bunch of classes, objects and relationships between them. In static languages, once a class has been created, that’s it, the only way to change it is to recompile and restart the program. In dynamic languages, no class is every really ‘finished’, it can be modified at any time.

Done right, that can be a great thing. It’s instructive to look at the way ActiveRecord’s behaviour gets built up in Ruby. ActiveRecord::Base doesn’t really do all that much, it concerns itself with the nuts and bolts of marshalling data to and from the database and building the basic accessors and support methods needed to do that. Its implementations of basic methods like #create, #update and #save are positively naïve. New behaviours are then layered on by including other modules. Modules like ActiveRecord::Locking::Optimistic, ActiveRecord::Validations, ActiveRecord::Callbacks and others all layer behaviour on top of these methods, usually by renaming the old method, so #update becomes #update_without_lock and ActiveRecord::Locking::Optimistic#update_with_lock becomes the new #update, at least until ActiveRecord::Timestamp changes its name to #update_without_timestamps and installs the behaviour that it needs.

It can make the code harder to understand though. Say you’re looking at ActiveRecord::Timestamp#update_with_timestamps and you want to look at how #update_without_timestamps does its thing. If you simply grep for def update in the source code, you’ll find the implementation in ActiveRecord::Base and miss out on the alterations introduced by the locking module. You can’t simply look at the source code, you have to look at the order in which it was executed, and that can be a headache. Once you understand that you need to look at load order, and you find where that is defined, things get much easier, but things can get seriously scary when plugins come into play too.

What’s needed (or at least desirable) is a way of understanding what the state of the program and its classes will be at runtime. It’s easy in static languages, it’ll look like the source code says it looks. It looks like the source code says it will look in Ruby too, but Ruby source code can get a good deal more twisty. Even attr_accessor has the potential to blow minds…

Would you believe it’s a solved problem? It’s been solved for years too. Heck, the solution is older than David Heinemeier Hansson.

Counsel of Perfection

I’m thinking of Smalltalk, but I’m pretty sure that the Lisp Machine folks had similar tools at their disposal. In a Smalltalk image, you can browse the source code at any time, and it always reflects the current specification of the system. In a Smalltalk implementation of ActiveRecord, as soon as you rename #update to #updateWithouLocking that’s what it’s called. When, later you’re looking at the definition of #update and you want to know what #updateWithoutLocking looks like, you can browse straight too it. The knowledge that #updateWithoutLocking used to be called #update hasn’t been lost, you’ll find it in your changes, but it’s irrelevant to how the system behaves now.

I’ve been admiring Smalltalk from afar for a long time now, but until recently it wasn’t a language I spent any time programming in. I was more likely to think “How does Smalltalk do that?”, then fire up my local Squeak image and do a little fossicking about until I had a grasp on how things worked. Then I’d close it down and do something similar (or curse because I couldn’t) in Perl or Ruby. You can learn a great deal just doing that; if nothing else I’ve learned that I like Smalltalk’s message selector syntax a great deal.

Then, a few weeks ago, I watched a couple of screencasts of experienced Smalltalk developers doing their thing and… wow. I’ve always known that Smalltalkers rave about the environment, but I’d not really seen it in full flow.

So, I’ve been doing a few Code Katas in Squeak and starting to get a feel for working in the environment. It’s really, really lovely. Watch this space for more ‘coo er gosh’ type posts. Who know, maybe even an embarrassing screencast – if nothing else but so I can get some feedback from experienced Smalltalkers on what I’m doing wrong.

What about Ruby?

Short of someone coming up with a Smalltalk style interactive development environment for Ruby in the next half hour, this isn’t an immediately useful solution to the problem in Ruby.

But is it really a problem?

Having taken the time to explain why dynamic modification of classes can be a problem, I should point out that I don’t think it’s really a problem in practice. Sure, you can do Very Bad Things with it, but so what? You can do the same bad things (and more) in Smalltalk, if you couldn’t, it would be (nearly?) impossible to implement the Smalltalk environment in the first place. In the absence of an interactive programming environment, you need to take more care with structuring and organizing your code. Impose some standards on yourself – in Rails for instance, the various ActiveFoo classes are structured similarly: first there’s the ‘active_foo.rb’ file, which is responsible for loading up all the packages that define the class (generally found in the ‘active_foo’ sub directory) and includeing them in ActiveFoo::Base in the correct order. This convention helps comprehension. About the only thing I’d like to see further is some documentation in each module describing the behaviour they expect of any class they’re included in – behaviour like that of ActiveRecord::Validations is just too useful not to be pinched for other classes.

As always with these things, it boils down to the good taste and skill of the programmer. Write code that communicates your intent clearly to your fellow programmers – mere compilation is the least of your worries.

Updates

In the comments, Phil Toland has come up with links to some Smaltalk screencasts and a long presentation from Avi Bryant that includes some interaction with a Squeak image.

NB: For ‘ruby’, you can read ‘pretty much every dynamic OO language ever’. Except Matthew Huntbach didn’t write an article called “What’s wrong with pretty much every dynamic OO language ever”.