Just A Summary

Right, we’ve cut a Typo 5 gem and it’s on rubyforge and heading to various mirrors I hope. Frédéric’s writing the release notification which will be appearing on Typosphere Real Soon Now.

It’s been a surprisingly tricky process – we’re now requiring Rails 2.0.2 because the workings of view_paths have changed in a way which means we can’t quite make themes with Rails 2.0 and 2.0.2 and working with the edge seems like the more sensible proposition. If you’re on the bleeding edge, you should find that you get the right Rails via svn:externals anyway.

Typo futures

Meanwhile, I’ve been playing with stet and I’ve come to the conclusion that, although there’s mileage to be had in a radically slimmed down approach to the way Typo works, I’m better off simply removing the misfeatures from Typo and building from there – there’s a surprising amount of stuff that needs to be done in a competent blogging engine that Typo gets right – starting again would be throwing the baby out with the bathwater I think.

However, this does mean that if you’re following the Typo SVN trunk, you’ll be seeing a reduction in features in the short term. We’ll be copying the current trunk to a 5-0-stable branch before we start with the featurectomies though, so if you’re just after bugfixes, you’ll be better off there.

Multiblogging

We’re aiming to have multiblogging in the next release, but we’re rethinking the how of it. Right now, the ‘Blog’ object adds a bunch of complexity to code that would be much happier simply assuming that it has the database to itself. So we’re going to look at switching to a database per blog approach, that way our core code can pretty much forget about the complexities of multiblogging, and (at least initially) anyone who wants multiblogging can get there by monkeying with configuration files – of course, we intend to add a web based admin interface once things settle down and we know how things are going to work.

Caching

Caching is always a bugbear in any typo installation. Because we want to be installable on the widest possible range of hosts, we can’t rely on the presence of handy tools like ‘memcached’. Also, some of our users are operating under some fairly severe memory and process constraints, so it makes sense to have the webserve serve static files as much as possible. Meanwhile, tools like Evan Weaver’s Interlock are pointing the way towards seriously effective fragment caching. I shall be looking into implementing something that conforms to the interlock interface, but which can use an arbitrary cache backing store for fragments and maintain a full page cache. It’ll be interesting to find out if this is doable…

Atom Publishing Protocol

ActionWebService is going to go away – it’s already in the ousted branch of the rails SVN repository, and including it in Typo to support the various different admin APIs is getting painful. So, we’re going to preempt it. We won’t be getting rid of the various XMLRPC APIs until the pain becomes too great, but we are going to be concentrating on implementing, and strongly favouring, the Atom Publishing Protocol.

Feeds for everything

In particular, we’ll be adding atom feeds for all sorts of administrative data as a means of enabling people to write external tools for, say, spam protection, comment moderation and notification tasks. Right now, there’s a great deal of computation happening on the server side every time someone, say, comments on a post – in the kind of resource limited environments some people are running Typo in, that’s too much work. Switching to a feed + APP approach should help enormously with resource utilization.

Speaking of resources…

Using the server to render article previews is… suboptimal. Expect to see a javascript based preview system akin to the one I use for comments here.

So, if you’ve been watching the Typo tree, you’ll see there’s been a fair amount of activity on it since Rails 2.0 got released. There’s a new default theme replacing the rather creaky ‘azure’, and a fair amount of work on getting our code compatible with the current state of Rails. As we work on this, it becomes apparent that Typo’s code is getting horribly brittle. I have said before that there’s been several places where we’ve zigged before Rails zagged, and we’re paying the price for that. It doesn’t help that our test coverage is distinctly ropy either – and I’m probably guiltier than most for letting things get into that state.

So, our goal is to get what we have cleaned up and working with Rails 2 before releasing Typo 5.0. Once that’s done, that line of code will go into maintenance mode – there are still plenty of bugs to fix and documentation to write, but I’m afraid that extending that base is becoming too much of a chore.

Which is why I have a new path in my local svk repository, //stet. I’m using this for experimental development of a new, slimmed down blogging engine that will be, first and foremost, a capable Atom Publishing Protocol host. Things like spam processing will be removed from the core of the application, but we’ll provide a suite of webservice clients that will consume the ‘unmoderated feedback’ webfeed and use APP to either approve or delete the feedback as appropriate.

Theming (at least initially) will probably be confined to Javascript and CSS changes, and I’m even thinking of exposing the sidebars as Atom collections – certainly I expect that, in the first cut, sidebars will be static – if you want content that looks dynamic you’ll have to do it via javascript.

My initial goal is to slim things down as far as I possibly can – I want to build a blogging engine that can cope with the tight memory constraints of shared hosting by off loading much of the heavy lifting to client boxes. After all, I have far more processing capability available to me on the laptop I’m typing this on than the slice of Site5’s hosting infrastructure that’s actually running the blog. By making things small and static, I also hope to wring good performance numbers out of the tool as well – expect aggressive page caching at the very least.

Another important goal is easy migration of Typo databases. I expect to be writing models and controllers from the ground up, but converting the database should just be a matter of running a migration.

Experimental

Of course, stet’s currently very experimental – about the only thing that’s actually written so far are a couple of routing plugins which should help radically simplify our routes.rb (expect an article’s url to change from /articles/2007/12/16/rails-20-and-the-future-of-type to /article/2007/12/16/rails-20-and-the-future-of-typo, but with a redirect in place to cater for the old style urls). I may have grandish plans for the thing, but I could equally discover that I’m off up a blind alley, in which case you can expect me to return to the current typo codebase with a few more lessons learned.

ActiveResource?

I remain unconvinced by ActiveResource as a technology. I agree with the authors of RESTful Webservices – good webservices are joined up. They take full advantage of what could be described as the defining technology of the world wide web, the URL based hyperlink to knit resources together in a discoverable fashion. An ActiveResource based webservice may well be a good HTTP citizen, but it’s still not really ‘webby’ enough for my taste. Which means the Atom Publishing Protocol will remain my friend for most of the things I hope to do with stet. It may be harder to write a good APP server, but I’m convinced that it’s a much better interface for clients, and you should always favour ease of use over ease of implementation. If nothing else, we’re aiming to have more users than developers. Many more.

Here’s a problem I first came across when I was about 13 and helping do the stock check at the family firm. The parts department kept all their various spare parts racks of parts bins. Each bin was ‘numbered’ with an alphanumeric id. We had printouts of all the bin numbers along with their expected contents and we’d go along the racks counting the bins’ contents and checking them off against the print out. What confused me at the time was the way the printouts were organized. Instead of the obvious ordering, “A1, A2, A3, ..., A99”, the lists were ordered like “A1, A10, A11, ..., A2, A20, A21, ...”. After a bit of thought I realised that the computer was sorting the numeric bits of the bin numbers as if they were just sequences of strange letters. A bit more thought made me realise why, post computerisation, people were starting to use bin numbers like “A01, A02, ...”. Computers were more important than people so, in order to make sorting things easier, just add spurious leading 0s to make the number field a fixed width and Robert’s your parent’s brother.

27 years later and computers are still crap at sorting things in a sensible fashion. Back before Moore’s Law was really kicking in, I suppose it was excusable, but surely we’ve moved past that now.

Over on the labnotes blog, there’s an example of some ruby code that attempts to do ‘human’ sorting:

module Enumerable
  def sensible_sort
    sort_by { |key| key.split(/(\d+)/).map { |v| v =~ /\d/ ? v.to_i : v } }
  end
end

It’s okay, as far as it goes. It certainly solves the parts bin problem I outlined above, but it’s not ideal. For example, you might expect ['-1', '1', '1.02', '1.1'].sensible_sort to leave the order unchanged, but what you actually get is ‘1, 1.02, 1.1, -1’. Not ideal. Let’s rewrite sensible sort as

module Enumerable
  def sensible_sort
    sort_by {|k| k.split(/([-+]?\d+(?:\.\d+)?(?:[-+]?[eE]\d+)?)/).map {|v| Float(v) rescue v}}
  end
end

That ugly regular expression should match a far wider selection of string representations of numbers. Certainly our ‘bad’ list is now sorted correctly.

But what about “a-1”, “a-2”. Using the implementation above, they’d get sorted as “a-2, a-1”, which can’t be right, can it? Let’s extend it a bit more and make sure we only worry about the ’+’ and ’-’ if they’re at the beginning of a line or preceded by whitespace.

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v}}
  end
end

And that works fine, until you find that “B” sorts before “a”. Let’s catch that as well:

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

Yay!

Oh, wait a minute, what about version numbers? How should we sort, say “perl 5.8.0” and “perl 5.10.0”? The 5.8.0 form should definitely come first… Hmm…

How about

module Enumerable
  def sensible_sort
   sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

How far down does this thing go?

I just noticed that ”.1” sorts after “1”. Time for another tweak…

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?(?:\.\d+|\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?))/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

but that doesn’t work with version numbers like ”.8.2”, ”.10.2”...

Time passes… Thorin sits down and sings about gold

I was planning on giving an extension of the regex that caught this issue as well, but I’m afraid I’ve stumped myself – I can’t do it with a single regular expression unless I can use a fixed width lookbehind assertion, but they’re only available in Perl. Of course, it’s still possible to fix it, but doing so will take more thought than I have available to me at this time on a Sunday morning. And all this is before we get onto making sure that “1/2” sorts between “0” and “1”. And phone numbers. After all, “01915551238” is ‘obviously’ the same as “0191 555 1238” and “0191 555-1238”, so they should end up next to each other in the sorted list.

It looks like this is a ‘three pipe problem’ after all. I shall probably return to this…

Worrying about test coverage when you’re doing Test- or Behaviour-driven development is like worrying about the price of fish in Zimbabwe when you’re flying a kite.

Your tests are there to help you discover your interface and to provide you with an ongoing stream of small bugs to fix. If you write them cleanly, and keep them well factored (you are refactoring your tests, aren’t you?) they will help to document your intent too. Ensuring that every code path is exercised might be intellectually satisfying but that satisfaction costs time, and time is money. And that’s before you start worrying about your code’s malleability. Cover the happy path and the edge cases you know how to deal with and move on. If you’ve screwed something up, it will get found during acceptance testing (or out in the wild) and you can write a few more tests to isolate the problem, fix it, and move on.

If you deliberately add a test that passes without requiring you to write another line of code, ask yourself why you bothered with it. The test isn’t isolating a bug or specifying new behaviour. If you’re lucky, it merely confirms something you already know, and if you’re unlucky, you just introduced a bug in your test suite. Better to move on and either pull a new bug off the queue, or turn a feature request into a new bug – “Feature X doesn’t work!” – fix it, refactor and move on to the next. Further down the road, you may discover that the feature you were about to exhaustively test doesn’t even need to be there, or maybe it works differently than you expected. Aren’t you glad you didn’t worry about 100% coverage then? Maybe you will need to revisit the tests and cover more cases in the future. But future you knows more about the problem domain and can do a better job of working out what the behaviour needs to be. And if future you is likely to do a worse job than you right now, you may have bigger problems than the code to worry about.

Has anyone written an Atompub client in JavaScript yet?

Do you have a PC, Xbox 360 or anything else that will run the Valve Orange Box? If so, get hold of a copy and fire up the Portal subgame.

That is all.

So, we’re doing a cobranding exercise at work. The idea being we serve up a branded version of amazing tunes in a subdomain of our partner, their users get a skinned version of the site that feels like part of the partner’s site, we get an influx of new users and everybody is happy. One aspect of this is we’re using the partner’s site to handle authentication.

Today, we got all our ducks in a row and started authenticating against the partner’s SOAP service as part of our user testing. So off I went to the partner site and set up an account…

At amazing tunes, we’re pretty scrupulous about password security, we never store plaintext passwords, any request that involves a password being sent is done over an https connection. It’s just the right thing to do.

After I’d finished setting up my account on the partner’s website I was presented with a screen that looked something like:

Your username is: pdcawley

Your password is: fucknuckle

At least it was an https connection, but it doesn’t exactly fill me with delight.

Yes, I’ve upgraded to Leopard. Yes, it’s spiffy.

The headline features like Time Machine are great, but I’m loving the little details more.

For instance, the first time you go to run an application that’s been downloaded from the net, a dialogue pops up reminding you that the file has been downloaded and asking if you want to trust it. It’s a nice bit of security, and it asks you the question at the right time.

Tiger asks a similar question, but it does it at entirely the wrong moment. What happens is that Safari stops and shows you a dialogue that says something like “The file you downloaded may contain an application, do you want to continue?”. Well, of course I want to continue, I’m surfing the web, not trying to run this new app. My reaction to this sort of dialogue is to think “I just don’t care about that right now, go away and stop annoying me!”.

The way it works under the covers is pretty neat too. With any luck the OS X Finder team will take a leaf out of its book and think about replacing those annoying .DS_Store files with directory metadata along the same lines. But I’m not holding my breath.

It’s been a while since I wrote anything here, mostly because I’ve been indecently busy. Well, up until last Sunday when I managed to trip over a low wall in our garden and, in the process take off a large amount of the skin on my left shin. A mere flesh wound, but painful, embarrassing and it’s left me sporting what looks like a foot long, white, elastoplast. The worrying bit is that, when I went to get my wounds dressed, the nurse took one look at my bulk and demanded a urine sample. Which tested positive for glucose. For the second time in a year. So, interspersed with getting my dressings changed at the GP’s, I got a blood test and it looks like I’m diabetic.

Given my weight and what I understand about the workings of late onset diabetes, it’s not really surprising. I’m sure it’s better to get an early diagnosis than a late one, but it does rather give one pause for thought. Somehow I doubt I’m going to find it easy to change my diet and lifestyle; if it were easy, I would have done it already. I like food. I don’t like exercise – it’s a pain in the arse, time consuming and boring. About the only form of outdoor exercise I ever enjoyed was sailing, but the old adage that you can recreate the sensation of sailing by standing in a cold shower tearing up 50 pound notes is depressingly accurate and I simply don’t have the 50 pound notes to spare.

Crap.

Anyway, normal blogging service should be resuming shortly; I’ve got a couple more nouns to write up as essays, and a long, half written, tutorial on writing pidgins in Ruby. Watch this space.

Some bugs are easy to overlook. One that has a habit of catching me out is a Rails filter that returns false occasionally when it’s being evaluated purely for its side effects. Here’s how I’ve started working round the issue:

def side_effect_filter
  return if some_conditions_not_met?
  ...
ensure
  return true
end

What happens here is that the ensure catches any return and returns true instead. The catch is that if something throws an uncaught exception anywhere, it too gets caught by the ensure and true is returned. Which may not be what you were looking for. Here’s how to fix that issue:

def side_effect_filter
  error = nil

  return if some_conditions_not_met?
  ...
rescue Exception => error
ensure
  raise error if error
  return true
end

This catches the exception in a rescue and stashes it in the error variable, then the ensure checks to see if an exception was thrown and rethrows it, otherwise, it just returns true. Which is bulletproof, but ugly. Let’s wrap the ugliness up in a method:

def self.side_effect(method, &block)
  def_method(method) do
    error = nil
    begin
       instance_eval(&block)
    rescue LocalJumpError # catches an explicit return
    rescue Exception => error
    ensure
      raise error if error
      return true
    end 
  end
end

side_effect :side_effect_filter do
  return if some_conditions_not_met?
  ...
end

Again, not pretty inside, but all we actually care about anywhere else is that the interface is good and does what it’s supposed to do. Encapsulated ugliness has its own beauty. Especially if you get the interface right.

Homework

This should pluginize quite nicely, just install the method in ActionController::Base and ActiveRecord::Base and you have a very useful tool, but I’m still not sure that the method name is right, so I’m holding off on it. If someone were to come up with a bulletproof name and release a plugin, that would be wonderful though.

Updates

Fixed a scoping issue in the encapsulated version of the code. Replaced yield with instance_eval(&block)