Just A Summary

There’s something satisfying about reaching the point when you can’t decompose an object any further and all your methods are tiny and do one thing – it’s especially gratifying when you learn something new in the process. Sadly, it doesn’t happen as often as I’d like, there’s usually annoying bits and pieces where you have to placate the language in some fashion that breaks the flow of what you’re writing.

As I get a better handle on the way MooseX::Declare has changed Perl, I’m finding I have to do much less in the way of placation.

Here’s an example. For context, I’m writing a traffic shaping tool. The basic client interface needs to look something like:

$policy->current_weight # => a percentage between 0 and 100

Not much of an interface really. The requirements state that we should be able to specify weights with 15 minute granularity for every day of the week. Our problem becomes one of mapping from a time to a number between 0 and 7 (days) * 24 (hours) * 4 (quarters) - 1 and looking up the weight in an array.

Here’s my first cut:

use MooseX::Declare;
class WeightVector {
    use DateTime;
    
    has vector => (
        isa => 'Array[Int]',
        is => 'ro',
        required => 1,
    );
    
    method current_weight {
        my $now = DateTime->now;
        my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
        return $self->vector->[$offset];
    }
}

Which is, I suppose, perfectly respectable. However, current_weight isn’t filling me with delight. First it finds the current time, then it converts the time into an offset, then it uses the offset to lookup the weight in the vector. Let’s introduce a method to find the weight at a specific time"¹":#decomp-motivation, the relevant code becomes:

method current_weight {
    $self->weight_at(DateTime->now);
}

method weight_at (DateTime $time) {
    my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
    return $self->vector->[$offset];
}

And again, we could rest here, but again, we’re doing two things. We’re converting from a time to an offset, then we’re looking up the value in the vector. Type conversions tend to happen again and again, so it’s good if we can specify them separately. We could write a time_to_offset helper method, but we’re in Mooseland now; there’s a better way. Let’s introduce a formal Moose type and define a coercion for it. Here’s the type definition stanza of the code. I’ve taken the opportunity to add types which do bounds checking for the vector as well, while I’m about it.²

use MooseX::Declare;
class WeightVector {
    use DateTime;
    use Moose::Autobox;
    use MooseX::Types -declare => [qw(SlotOffset VectorOfWeights PercentageInt)];
    use MooseX::Types::Moose qw(ArrayRef Int);
    
    use constant TOTAL_SLOTS = 7 * 24 * 4;
    
    BEGIN {
        subtype PercentageInt,
            as Int,
            where { 0 <= $_ && $_ <= 100 },
            message { "$_ does not is not an integeter between 0 and 100" };
    
        subtype VectorOfWeights,
            as ArrayRef[PercentageInt],
            where { $_->length == TOTAL_SLOTS }
            message { "Vector must have ".TOTAL_SLOTS." entries, not ".$_->length };
    
        subtype SlotOffset,
            as Int,
            where { 0 <= $_ && $_ < TOTAL_SLOTS };
    
        class_type 'DateTime';
    
        coerce SlotOffset,
            from 'DateTime',
            via { ($_->wday_0 * 7 * 24 + $_->hours) * 4 + int($_->minutes / 15) };
     
        # Let's allow clients not to care about using DateTime by allowing 
        # them to simply pass the results of calling 'time()' - It's not like it's
        # still 1970...
    
        coerce SlotOffset
            from subtype(as => Int, where { $_ > TOTAL_SLOTS }
            via { to_SlotOffset(DateTime->from_epoch(epoch => $_) };
    }
    
    has vector => (
        is => 'ro',
        isa => VectorOfWeights,
        required => 1,
    }
    
    ...

Now, if we were programming in plain old Moose, we could rewrite weight_at like so:

sub weight_at {
    my $self = shift;
    my $offset = to_SlotOffset(shift);
    $self->vector->[$offset]
}

Which would be pretty sweet, but we’re using MooseX::Declare; there’s an even better way:

method weight_at (SlotOffset $offset does coerce) {
    $self->vector->[$offset];
}

Sweet!

We could stop there, but I had an insight. What we’ve got here is basically a wrapper around a delegation to our vector, and Moose’s new native types feature let us express the delegation to the vector quite neatly, like so:

has vector => (
    isa => 'VectorOfWeights',
    is => 'ro',
    required => 1,
    traits => ['Array'],
    handles {
        weight_at => 'get',
    },
);

...

around weight_at (SlotOffset $offset does coerce) {
    $self->$orig($offset);
}

This could be overkill when vector is a simple ArrayRef as we have here, but the pattern of delegating declaratively in the attribute definition and then munging arguments in an around handler is applicable to more than just argument transformation. A typical delegation pattern involves having the delegating object passing itself in as an argument to the method delegated to. The nature of Moose’s handles declarations makes that impossible to do within the attribute declaration, but it’s easy to fix with an around helper:

around delegated_method (Any @args) {
    $self->$orig($self, @args);
}

(If you’re wrapping more than one method in this fashion, you should probably consider using a plain old Moose style around handler, which lets you wrap multiple methods with around @delegated_methods => sub {...}

So, at the end of all that, and after we’ve extracted our Type declarations into WeightVector::Types, we have:

use MooseX::Declare;
class WeightVector {
    use WeightVector::Types qw(VectorOfWeights PercentageInt SlotOffset);
    
    has vector => (
        isa => 'VectorOfWeights',
        is => 'ro',
        required => 1,
        traits => ['Array'],
        handles {
            weight_at => 'get',
        },
    );
    
    method current_weight {
        $self->weight_at(time());
    }
    
    around weight_at (SlotOffset $offset does coerce) {
        $self->$orig($offset);
    }
}

And we’ve pushed all knowledge of DateTime off onto our type declarations and gained a boatload of handy bounds checking. We’ve also got a new tool for handling tricky delegation setups in the handles/around combo.

Notes

Motivation

I realise that this looks like a radical decomposition of the class with very little motivation, but it was driven by tests and by some other requirements that I’ve removed from the body of the post. In particular, the type coercions were driven by the need to build particular vectors for testing, a key method being:

method set_weight (PercentageInt $weight, 
                   SlotOffset $from does coerce,
                   SlotOffset $to does coerce)
{
    ...
}

Type coercion is wonderful

Generally, I’m not a fan of static typing. I’m from the “duck type all the way” school of programming,³ so most of my method declarations have no type declarations. But type declarations, especially ones that coerce, make so much sense on methods that make up the public protocol of a class. I only use type declarations on internal methods when I need a narrower coercion, or if I’m using MooseX::Multimethods, which I still haven’t used for anything but exploration.

Updates

Thanks to Chris Dolan for spotting that I’d got the SlotOffset coercion completely wrong. The real code’s doing the right thing, but that’s what comes of recreating code from memory.

¹ This was actually motivated by trying to write tests to verify that the weights were correctly set.

² I’m declaring these in a BEGIN block of the class itself mostly for explanatory purposes – there’s a good case for moving them out into a separate file and pulling it in with use.

³ Except during my periodic attempts to learn Haskell. I’ve learned Haskell at least three times now.

In Ruby, when you’re doing division on integers, things can get a little counter intuitive. For instance, 6/4 is obviously 1. At least, it is until you decide that you’d rather have numbers that behave a little more like ‘real’ numbers and you do require 'mathn', a module in the Ruby standard library (ie, it comes as part of ruby). Then you enter a whole new world of rational maths, where 6 / 4 returns 3/2.

Several very fine and useful Ruby gems rely on the workings of mathn, including ruby-units, which is a spiffy tool for avoiding problems when one team is working in kilometres and the other in miles and it’s no fun at all when your space probe is suddenly incommunicado.

Other fine and dandy Ruby gems include ultrasphinx and webrat. Both of these two (and no doubt others) rely on the the fact that 302/100 == 3.

Hmm… can you see my problem?

Please, if you’re working on a gem that you intend to publish widely, then adopt the practice of never trusting that dividing an integer by another integer will return a third integer. You’re not even making yourself a hostage to some other gem, you’re making yourself a hostage to the standard library. Always do (an_integer / another).to_i and your code will be so much more robust.

I’ve got a pull request and lighthouse ticket in for webrat and, once I’ve hit ‘publish’ on this post, I shall be doing the same thing for ultrasphinx, but I’m sure there are other gems out there with the same problems. Please people, check your assumptions.

Back in (crikey) February, I gave a talk at the London Perl Mongers’ technical meeting about Moose for Ruby Programmers and wrote it up here. Mike Whittaker was in the front row of the audience with his iPhone and, a couple of minutes in, started a voice recording and gave me a copy.

So… finally… I’ve taken the time I should have been using to write another article for The H and wrestled the slides and the audio into something like sync and uploaded the results to Vimeo for your viewing pleasure.

An introduction to MooseX::Declare from Piers Cawley on Vimeo.

I got a bit of stick on IRC last night for some of the choices I’d made when I was writing Test::Class::Sugar, in particular because one of the prerequisites is chromatic’s handy and opinionated Modern::Perl module. The ‘controversial’ aspect of Modern::Perl is that, when you use it, your code won’t run on any Perl before Perl 5, version 10.

The thing is, I don’t care about older perls any more. Version 10 features like ~~ and // are too convenient to fart about writing circumlocutions just to run on a version of Perl that I have no intention of ever using again.

Actually, that’s not quite true, those version 10 features are too convenient for me to fart about working around their absence. If you think that a module I write is useful enough that you want it to work on version 8, then of course I’ll accept your patch. But don’t be surprised if, when I start adding new features, I break the backward compatibility.

Also, on the happy day that version 10.1 escapes the pumpking patch, I’ll be setting that as my minimum perl version even if chromatic doesn’t bump the version number in Modern::Perl.

It’s all about the virtues

Your context is different from mine. I’m writing in Perl again for my own amusement more than anything. There are developments in modern Perl – tools like Moose and Devel::Declare – that I think are exciting and important. The Announcements project I started was as much about playing with the new tools as it was about trying to write something of wider utility. Test::Class::Sugar arose as a direct result of attempting to write Announcements and the desire to write test classes without hoopage. My principle drive then, is impatience to get Test::Class::Sugar to the point where I can get back to writing Announcements.

But then laziness and hubris kick in. So the code needs some polish. The parser and the code generator need to be disentangled, I need to get Adrian Howard to apply the little patch I had to make to Test::Class. Laziness demands I document it.

Impatience tells me to lean on the features of modern perl – that way I can get back to being a user of the new library as quickly as possible. Laziness tells me that I’m not going to need backwards compatibilty. Hubris tells me my work is good enough that someone who does will like it enough to send me a patch.

Everybody wins.

In which your correspondent does magical battle with the guts of Perl and emerges bloodied, but unbowed with a useful principle to code by.

Skip to the conclusion if you’re uncomfortable with the guts of the Perl runtime

Test::Class had me tearing my hair out earlier. There I was, happily transforming

test something {
  ok 1;
};

into something very like¹:

*test_something = 
  Sub::Name::subname(
      'test_something' 
    => sub : Test { ok 1 }
  );

through the magic of Devel::Declare, but Test::Class didn’t seem to be playing fair. Instead of letting my tests run happily, it was complaining that it:

cannot test anonymous subs – you probably loaded a Test::Class too late (after the CHECK block was run). See ‘A NOTE ON LOADING TEST CLASSES’ in perldoc Test::Class for more details

The thing is, I wasn’t loading Test::Class too late. The problem is that, at the point I applied the Test attribute to my sub, the sub didn’t have a name and, because of the constraints you’re operating under when you’re using Devel::Declare to do code transformation, there was no obvious way to give it a name in time.

Incompatible magics

The trouble is, Test::Class does what it does through the magic of compile time code attributes, and, further, it relies on the fact that if a perl subroutine that gets inserted into the symbol table like this:

sub has_a_name {...}

Then, when you get hold of a reference to that code by other means (say, in the subroutine that handles the setting of an attribute, that code ref knows its own name. However, if a subroutine that ends up in the symbol table like this:

*anonymous_ref = sub {...};

Doesn’t know its name, unless you take advantage of the Sub::Name module.

So, in my generated code, I was giving my coderef a name, but it was happening to late. At the point that Test::Class::Test method was seeing the coderef, the coderef was anonymous.

My magic and Test::Class’s magic were incompatible.

The thing is, both sorts of magic are really just sugar for some pretty mundane donkey work. Test::Class does what it does through attributes because no flesh and blood programmer in their right mind would want to write something like this every time they wanted to write a test method:

sub test_something {
  ...
}
__PACKAGE__->mark_as_special_method('test_something', 'test', '3');

In fact, mark_as_special_method doesn’t even exist as its own subroutine. The code that marks a method as special is just part of the body of the Test attribute handler.

Conclusion

Which brings me neatly to my conclusion.

When you’re designing a module that does anything magical, consider starting with a mundane core API that handles the business side of things. Then layer your magic on top of that API. Then document the API and the magic. Obviously the magic bits go up front in the docs, and the API goes in its own section (or even podfile) down at the bottom, where only eejits like me, who want their magic to work slightly different to yours, will bother reading it.

Obviously, I’m motivated by an issue I’m having with a particular module from CPAN, but the principle of separating the magic and the mundane is applicable everywhere. It’s called Separation of Concerns, or The Single Responsiblity Pattern. I call it a Just Story.

You’ll find the pattern in well designed websites that are using unobtrusive javascript to wave an AJAX wand over the site. You’ll see it woven through books like The Structure and Interpretation of Computer Programmers – where it’s called an Abstraction Barrier.

Patches sent

It turns out to be very easy to add mark_as_special_method (though I actually wrote it as ‘add_testinfo’ in the patch) to Test::Class. It’s about as straightforward an Extract Method refactoring as I’ve ever done – even without automated tools, I managed not to fuck it up. There’s a patch in Adrian Howard’s inbox, and I’m hopeful that it’ll be applied soon.

¹ Not exactly – that’s the result of calling the shadowed test subroutine which was the result of the code transformation.

Check out the osfameron fork of Devel::Declare for the beginnings of some decent documentation which explains what’s going on.

There’s a refactoring principle that states that, when you start doing the same thing for the third time, you should refactor to remove the duplication.

I’m starting to wonder if there’s a Smalltalk principle which states that, when you start doing the same thing the second time, you should search the image for the obviously named method (or use the method finder to find some candidate by feeding it some inputs and an expected answer), because the odds are good that it’s only the second time for you – it might be the millionth time for the image you’re working in.

Ruby’s primitives (Strings, Hashes, Arrays, Numbers – anything that has a literal syntax) are fine things. But that doesn’t mean you should use them everywhere. You’re often much better off wrapping them up in your own Value Objects.

Something I was working on at Railscamp this weekend threw up a great example of why it makes sense to replace primitives with more specific objects as soon as possible. Tom Morris asked me to take a look at Rena, an RDF/Semantic Web thingy.

The RDF spec describes two types of literals: a plain literal, which is a string with an optional language attribute and a typed literal, which is a string and an encoding (so the string might represent an integer, float, or anything else that your schema feels like expressing).

These literals can be output in one of (at least) two formats. We’ll start by looking at Literal#to_trix and see where that takes us:

def to_trix
  if @lang != nil && @lang != ""
    out = "<plainLiteral xml:lang=\"" + @lang + "\">"
  else
    out = "<plainLiteral>"
  end
  out += @contents
  out += "</plainLiteral>"
  return out
end

If we look over at TypedLiteral#to_trix we see a much more straightforward implementation:

def to_trix
  "<typedLiteral datatype=\"#{@encoding}\">#{@contents}</typedLiteral>"
end

How do we eliminate that ugly conditional at the beginning of Literal#to_trix, and analogous conditionals in Literal#to_n3 and TypedLiteral#to_n3?

My first thought was that I wanted to be able to write something like:

def to_trix
  "<plainLiteral#{@lang.to_trix}>#{@contents}</plainLiteral>"
end

But I didn’t want every string in the world suddenly acquiring a to_trix method. So, the solution was to intoduce a Literal::Language class and coerce our language into it, so Literal#initialize became:

def initialize(contents, lang = nil)
  @contents = contents
  @lang = Language.coerce(lang)
end  

And Language would look something like:

def self.coerce(lang)
  if lang.is_a?(self)
    return lang
  end

  new(lang.to_s.downcase)
end

def initialize(lang)
  @value = lang
end

def to_trix
  if @value == ''
    ''
  else
    " xml:lang=\"#{@value}\""
  end
end    

That ugly conditional’s still there though, so we introduced the Null Object pattern, and things started to look a good deal cleaner:

class Language
  class Null
    include Singleton
    
    def to_trix
      ''
    end
  end

  def self.coerce(lang)
    case lang
    when self
      return lang
    when nil, ''
      return Null.instance
    else
      return new(lang.to_s.downcase)
    end
  end

  ...

  def to_trix
    " xml:lang=\"#{@value}\""
  end
end

At this point, we’re still just pushing code around. If anything, we’ve got more lines of code now than when we started, but we’re starting to move behaviour nearer to the data it relates to, and our objects are starting to look like objects rather than data structures. So, we press on and make a TypedLiteral::Encoding class and, at this point things start to look interesting. TypedLiteral is starting to look almost exactly the same as Literal, but with an Encoding rather than a language.

That strange leading space in Language#to_trix is starting bug me. Let’s rewrite like so:

class Literal 
  class Language
    def format_as_trix(literal)
      "<plainLiteral xml:lang=\"#{@value}\">#{literal}</plainLiteral>"
    end

    class Null
      def format_as_trix(literal)
        "<plainLiteral>#{literal}</plainLiteral>"
      end
    end
  end

  def to_trix
    @lang.format_as_trix(@contents)
  end
end

If we make analogous change to TypedLiteral and TypedLiteral::Encoding it’s obvious that TypedLiteral and Literal were essentially the same class. Renaming @lang and @encoding to @language_or_encoding makes this blindingly obvious, so we’ll remove all of TypedLiteral’s methods except initialize. All that remains is to introduce Literal.untyped and Literal.typed factory methods to Literal, and make Literal.new into a private method and we can remove TypedLiteral in its entireity. So we change the specs to reflect the new API (wrong way round I know). Now we have a chunk of shorter, clearer code that will hopefully be easier to extend to cope with outputting literals in other formats.

Retrospective

I realise that patterns aren’t the goal of development, but by the end of the process we have a Strategy (Language/Encoding), a couple of Factory Methods (Literal.typed, Literal.untyped) and a couple of factoryish methods (Language.coerce, Encoding.coerce).

The most important aspect of the change was the introduction of the two new value object classes. Once they were introduced, they became the obvious places in which to put the varying behaviour and eliminate the repeatition of conditional code from the to_* methods. If there were to be a third output style, I would look at introducing classes like N3Stream, TrixStream and WhateverStream and have a scheme like:

def to_n3
  print_on( N3Stream.new )
end

def print_on(stream)
  language_or_encoding.print_on(stream, value)
end

but that’s almost certainly over complicating things right now.

The other thing I like about this kind of refactoring is that it drives the code towards methods and classes which obey the single responsibility principle and, at the end of the process, not only do we have fewer lines of code in total, but the individual methods involved are all substantially shorter and closer to the left hand margin.

I really should start doing this kind of thing more in my Rails practice – I keep being put off by the fact that the composed_of helper is so annoyingly not quite right and, rather than submitting a patch or making a plugin I go “Ah well… I can live with a string for a bit longer…” and I know_. From hard won experience at that, that it’s going to come and bite me. It’s already bitten Rails recently when Ruby got a new @String#tochars@ which doesn’t work like the ActiveSupport version.

Notes

If you want to see the gory details of how the change got made, Tom has merged this weekends changes into the github repository. It didn’t happen in quite the order I’ve described it in this post, but neither is this post a complete fabrication.

Changes

Corrected a stupid typo in the first block of code. Ugly condition is actually if @lang != nil && @lang != ''

Usability testing is wonderful. But wow, its humiliating.

I’ve spent the last few weeks working on the Amazingtunes in page player. Amazingtunes is a music site, so we need to play music. However, we don’t like the way that most music sites work; either the music stops as you go from one page to another, or the player is a huge Flash app running in its own window. There has to be a better way. There needs to be a popup window if you want to eliminate stop/start behaviour, but there’s surely no reason not to keep the controls on the main page.

So, we set about writing somthing that did just that. We settled on using Jeroen Wijering’s excellent flvPlayer, which handles the media formats we need and has good Javascript control and communications. This sits in the child window and we use Javascript cross-window communication to have a player controller in the main window that looks something like:

Uploaded with plasq’s Skitch!

This is all done in HTML and and Javascript, the progress bar does the Safari trick of running behind the tune data links, the buttons do their AJAX magic and the whole thing is rather slick, though I say so myself.

At least, we thought it was slick until we pointed the usertesting.com legions at it. Without exception, they ignore the in page player, foreground the popup and use the teeny weeny controls on the flash player. Originally, the popup window didn’t even display any transport controls, it just had a picture of some speakers and some text asking the user not to close it because it was playing the tunes. We added transport controls as a stopgap while we made the in page player work properly.

I sound like I’m whinging don’t I? It’s certainly a blow to the ego to see something we spent so much time and attention on being ignored by our sample users. On at least one occasion, while watching the screencasts I found myself boggling at the things the users did, and if I didn’t shout “Just play some bloody music!” at the screen, then I came worryingly close.

It would be easy to retreat into a state of denial: “They’re not our target users! They’re stupid! They’re American! If they would only magically intuit the way we think they should use the site!”. And maybe it would be comforting to do so, for a while. The right thing to do is to suck it up – take away from those videos the sure and certain knowledge that bits of the site don’t work and do something about it.

We may dislike the ‘popup window for transport controls’ model of controlling music playback, but users are cool with it. And it’s not as if the work we did on making the in page player work is going to be wasted – widget is straightforwardly event driven so it’ll work just as well in the popup window, and the communication protocol will be much simpler. Having the player in its own window means we’ll be able to extend its interface in ways that would be hard when the player had to share window space with the rest of the page. In the end, it’s all good.

But… damn that in page player was sweet. I learned Javascript as I wrote it (mostly by pretending it was Perl with odd syntas) and I’m bloody proud of it. I’ll happily replace it with the next iteration (which I’m already working on), but it’ll be with a pang of remorse all the same.

You know that voice at the back of your head that says things like “That’s not really a data structure, that’s an object that is!”?

You should listen to it.

I’ve been battling away with having a million and one things monkeying with a result set that consists of an array of arrays, when really it should have been a ResultSet which has a collection of ResultSet::Rows. Once I bit that bullet, I got to do things like add ActiveRecord::Errors to each row and generally start to bring a bunch of big guns to bear on a problem.

So, having listened to the voice at last I’m going through and removing code with gay abandon. Sometimes I think removing code is one of the great joys of being a programmer – it’s the code’s way of telling you you got something right.

So, today’s slogan is “Reify early and reify often”.

My dad drives a vintage Fraser Nash. I say drives, but that’s only half the battle, a large part of his Nash time is spent fettling it. It’s an old car; bits wear out, break or drop off. And because it’s an old car, you can’t just nip round to Halfords and pic up a replacement; nor can you head down to the breaker’s yard and cannibalize something else. So he has a lathe and a milling machine and a bewildering collection of tools. When he needs a part, he will disappear into the machine shop and, after sufficient swearing and/or bleeding, he will emerge with a newly made part. For dad, it’s all part of the fun of running a vintage car. If he weren’t able to do the work, the Nash would have had to remain a pleasant pipedream.

I don’t know my way around a machine shop, except in the vaguest and most theoretical way. The tools I’ve grown up knowing to use are programming languages, editors, fine manuals and the mental tools a grounding in mathematics brings.

So, when I’m putting a new photography business together, and I realise that a couple of the supporting software tools that I had vaguely assumed ‘should exist’ don’t actually exist, I know that it doesn’t matter. I may not know Cocoa programming yet, but I know programming, so I’m confident that, like dad in his machine shop, I’ll be able to knock something up that does the job.

On reflection, I realised that this is probably a good thing. If I can set up and run the business with a combination of off the shelf software, then it’s trivial for potential competitors to reverse engineer the business and do the same (let’s assume here that the business is a success) and I’m left competing on margin in a service industry. No fun at all.

Being able to make my own tools gives me a competitive edge.

Why aren’t there more tool makers?