Just A Summary

Piers Cawley Practices Punditry

Pretty Saro

Pretty Saro by pdcawley

I’m a wee bit late writing this because it’s mostly about my summer trip to the Swannanoa Gathering at Warren Wilson College near Asheville, North Carolina. It just so happened that YAPC (Yet Another Perl Conference) was held in Asheville this year, the week before the gathering’s Traditional Song Week. Well, I’m a perl hacker. I sing traditional songs. My employer was paying for me to attend YAPC and were willing for me to extend my stay in America by a week. It was a no brainer really.

YAPC was bloody good this year. Perl 5 development is moving forward and the community is buzzing because of it. Lots of “… and we’re hiring” slides. And that’s before we get to the pleasure of catching up with friends that I only see online most of the time. If you’re working in any field that has grassroots conferences associated with it, I can’t recommend attending them highly enough.

On to Swannanoa

At Swannanoa, I met Sheila Kay Adams and immediately switched my schedule to spend as much time singing with her as possible. Sheila’s a seventh generation ballad singer from the Sodom Laurel community in Madison Count. Her “grannie Berzil” Wallin remembered Cecil Sharp coming to Madison County and collecting songs from the family.

Pedigree in singing shouldn’t matter, but it turns out it does. Sheila grew up in a community which was changing, but which still had old ‘love songs’ as an important part of how it understood the world. Today, not so much. People still relate to the world and understand it through songs, but the songs are more likely to be contemporary. Lyrically, many of the love songs that Sheila and her family sing could have been written yesterday, but their performance is radically different from contemporary style. One voice, unaccompanied, a style that requires the listener to concentrate on the song rather than any aspects of production. Not something you’re going to dance to at your wedding, say.

Sheila’s classes, on Meeting House Songs (more later) and her Ballads class with Bobbie McMillon were just wonderful; I won’t forget in a hurry the sound Bobbie singing “A conversation with Death” as a thunderstorm grumbled across the campus in the background. Spine tingling stuff. Sheila’s description of how she learned songs “knee to knee” has been helpful too. The way it would work was that the teacher and student would be sat out on the porch, often doing some chore or another, and the teacher would sing the first verse of a song. The student would sing it back and the teacher would sing the second verse. The student would then sing the first two verses then the teacher would sing the third verse and the student would sing the first three verses and so on, until the student was singing the whole song back to the teacher. A time consuming process to be sure, but it works.

I know this because I learned Pretty Saro from a recording of Sheila’s late husband Jim Taylor, using a variant of the method, “knee to CD” if you like. I’d play the first verse, hit pause and sing it back, play the second verse, pause, repeat the first two… and so on. And in very short order I had the words and tune (up to a point; I listened back to that recording again recently and I’m singing a different tune now) and could start the process of actually learning the song, which involved singing it it lots, listening to other recordings, singing it some more and then taking the song out and trying it out in front of audiences and listening to what works and what doesn’t. I’ve never really finished learning a song; this recording is a snapshot. I hope you enjoy it.

References in Child of the Library

I sometimes think that I should have published the lyrics to Child of the Library with a bibliography. The references in the second verse are all obvious to me, but I’m a white middle class English boy who grew up around boats. My childhood reading and yours may not intersect all that much.


The Walkers and the Blacketts

Also known as the Swallows and Amazons. Swallows and Amazons is the first ‘big’ book that I can remember reading for myself. We were in Cornwall, holidaying at the same place my mum’s been going to since she was a kid. Mum was reading Swallows and Amazons to us, and it was great, but I was impatient to find out what happened next, so I took the book to bed with me and read it for myself. I haven’t stopped yet. Swallows and Amazons was the book that opened my door to reading for pleasure. It opened up a way of looking at the world too. I can’t imagine who I’d be if I’d never read any Ransome.

The Pevensies

The Pevensies are the family in The Lion, The Witch And The Wardrobe, the first of the Narnia books. I have a slightly troubled relationship with these books now. I really don’t get on with C. S. Lewis’s view of the world (I once hurled a taped reading of The Screwtape Letters out of the car rather than listen to another word of the bloody things), but I wasn’t reading between the lines when I was nine. I was just loving the stories and the images they put in my head. And what images…

Simp, the Canine Cannonball

What do you mean, you’ve never read Cannonball Simp? You poor thing!

Cannonball Simp, by John Burningham is the library book for me. Back when I was 4 or 5, I would walk with my dad from Regent’s Square to the Doncaster Central Library every Saturday morning, with three books clutched under my arm. I’d hand them over the counter, receiving in exchange 3 buff cardboard library slip holders. I would then go over to the childrens’ books section and pull out three new (to me) books and take them to the counter. The librarian would remove the slip from the library bookplate, place it in one of my surrendered holders and stamp the slip and the bookplate with the date, three weeks hence, by which the book must be renewed.

Well, that’s how it worked until weekend I came back with Cannonball Simp. I loved it. I didn’t want to take it back because I hadn’t learned it yet. Sure I could ‘read’ along with Dad – I knew all the words by then – but the pictures were another thing entirely. They were beautiful. They still are.

That’s when I learned of the magic of ‘renewal’. Instead of handing the book back, I showed it to the librarian and said “I’d like to renew this, please,” and instead of giving me my library card back, they just stamped the book and slip with a new date. Wow! It was like I owned the book.

I don’t know how many times I renewed that book. I’m afraid I don’t remember the words any more. But I still remember the pleasure that it gave me.

Galadriel the Fair

On the last day of junior school our form teacher, Miss Rees, wrote a long list of books on the board and asked us to copy the list into the back of our exercise books. She said that these were books we should try and read. I don’t have the exercise book any more and I remember very little of the list. I do remember that I’d already read some of them. And I remember The Hobbit and The Lord of the Rings. Oh boy, do I remember The Hobbit and The Lord of the Rings.

Terry Pratchett said that if you’re a 14 year old boy and you don’t think that The Lord of the Rings is the best book ever written then there may be something wrong with you, but if you’re 28 and you still think it’s the best book ever written then there’s definitely something wrong with you. I was that 14 year old boy. I stayed up all night reading it. I fell asleep reading it. I read the appendices. I worked out how to write my name in Elvish script. I even read The Silmarillion and thought I enjoyed it. I can still recite some of the poetry. In Elvish. I’ve forgotten the words of Cannonball Simp, but get me going with “A Elbereth Gilthoniel…” and I can just about reel off the rest of the poem. What’s wrong with me?

By the time I was 28, LotR wasn’t the best book ever written. I’m no longer sure that there is such a thing as “The Best Book Ever Written”. Still, if you haven’t at least read The Hobbit you should take steps to rectify matters. Or wait for the Peter Jackson movie.

The Daughter of a Pirate King

Confession time: I haven’t read Pippi Longstocking. The ‘I’ of A Child of the Library is a composite of Gill and me. Pippi Longstocking was her Nancy Blackett. The Bastables were her Pevensies.

Paddington the Bear

Once I’d learned that reading for myself was pure pleasure, I read anything and everything I could find in the library and I discovered Michael Bond’s Paddington Bear. Paddington was a well meaning young bear from Peru who was found, wearing a duffel coat and a label reading “Please look after this bear”, at Paddington Station by the Brown Family. The Browns took him in immediately, named him Paddington after the station and… ‘hijinks ensue’.

I would be quietly reading these books to myself when some episode or another (“Baked Elastic”; the Russian ballet dancer; the wobbly table…) would cause me to laugh out loud and my brother would demand to know what was funny and made me read whatever it was aloud to him. I have to confess, I resented this, but not enough to dent my enjoyment. The Paddington books are made of joy.

Published on Thu, 01 Sep 2011 09:44:00 GMT by Piers Cawley under . Tags , , ,

The Further Adventures of Child of the LIbrary

Or… what I did this summer.

Summer as been frantic. Mostly joyous, but frantic.

I had talks accepted at both YAPC and OSCON. Because YAPC was in Asheville, and the Swannanoa Gathering Traditional Song Week fell the week after YAPC, that meant I flew out to Asheville for an intense fortnight of Perl community engagement followed by a week spent singing myself hoarse and being blown away by Sheila Kay Adams’s singing and her stories of mountain life and listening to future stars like Sam Gleaves and inspiring activists like Saro Lynch Thomason. I could write entire posts on every one of those, and that’s before I get on to the magic of watching the sun go down and the fireflies come up from the grass of the Warren Wilson College’s natural amphitheatre. Magical so it was.

On the last night of the gathering, there was a student showcase. Of course, I sang Child of the Library. Rather embarrassingly, I skipped a verse, but the response was great. Several people came up to me afterwards and told me their library stories and, I hope, went home with a determination to help protect their libraries.

Then it was back to the UK for a fortnight before heading off again to OSCON in Portland, OR. I was down to give two talks, one on Higher Order Javascript and a second on Polymorphic Dispatch which proved to be a bear to write. I’d planned to have everything done and dusted and the slides learned by the time I flew out, but spent a huge amount of time blocked on how I was going to present the ideas using a motivating problem that hit the sweet spot between too simple and too complicated. It wasn’t until I arrived in Portland and talked things through with my host, David Wheeler, that I got that worked out. The slides were written in a tearing rush and weren’t as good as I would like them to be. I still owe the talk attendees a full writeup, so I’ll take this opportunity to apologise for the lateness and promise that a writeup will arrive. The Javascript talk was based on one I’ve given before, but I’m starting to realise that it’s a full tutorial masquerading as a short talk. More on that later, I hope.

What with flapping about the unwritten talk, and the 10 minute gap between them, I didn’t really worry overmuch about the Perl lightning talks. It’s been my practice to sing a lightly massaged version of Lou and Peter Berryman’s very splendid song A Chat With Your Mother with tailored verses about different language communities and various Perl luminaries. I had decided to retire it, but at YAPC I came up with a snide verse about myself and another about Larry Wall, so it felt renewed enough to be worth singing this year as well. I ended up singing Child of the Library again.

The response was phenomenal. I’ve been singing it for long enough now that I know it gets people. Hell, when I was writing it, there were verses that were hard to sing because they got me and I was choking up as I tried to sing them. But the lightning talk got a standing ovation. Again people were telling me library stories and I found that the UK isn’t the only country with local governments stupid enough to consider closing libraries, it was happening in the US too. So, fired up by that response, I went to see Sarah Novotny and begged her for five minutes on the OSCON stage before the closing keynote. Bless her, she let me have it.

Pics, or it didn’t happen!

I have proof too! The main stage at O’Reilly events has serious video equipment pointed at it. Because it’s important that people get to see serious talks about known bugs and exploits in wetware. Because my performance wasn’t on the schedule, and I didn’t have a video release all signed and ready to go beforehand, it’s taken a while to get the video available. But last night, that changed, so here I am, in all my corpulent glory. Enjoy. And please, spread this video as far and as wide as you can. Libraries are important.

Published on Wed, 31 Aug 2011 13:51:00 GMT by Piers Cawley under . Tags , , , , ,

A Child of the Library, revised

Here’s a revised version of A Child of the Library.

Child Of The Library (Revised) by pdcawley

Whenever I learn a new song, there’s a period of making it “mine”. This happens as I sing it out to different audiences and find out which bits work, which bits are hard to sing and all the other little details that you only find out when there’s a living, breathing, listening and (hopefully) singing audience in front of you.

It turns out that the same things happens with songs I’ve written.

The biggest stumbling block of the original version is the last line of the first verse:

My wife met Pippi Langstumpf, I met Paddington the Bear

Someone on the comment thread suggested:

Heidi, Pippi Longstocking and Paddington the Bear

Which is fine, except that means singing ‘PIPpi LONGstockING’, and I’m not happy with putting the emphasis on the WRONG sylLABles if I can possibly help it. In the revised version, I think I’ve cracked it. The line is now:

The daughter of a pirate king and Paddington the bear

Which is much easier to sing and follows the rest of the verse by being a more oblique reference to the character. If you know who Pippi is, then it’s obvious who I’m singing about. If you don’t, then maybe I’ve piqued your curiosity.

The first recording also had two slightly different tunes for the verses, this version has settled on just the one.

And, for people who care about that sort of thing, this was recorded with a click track at 110bpm rather than speeding up over the course of the song, which should make life a little easier for anyone taking part in Rabid Gravy’s project to remix and rerecord different versions of the song.

If you’re concerned about Library closures, a good place to start is Voices for the Library. If you’re on twitter, you might also start following @ukpling. If you’re not concerned about library closures, why on earth did you read this far, and what kind of excuse for a human being are you?

Published on Mon, 04 Apr 2011 05:45:00 GMT by Piers Cawley under . Tags , , , , ,

Lands' End

Lands’ End Mail Order Catalogue by pdcawley

This is by Chris Manners, a Yorkshireman who I met when we were both exiled in Essex. The commute into London from Essex was made so much more bearable if we managed to share it with Chris and Tim Blyth. The day Chris moved back home was a day of mixed feelings, we were sorry to lose his company every but delighted for him too. As well as being good company, Chris is an accomplished singer, guitarist and, as this song proves, songwriter. I’m very pleased that he’s given me permission to record this.


Published on Mon, 21 Feb 2011 01:44:00 GMT by Piers Cawley under . Tags , , ,

Save Our Libraries

So, on Saturday, the opening line, and pretty much the entire tune, of a song banged on my head as we went to our local Library to fill our boots with books and generally get with the “Save our Libraries” message. Here it is. Sing it out. Sing it loud.

Child of the Library by pdcawley

A Child of the Library

I’m a Child of the Lib’ry, it made me who I am,
It taught me about freedom and the fellowship of Man
A sea of story waits for you behind the lib’ry door,
Don’t say we can’t afford them any more.

The Lib’ry’s where I made some friends I’ve known my whole life through
The Walkers and the Blacketts and the Pevensies so true.
Simp the canine cannonball, Galadriel the fair.
The daughter of a pirate king and Paddington the Bear

I’ve travelled South with Shackleton and all his gallant crew
And to the African interior that Mary Kingsley knew
I’ve rode the trackless prairie where the bison used to roam
An travelled round the Universe, not half an hour from home.

And as I grew the libr’y fed my curiosity,
All there for the asking. All of it for free.
It’s there I found the stories that I couldn’t find at home.
It’s where I learned I was myself and not my father’s clone.

So make friends with your library, don’t let it fade away.
Teach your kids the lib’ry’s where you go on Saturday.
Don’t let the bastards tell you they will cost to much to save
While they’re shovelling our taxes down the hole the bankers made

So make a stand for the lib’ry. Stand up while you can.
Stand up for your freedom. Stand for your fellow man.
Ignorance is never bliss, don’t close the lib’ry door.
For a lib’ry lost is lost forever more.

Lyrics © 2011 Piers and Gill Cawley
Music © 2011 Piers Cawley

Licenced under a Creative Commons BY-NC-SA License.

Anyone who sings this is a friend of ours.

Notes and Updates

Mike Whitaker has worked out a set of chords for this, if you’re happier singing with a guitar. Frankly, instrumentation scares me, I’m much happier singing unaccompanied but I realise I’m in a minority on this.

When you sing it, try making the song your own by changing things to suit your own. Julie Chilton’s posted an alternate first verse in the comments, but the world is the mollusc of your choice.

The ‘I’ in the song, isn’t quite me and it isn’t quite Gill either. Most importantly, both of us grew up in houses where no book was out of bounds. Our fathers used to take us to our respective local libraries (Lewisham for Gill, Doncaster Central and then Bawtry for me) every week or so on a Saturday morning. As soon as I was old enough, I’d ride my bike from our home in Scrooby to Bawtry with a carrier full of books hanging off the handlebars. Bawtry is currently in the crosshairs of Doncaster’s idiot mayor – the bloke who suggested getting kindles for everyone instead – along with 13 other libraries in the district.

There’s a revised version of the recording with additional notes at: http://www.bofh.org.uk/2011/04/04/a-child-of-the-library-revised, or just press play:

Child Of The Library (Revised) by pdcawley

August 9th 2011

I’ve updated the lyrics here to reflect what I sing now. A few of the clumsier bits have been fixed. Hopefully there will be video from my performance at OSCON 2011, which has the ‘right’ lyrics.

Published on Wed, 09 Feb 2011 02:50:00 GMT by Piers Cawley under . Tags , , , , ,

The OSCON Proposal I really, really want to be accepted

Open Ears, Open Mind, Open Mouth. Music Making Made Easy


Our bodies are the most versatile and sophisticated musical instrument we know. From the complexities of making at beat with our hands and feet to the surprising simplicity of harmony singing, we are all of us musicians.


Musicmaking isn’t some kind of sophisticated profession that requires the intervention of gatekeepers and techno priests. You don’t need autotune, you don’t need a record label, you don’t need drums, a guitar or anything else but your hands, feet, ears, brain and mouth to make music that will satisfy you for the rest of your life.

By the end of this talk I promise that, unless you are one of a tiny, tiny minority of people, you’ll not be tone deaf, you’ll be damn near pitch perfect. And you’ll have a song in your head that, unless I have seriously misjudged the people who come to OSCON, you’ll want to teach to everyone you know.

Come along. Clap your hands. Stamp your feet and sing. What have you got to lose?

Published on Mon, 07 Feb 2011 17:28:00 GMT by Piers Cawley under , , . Tags , ,

Repertoire Project: Lottery Song

Lottery Song by pdcawley

This is the only song I’ve written. I don’t really think of myself as a songwriter at all, but Gill was doing a songwriting module on the Folk degree at Newcastle University and was writing a suite of songs about the lottery. She was stuck trying to write something about how lottery winners think the money won’t change them but it does. As I was driving her into town for her lecture, I suggested that maybe what she should do was change the viewpoint and try and write something from the point of view of a friend of the big winner.

I dropped her off at the Armstrong Building and, as I pulled out of the university car park, this song fell on me. It arrived pretty much fully formed – I had a tune and all the verses by the time I got back to our house in Bensham, so I wrote it down fast as I could, tweaked a few bits and pieces, kicked out a verse that didn’t belong and recorded the tune into garageband with my laptop’s internal mic just so I didn’t forget it.

“I’ve written that song!” I said as I picked Gill up.

She nearly killed me.

Published on Mon, 31 Jan 2011 16:00:00 GMT by Piers Cawley under . Tags , , , , ,

A Handy Builder Pattern

I'm working on a web service, and that means that I need to build lots and lots of mildly different looking HTTP requests with various combinations of headers and requested URLs. The camel's back got broken this morning when I realised I didn't want to be writing a method called ssl_request_from_uk_with_bad_cert, which builds me an HTTP::Request with a particular combination of headers, that I can use with Plack::Test to test our webservice. The method name describes what's wanted, but the code is sopping wet and in desperate need of DRYing up.

My first cut was to write a simple request builder:

package RequestBuilder;
use Moose;

with 'MooseX::OneArgNew' => {
  type => 'HTTP::Request',
  init_arg => '_req',

has _req => (
  is => 'ro',
  isa => 'HTTP::Request',
  handles => [qw(header headers)],

sub ssl_request {
  my $class = shift;
    GET => 'https://localhost/',
      SSLSessionID => 'deadbeef',

sub from_uk {
  my $self = shift;
  $self->header('X-IP-Is-UK-Combined' => 'yes');
  return $self;

sub with_bad_cert {
  my $self = shift;
  $self->header('SSLClientCertStatus' => 'NoClientCert');
  return $self;

sub final {
  my $self = shift;

Now I have a builder, I can compose meaningful fragments to build a final request. So I might write:

my $r = RequestBuilder->ssl_request

But things got sticky when I wanted to to take a valid request and bend it out of shape to ensure that it failed correctly. I was writing things like:

use RequestBuilder;

sub valid_request {
  my($self, $is_final) = @_;
  my $r = RequestBuilder->ssl_request->from_uk;
  return $is_final ? $r->final : $r;

test "we reject requests from outside the UK" => sub {
  my $self = shift;
  my $r = $self->valid_request(0)->from('Turkmenistan')->final;


which works, but is uglier than a very ugly thing indeed. What I wanted was some way of having the finalization magically happen at the point of use but have some way of extending any intermediate results. The interface I came up with looks like this:

use RequestBuilder;

sub valid_request {
  build_request {

test "we reject requests from outside the UK" => sub {
  my $self = shift;
  my $r = build_request {


When the builder is returned from the outermost build_request block, it gets finalized.

"How does that work then?" I hear you ask. It's quite simple, once you know about Moose::Exporter, old fashioned perl prototypes and the Tao of dynamic scope. We just add something like the following to the end of our RequestBuilder:

our $building;

use Moose::Exporter;
  as_is => ['build_request'],

sub build_request (&) {
  my $block = shift;
  my $builder = do {
    local $building = 1;
    local $_ = __PACKAGE__;
  return $building ? $builder : $builder->_req;

So, build_request has a & prototype, which means it takes a block as its first argument. Perl treats an initial prototyped block argument as slightly magical and doesn't require the use of the sub keyword (though you can use it if you want). It then uses a do block to dynamically set $building to 1 and $_ to be the current __PACKAGE__ name (because $_ is shorter to type than RequestBuilder) before calling the block to get a builder. Then it checks whether we're still building. If we are, it returns the builder. If we aren't, then the block was the outermost block, so build_request returns the built request.

I'm not quite ready to extract this pattern into a parameterized role - I need to make sure it's robust enough, but it's certainly something to think about when you next need to make a builder for your tests.

Published on Fri, 31 Dec 2010 08:17:00 GMT by Piers Cawley under , . Tags , , , ,

Asynchronous Streams

In Higher Order Javascript, I introduced Streams and showed how to use them to implement a lazy sort. I think that's neat all by itself, but it's not directly useful in the asynchronous, event driven execution environment that is the average web page. We'd like a structure where we spend less time twiddling our thumbs as we wait for force to return something to us.

Non blocking streams

What if we change the protocol of our stream to something more asynchronous? Obviously, we'd still have a head element which is immediately available and some kind of promise to compute the next stream. But rather than promising to return a new stream, the promise of a non-blocking stream is a promise to call a function we supply with the value it computes. Here's a CoffeeScript implementation of what we're talking about:

the_empty_stream =
    is_empty: true

continue_with = (b) -> window.setTimeout((=> b()), 0)

class NonBlockingStream
  constructor: (@head, promise) -> 
    this.force_into = (block) ->
      if promise.length == 0
        continue_with -> block promise()
        continue_with -> promise block
  is_empty: false

I've written NonBlockingStream to work with both 0 argument functions (like the promise of an ordinary stream) and with 1 argument promises that will be responsible for calling their block when appropriate. force_into doesn't block by virtue of the fact that we execute our promise using setTimeout - a timeout of 0 milliseconds doesn't mean execute the block immediately but asks for it to be executed as soon as possible. The fat arrow (=>) used to build the function passed to setTimeout ensures that the code will be executed with this bound to the stream instead of the global window object.

This new non blocking stream is a much better citizen on a webpage. It yields to the event loop at every opportunity and gets out the way of other, possibly more important events. But so far we know how to use it for things like finding primes or the top 5 entries of 1000. Not exactly useful in the browser…

Streams in the real world

Listen to any web usability guru for more than about ten minutes and they'll tell you that pagination is evil. If you have a resource (say a blog index page) that logically should have 1000 entries on it, then breaking it up into multiple pages is a sin. The reader should simply be able to scroll through all 1000 entries. But most users don't scroll through every entry, and rendering 1000 entries is time consuming. Sites like Google reader and Twitter solve this by doing 'just in time' fetching.

Suppose we wanted to re-engineer this blog to use the endless page pattern, then I might think of using streams. We'd like to serve up an index page that sets up all the headers and navigational stuff, but which populates its articles via an unbounded stream of articles. Let's say that an article looks something like this:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
    <title>The current article</title>
    <link rel="next" href="the-next-article.xhtml" />
      <h2>The current article</h2>
      <p>Yada yada yada...</p>

You can think of this as a kind of stream. The 'head' of the stream is the contents of the html body tag, and the promise is the <link rel="next" ... /> tag in the head. Let's write a function which, given a document like this will make us a stream. We'll use jQuery because, well, why not?

$ = jQuery

doc2stream = (data) ->
  new NonBlockingStream $("body", data), (block) ->
    next = $("head link[rel=next]", data).attr('href')
    if next
        url: next
        success: (data) -> block(doc2stream data)
        dataType: 'xml'
      block the_empty_stream

Note that our promise doesn't call the block it's forced with immediately. Instead it fires off an asynchronous request for the next article in the stream with a success callback that (finally) calls the block.

Putting it together

The trick now is to get the stream up and running from our article index. Let's assume we have an initial page along the lines of:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
    <title>Some blog articles</title>
    <link rel="start" href="the-first-article" />
      <h1>Some blog articles</h1>
    <section id="articles">
        <p>If you can see this, I'm probably fetching a real article</p>


This is where we shove all the navigational bits and pieces of our blog, the links to atom feeds, sidebars with associate links and whatever. But it's devoid of content. We need to fix that by finishing off our asynch-fetcher.coffee1. First, we need to setup our stream:

articles = new NonBlockingStream '', (block) -> 
    url: $("head link[rel=first]").attr('href')
    success: (data) -> block doc2stream data
    dataType: 'xml'

We can't simply call doc2stream because that expects to find the link to the next article in a [rel=next] link in the head, but we're using a 'first' link here. So we make a stream with the empty string a as a dummy head and a promise to fetch the article linked to by our [rel=first] and turn that into a stream via doc2stream

Next we need a function to update the articles variable and extract the article element from the head of a stream and insert it in our #articles section just before the footer. Once this is defined we force the first article into it.

show_next = (stream) ->
  articles = stream
  $("#articles footer").before(
    $("article", stream.head).clone().addClass("last-art")

articles.force_into show_next

We also arrange for show_next to add a last-art class to the newly inserted article, which we'll use as a target in the watcher function we set up to handle fetching new articles as they are needed:

$.fn.is_in_view = () ->
  $(this).position().top < ($("body").scrollTop() + $(window).height())

watcher = ->
  last_art = $(".last-art")
  if last_art.is_in_view()
    articles.force_into (str) -> 
      unless str.is_empty
        show_next str
        window.setTimeout watcher, 100
    window.setTimeout watcher, 100

window.setTimeout watcher, 100

Our heuristic for judging when to fetch the next article is simple: if the article tagged with the last-art class is in the viewport, then it's time to go about fetching the next one. This assumes that our writing is compelling enough that by the time the reader gets to the bottom of an article, the next one will have been succesfully fetched. This may be an optimistic heuristic, but we're all about "for the purposes of illustration" here.

To get this to work, we add a simple is_in_view method to jQuery.fn. This tests if the selected element's top is placed higher than the bottom of the viewport. With that in place, we can write watcher which checks if the last article is in view. If it when it is, watcher removes the last-art marker class and kicks off the process of fetching the next article. We use window.setTimeout to ensure that we keep fetching next articles as long as their are articles to fetch and our reader is reading them.2


  • A similar stream could be set up for each article's comments, after which we might find that we should parameterize show_next and watcher in some fashion.
  • Real world use would probably involve serving up a few articles in the body of a blog front page - if only so that Google has something to index. tag indices or search results could be served up empty and populated on demand.
  • I've not (yet) redone this blog to use this pattern, but I've tested the code presented here and it does work.
  • For caching purposes, it may better for searchs and the like to return a small lump of JSON with a head link to the statically cached article document and a promise link to the next lump of JSON in the results. If we write doc2stream right, it should be possible to completely isolate the rest of the page from this decision, which seems like a win to me.


1 Which our server autogenerates from the coffeescript we're actually writing.

2 Or, at least, scrolling past them.

Published on Wed, 24 Nov 2010 04:46:00 GMT by Piers Cawley under . Tags , , , ,

Powered by Publify – Thème Frédéric de Villamil | Photo Glenn