Just A Summary

Piers Cawley Practices Punditry

Asynchronous Streams

Posted by Piers Cawley Wed, 24 Nov 2010 04:46:00 GMT

In Higher Order Javascript, I introduced Streams and showed how to use them to implement a lazy sort. I think that's neat all by itself, but it's not directly useful in the asynchronous, event driven execution environment that is the average web page. We'd like a structure where we spend less time twiddling our thumbs as we wait for force to return something to us.

Non blocking streams

What if we change the protocol of our stream to something more asynchronous? Obviously, we'd still have a head element which is immediately available and some kind of promise to compute the next stream. But rather than promising to return a new stream, the promise of a non-blocking stream is a promise to call a function we supply with the value it computes. Here's a CoffeeScript implementation of what we're talking about: 

the_empty_stream =
    is_empty: true

continue_with = (b) -> window.setTimeout((=> b()), 0)

class NonBlockingStream
  constructor: (@head, promise) -> 
    this.force_into = (block) ->
      if promise.length == 0
        continue_with -> block promise()
      else
        continue_with -> promise block
  is_empty: false

I've written NonBlockingStream to work with both 0 argument functions (like the promise of an ordinary stream) and with 1 argument promises that will be responsible for calling their block when appropriate. force_into doesn't block by virtue of the fact that we execute our promise using setTimeout - a timeout of 0 milliseconds doesn't mean execute the block immediately but asks for it to be executed as soon as possible. The fat arrow (=>) used to build the function passed to setTimeout ensures that the code will be executed with this bound to the stream instead of the global window object.

This new non blocking stream is a much better citizen on a webpage. It yields to the event loop at every opportunity and gets out the way of other, possibly more important events. But so far we know how to use it for things like finding primes or the top 5 entries of 1000. Not exactly useful in the browser…

Streams in the real world

Listen to any web usability guru for more than about ten minutes and they'll tell you that pagination is evil. If you have a resource (say a blog index page) that logically should have 1000 entries on it, then breaking it up into multiple pages is a sin. The reader should simply be able to scroll through all 1000 entries. But most users don't scroll through every entry, and rendering 1000 entries is time consuming. Sites like Google reader and Twitter solve this by doing 'just in time' fetching.

Suppose we wanted to re-engineer this blog to use the endless page pattern, then I might think of using streams. We'd like to serve up an index page that sets up all the headers and navigational stuff, but which populates its articles via an unbounded stream of articles. Let's say that an article looks something like this: 

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>The current article</title>
    <link rel="next" href="the-next-article.xhtml" />
  </head>
  <body>
    <article>
      <h2>The current article</h2>
      <p>Yada yada yada...</p>
    </article>
  </body>
</html>

You can think of this as a kind of stream. The 'head' of the stream is the contents of the html body tag, and the promise is the <link rel="next" ... /> tag in the head. Let's write a function which, given a document like this will make us a stream. We'll use jQuery because, well, why not? 

$ = jQuery

doc2stream = (data) ->
  new NonBlockingStream $("body", data), (block) ->
    next = $("head link[rel=next]", data).attr('href')
    if next
      $.ajax
        url: next
        success: (data) -> block(doc2stream data)
        dataType: 'xml'
    else
      block the_empty_stream

Note that our promise doesn't call the block it's forced with immediately. Instead it fires off an asynchronous request for the next article in the stream with a success callback that (finally) calls the block.

Putting it together

The trick now is to get the stream up and running from our article index. Let's assume we have an initial page along the lines of: 

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Some blog articles</title>
    <link rel="start" href="the-first-article" />
    <script 
       type="text/javascript"  
       src="http://code.jquery.com/jquery.min.js"></script>
  </head>
  <body>
    <header>
      <h1>Some blog articles</h1>
    </header>
    <section id="articles">
      <footer>
        <p>If you can see this, I'm probably fetching a real article</p>
      </footer>
    </section>
    <script
       type="text/javascript"
       src=".../asynch-fetcher.js"></script>      
    <footer>
      ...
    </footer>

  </body>
</html>

This is where we shove all the navigational bits and pieces of our blog, the links to atom feeds, sidebars with associate links and whatever. But it's devoid of content. We need to fix that by finishing off our asynch-fetcher.coffee1. First, we need to setup our stream: 

articles = new NonBlockingStream '', (block) -> 
  $.ajax
    url: $("head link[rel=first]").attr('href')
    success: (data) -> block doc2stream data
    dataType: 'xml'

We can't simply call doc2stream because that expects to find the link to the next article in a [rel=next] link in the head, but we're using a 'first' link here. So we make a stream with the empty string a as a dummy head and a promise to fetch the article linked to by our [rel=first] and turn that into a stream via doc2stream

Next we need a function to update the articles variable and extract the article element from the head of a stream and insert it in our #articles section just before the footer. Once this is defined we force the first article into it. 

show_next = (stream) ->
  articles = stream
  $("#articles footer").before(
    $("article", stream.head).clone().addClass("last-art")
  )

articles.force_into show_next

We also arrange for show_next to add a last-art class to the newly inserted article, which we'll use as a target in the watcher function we set up to handle fetching new articles as they are needed: 

$.fn.is_in_view = () ->
  $(this).position().top < ($("body").scrollTop() + $(window).height())

watcher = ->
  last_art = $(".last-art")
  if last_art.is_in_view()
    last_art.removeClass('last-art');
    articles.force_into (str) -> 
      unless str.is_empty
        show_next str
        window.setTimeout watcher, 100
  else
    window.setTimeout watcher, 100

window.setTimeout watcher, 100

Our heuristic for judging when to fetch the next article is simple: if the article tagged with the last-art class is in the viewport, then it's time to go about fetching the next one. This assumes that our writing is compelling enough that by the time the reader gets to the bottom of an article, the next one will have been succesfully fetched. This may be an optimistic heuristic, but we're all about "for the purposes of illustration" here.

To get this to work, we add a simple is_in_view method to jQuery.fn. This tests if the selected element's top is placed higher than the bottom of the viewport. With that in place, we can write watcher which checks if the last article is in view. If it when it is, watcher removes the last-art marker class and kicks off the process of fetching the next article. We use window.setTimeout to ensure that we keep fetching next articles as long as their are articles to fetch and our reader is reading them.2

Notes

  • A similar stream could be set up for each article's comments, after which we might find that we should parameterize show_next and watcher in some fashion.
  • Real world use would probably involve serving up a few articles in the body of a blog front page - if only so that Google has something to index. tag indices or search results could be served up empty and populated on demand.
  • I've not (yet) redone this blog to use this pattern, but I've tested the code presented here and it does work.
  • For caching purposes, it may better for searchs and the like to return a small lump of JSON with a head link to the statically cached article document and a promise link to the next lump of JSON in the results. If we write doc2stream right, it should be possible to completely isolate the rest of the page from this decision, which seems like a win to me.

Footnotes:

1 Which our server autogenerates from the coffeescript we're actually writing.

2 Or, at least, scrolling past them.

Higher Order Javascript 9

Posted by Piers Cawley Thu, 18 Nov 2010 14:52:00 GMT

To my surprise, several people have asked for the slides from my Øredev talk on Higher Order Javascript, and I’ve followed my usual practice of saying “Sorry, no”. Slide decks are a terrible teaching medium – they’re fine if they come with the presenter, but if they contain enough information to read as if they were a book, then I’m prepared to bet that they made a terrible presentation. Good presentations have a synergy; slides illustrate what the speaker is saying and neither the speech nor the slides should really stand alone. After all, if either could, why bother with the other?

But, the requests mean that some people found the information useful, so here’s the content (or something like it) written up as a blog post instead.

Higher order what?

Yes, yes. I know. I’m a Perl programmer. What am I doing writing about higher ordered javascript? In part it’s because Higher Order Javascript was the title of the talk that Giles Bowkett was due to give before he had to bale out and, while I’m sure I could give a Higher Order Perl talk, I doubt that it would have gone down particularly well in the webprogramming track of a conference that was heavily slanted towards Java, .NET and Agile. Nowadays Perl gets even less respect than Javascript and I simply wasn’t prepared to go kicking that particular stone uphill with a bunch of programmers who made their minds up on that particular score ages ago.

Anyway, higher ordered programming is language agnostic. So long as your language has first class functions (and ideally lexical closures) then you’re good to go. I first learned about higher order functions from Mark Jason Dominus’s Higher Order Perl talk when he was still calling it “Stolen Secrets of the Wizards of the Ivory Tower” and some time before he’d written the book on higher order programming in perl. In that talk, Dominus pointed me towards Abelson and Sussman on the Structure and Interpretation of Computer Programs (one of the great texts on programming – everyone who is serious about their craft should read it). Since then I’ve mucked about in Smalltalk, Ruby, Haskell, Scheme, Javascript and Clojure as well as Perl and higher order techniques had stood me in good stead in all of them.

Before we start

There will be code snippets here which I don’t pretend are written in what is normally considered good javascript style. In particular there’s no error checking and variable names are short and almost devoid of meaning. The short variable names help avoid line wrapping headaches within the blog format and the lack of error checking is to enable us to focus on the higher ordered techniques that we’re interested in. In the real world, we don’t live on the happy path and you are expected to think before you code. And write tests.

I do try to follow the Haskell convention that a list or array of stuff will be called something like xs and x will the variable used to hold the current element of xs as we iterate over it.

I’ve not seen that in Fowler…

Higher ordered programming is just another way of removing repetition from your program by extracting the ‘shape’ of an algorithm into a separate function. In the presentation I asked for a show of hands from people who’d done higher ordered programming before. A scattering of hands went up. Then I asked who’d used jQuery’s $.each(function {...}) and most hands in the room went up. Which means that most of the people in the room had used higher order programming and never realised it.

If you look at it right, $.each is just the result of applying the Extract Method refactoring on a for loop in a way that’s impossible to do in, say, Java. Here’s a barebones version of $.each

each = function (xs, f) {
  var i, len, each;
  for(i = 0, len = xs.length; i < len; i++) {
    f(xs[i]);
  }
}

The jQuery each does rather more than this, but at it’s core it’s about turning an iterative loop into a function that takes a collection and a callback function which is applied to every element in the collection. What could be easier?

Higher Leverage Javascript

What Javascript’s designers thinking? Seriously, if you’re going to provide a lovely language with first class functions, why on earth would you want to spell the keyword that makes a function ‘function’? Lisp is stuck with lambda because, well, because it’s a bloody old language and they didn’t know any better then. Arguably it’s why they ended up with the macro capabilities they did – anything to avoid spelling out lambda every time. Anything longer than Lambda’s just silly. So, for the rest of the examples I’m switching to CoffeeScript. There’s a crib sheet over on that site which should allow you to work out what’s going on, and if you take away nothing else from this article than “Investigate CoffeeScript”, you’ll be ahead of the game.

Refactoring with higher order functions

Consider sum and prod below:

sum = (xs) ->
  r = 0
  (r = r + x) for x in xs
  r

prod = (xs) ->
  r = 1
  (r = r * x) for x in xs
  r

Notice how they both have the same shape. They both set an initial result and then, for each element in the collection, the intermediate result with an element of the collection using an operator and, when they’ve finished iterating over the collection, they return the last value of the intermediate result.

Let’s apply Extract Temporary a few times to make the repetition even more obvious:

sum = (xs) ->
  init = 0; 
  f = (a,b) -> a + b
  r = init
  (r = f(r,x)) for x in xs
  r

prod = (xs) ->
  init = 1;
  f = (a,b) -> a * b
  r = init
  (r = f(r,x) for x in xs
  r

Now we can just Extract Method and replace the repetition:

inject = (i, f, xs) -> 
  r = i
  (r = f(r,x)) for x in xs
  r

sum = (xs) -> inject 0, ((a,b) -> a+b), xs

prod = (xs) -> inject 1, ((a,b) -> a*b), xs

If we were writing this in a language that didn’t have an OO feel to it, we’d probably call the extracted function foldl, but the OO languages that already have it, it tends to be called inject or, in Smalltalk inject: into: and implemented as part of the enumerating protocol of a collection. Smalltalkers will tell you that inject:into: is the fundamental method for enumerating a collection, pretty much everything else in the protocol can be implemented in terms of it (so it’s a good idea to make it as fast as you possibly can). In Ruby, meanwhile, inject is implemented in terms of each, which seems the wrong way around for me. Here’s a few typical functions for messing with collections, implemented in terms of inject:

grep = (pred, xs) -> 
  inject [], ((rs, x) -> if pred x then rs + [x] else rs), xs

map = (f, xs) ->
  inject [], ((rs, x) -> rs + [f x]), xs

each = (f, xs) -> 
  inject undefined, ((_, x) -> f x; _), xs

I see repetition!

Have you noticed it? Everything written in terms of inject has the form:

something = (..., xs) -> inject ..., xs

If we weren’t higher order programmers, we should probably just shrug our shoulders and give it up as a bad job, but we are higher order programmers, and we’re working in a language that allows functions to return functions. We’d like to be able to write:

sum = injector 0, (a,b) -> a + b

where injector is a function like inject but which returns a function of one argument which will iterate over the supplied collection.

We could write injector directly, but where’s the fun in that? Instead let’s write:

with_list = (f) ->
  arity = f.length
  (args...) ->
    if arity > 0 && args.length != arity - 1
      throw "Oops!"
    (xs) -> f args..., xs

So, with_list is a function which takes a function, f, of n arguments an returns a function of n – 1 arguments (args) that returns a function of one argument (xs) that applies f to a list of arguments made up of appending xs to args.

So that’s code that operates on code to write code that writes code. Confused? Don’t worry – it gets worse.

Once we have with_list we can dispose of sum and prod by writing:

injector = with_list inject

sum = injector 0, (a,b) -> a + b
prod = injector 1, (a,b) -> a * b

It doesn’t help us out with grep, map and each, but I’m not feeling clever enough to fix that at the moment so I’ll leave it as an exercise for the interested reader.

Functional vs. OO Style.

So far, I’ve been writing everything in a very functional style. If I’m honest, I prefer it that way for a lot of things, but I’m aware that I’m not in the majority on this front and some people would rather see inject and co as methods on objects like Allan Kay intended. So let’s see if we can write something which will allow us to transform our functions into methods. Essentially, we need to get rid of the last argument (xs) and make the functions operate on this instead.

We already have with_list which transforms a function on n arguments into a function of n – 1 arguments. Maybe we can write with_this like so:

with_this = (f) ->
  (args...) -> f args..., this

And, because I’m feeling mischievous, why not write:

Function.prototype.to_method = with_this with_this

And then we have:

Array::sum = sum.to_method()
Array::prod = prod.to_method()
Array::inject = inject.to_method()
Array::grep = grep.to_method()
Array::map = map.to_method()
Array::each = map.to_method()

Hopefully it’s obvious what’s going on here. If it isn’t, you might want to reread everything so far and press on when you’ve got the hang of it. Remember, higher order programming arises from applying the following principles:

Principle 1: Functions can take functions as arguments

Principle 2: Functions can return new functions

That’s all there is too it. Everything else is just working through the implications. Just remember that if counter_from = (n) -> () -> n = n + 1, then c1 = counter_from 1 and c2 = counter_from 2 are two different functions that do not share the same n. Easy, no?

Right, now you’ve got that straight, let’s move on to

Memoization

Let us suppose that you are a naïve mathematician who has decided to implement a function to find the nth number in the fibonacci sequence. You know that Fib0 is 0, Fib1 is 1 and that Fibn is Fibn-1 + Fibn-2 otherwise. So you write:

fib = (n) ->
  switch n
    when 0 then 0
    when 1 then 1
    else fib(n-1) + fib(n-2)

Obvious. Everything’s fine for small values of n, but it takes an age, and an awful lot of memory to find Fib50. What can be wrong.

Because you know something about higher order functions, you write:

calls = 0;
counted = (f) -> (args...) ->
  calls += 1
  f args...

fib = counted fib

You worry a little about that free variable, calls, but you’re busily debugging and you’re going to throw it away once you’ve worked out what’s going on. Instead you run a test program:

calls = 0;
print "#{fib 30} called 'fib' #{calls} times\n"
calls = 0;
print "#{fib 31} called 'fib' #{calls} times\n"

And you are shocked when the output is:

832040 called 'fib' 2692537 times
1346269, called fib 4356617 times

Clearly you do not have an efficient algorithm.

You could, at this point, sit down and think and come up with a more efficient algorithm (which, frankly, isn’t all that hard in the simple case of finding fibonacci numbers, but which can get harder in the case of, say, the Ackermann function). But, dammit, you’ve reduced the problem of finding Fibn to adding together two numbers you’ve already found, why not just make the computer remember that?

memoize1 = (f) -> 
  results = {}
  return (arg) ->
    if arg of results
      results[arg]
    else
      results[arg] = f arg

fib = memoize1 fib

And now when you run your test you get:

832040, called fib 31 times
1346269, called fib 1 times

I think we can safely say that that’s an improvement. And it’ll work for any function of one argument, where the argument is of a type that can be used as a key in a Javascript dictionary. Hmm… that’s possibly not as wide as we’d like.

However, usually when we go to memoize a function, we know something about that function and its expected arguments. If we can write a function to normalize the function arguments into a valid key, then we can write a more general memoize that takes the function to memoize and an optional normalization function:

memoize = (f, normalize) ->
  results =  {}
  normalize ||= (x) -> x
  (args...) -> 
    key = normalize args...
    if key of results
      results[key]
    else
      results[key] = f args...

Of course, given enough thought and/or access to Wikipedia, we could well come up with the closed form function to find Fibn

fib = (n) -> 
  Math.round(
    Math.pow(
      (1+Math.sqrt 5) / 2, 
      n
    ) 
    / Math.sqrt 5
  )

But for sheer readability, I’ll take the memoized form.

I promise to pay the bearer on demand…

Suppose you have 1000 things which are sortable, and you want to find the first ten. You could sort the entire list and just grab the first ten elements of the sorted list. But that’s an O(nlog2n) algorithm. What if we had a means of only sorting the first m elements and then stopping (or pausing)?

We do. Or we will. And to do this, we’re going to use a functional data structure called a Stream.

If you’ve spent any time around functional programmers, you will have come across a fundamental data structure called the Pair, which is usually used int list structure. A Pair is simply a pair of pointers. When pairs are used in what’s called list structure, the first (head, or car) pointer points to the value of the cell, and the second (tail or cdr) points to the next element in the list, or to a special value called nil or ‘the empty list’. Essentially a singly linked list then. When you’re following most introductions to lisp, the pair is pretty much all you’ll need to use because given that simple structure, you can build pretty much any structure you desire. Real world lisp programmers other, more specialised (faster) data structures available, but anything new or complicated tends to get built with pairs (including the data structure we’re actually interested in right now).

A stream is like a pair in that it has a head value and a tail. But a stream’s tail isn’t another stream, it’s a promise to compute another stream. So the head is akin to a lump of silver or gold – something with inherent value – and the tail is like a bank note, which is simply a ‘promise to pay the bearer on demand’ with another stream.

Show me the code!

This is all a bit abstract, so let’s look at a stream implementation.

the_empty_stream = 
  is_empty: true
  take: (n) -> []

class Stream
  constructor: (@head, p) ->
    tail = p
    forced = false
    this.force = ->
      if forced then tail
      else
        forced = true
        tail = tail()
    take: (n) ->
      if n == 0
        []
      else
        [this.head].concat this.force().take n - 1
  is_empty: false

force = (s) -> s.force()
take = (n, s) -> s.take n
is_empty = (s) -> s.is_empty

There you go. Admittedly, force is a little tricky. Calling stream.force() makes the stream evaluate the promise to calculate the tail, and returns that value. It’s written the way it is so that, once we’ve evaluated the promise function, we drop our reference to it, which should make our memory usage a little more efficient.

None of this makes it clear how we’ll actually use streams, so let’s add a few utility functions:

take_stream = (n, s) -> 
  if is_empty(s) || (n == 0)
    the_empty_stream
  else
    new Stream s.head, -> take_stream n-1, force s

filter_stream = (pred, s) -> 
  if is_empty s
    the_empty_stream
  else if pred s.head
     new Stream s.head, ->
       filter_stream pred, force s
  else
    filter_stream pred, force s

map_stream = (f, s) ->
  if is_empty s
    the_empty_stream
  else
    new Stream f(s.head), ->
      map_stream f, force s

cat_streams = (s, t) ->
  if is_empty s
    t
  else
    new Stream s.head -> cat_streams force(s), t

print_stream = (s) -> 
  if is_empty s
    return
  print "#{s.head}\n";
  t = force s
  if t.is_empty then return
  else print_stream t

Array::as_stream = ->
  switch this.length
    when 0 then the_empty_stream
    else
      [x,xs...] = this
      new Stream x, -> xs.as_stream()

array2stream = (xs) -> xs.as_stream()

It doesn’t look like much so far, but there’s more than enough to solve our top ten out of a thousand problem.

First let’s build ourselves an array of 1000 integers chose at random.

make_rands = (ceil) ->
  new Stream Math.floor(Math.random() * ceil), -> make_rands max

big_list = take 1000, make_rands 1000

The function make_rands returns an infinite stream of random numbers and we take the first 1000. What we want to do now is a partial quicksort. First, we write qspart which partitions an array of integers into three arrays. All the members of the original array that are less than its first element, all the members that are equal to it and all the members that are greater than it.

qspart = (xs) ->
  [p, ys...] = xs
  inject(
    [[], [p], []], 
    ((bins, each) ->
      i = 
        if each < p then 0
        else if each > p then 2 
        else 1
      bins[i].push each
      bins
    ),
    ys
  )

This would turn the list [2,3,5,2,1,0] into [[1,0],[2,2],[3,5]]. Now we just have to work out how to make that into a sorted stream.

make_sorted_stream = (xs) ->
  switch xs.length
    when 0 then the_empty_stream
    else
      [ls,[p,ps...],hs] = qspart xs
      cat_streams(
        make_sorted_stream(ls),
        new Stream p, ->
          cat_streams(
            ps.as_stream(),
            make_sorted_stream(hs)
          )
      )

We’re partitioning the list into a list of those less than the pivot (ls), a list of elements equal to the pivot ([p,ps]) and a list of those greater than the pivot (rs). Then we build a stream by concatenating the sorted stream built on ls with stream whose head is the pivot and which promises to supply a stream made by concatenating the stream made from all the other elements equal to the pivot with the sorted stream made on rs. We’re reasonably confident that this should terminate because the sorted stream on an empty array is the empty stream, and every sub list made by qspart is shorter than the list in its argument.

If we wrap qspart to get some kind of trace then get the top 3 entries from a list of 10 random numbers, like so:

qs = ((f) -> (xs) -> 
  res = f xs;
  print "# qspart([#{xs}] -> #{res[0]}|#{res[1]}|#{res[2]}\n"
  res)(qs)

print_stream(
  take_stream(
    3, 
    make_sorted_stream(
      take(10, make_rands 50)
    )
  )
)

We see output like:

# qspart([41,7,6,13,43,47,23,15,34,22]) -> 7,6,13,23,15,34,22|41|43,47
# qspart([7,6,13,23,15,34,22]) -> 6|7|13,23,15,34,22
6
# qspart([13,23,15,34,22]) -> |13|23,15,34,22
7
# qspart([23,15,34,22]) -> 15,22|23|34
# qspart([15,22]) -> |15|22
13

Which gives us some idea of the shape of the computation. Notice how we only call qspart when necessary and we’re doing the minimum amount of work needed to get the first three entries from the sorted list.

Higher Order Javascript in the Real World

Now you know what higher order programming looks like, you’ll start seeing it everywhere in libraries like jQuery, YUI and Node.js. I’ve not covered what’s called ‘continuation passing style’, but it’s almost impossible to do asynchronous stuff without it.

So, don your HOF sensing glasses and go play.

One last thing

Why does this stuff get called higher order programming?

I think the idea is that functions that manipulate functions are of a higher order than functions that manipulate other ‘lower’ data structures. I tend to think that it belongs in a class with ‘metaprogramming’ – words and phrases that may lead one to infer that the meta- or higher order programmer is doing something inherently harder than the bread and butter programming indulged in by all those plain old programmers.

When it comes down to it, that higher order functions are just functions and metaprogramming is just programming and the sooner we learn that, the faster we can improve our programming practice.

Update

For those of you who are finding the CoffeeScript a wee bit daunting (I personally like the lack of noise, but your mileage may vary), there’s a thread on sitepoint which translates many of the examples here into more idiomatic Javascript than the CoffeeScript compiler can manage. You may find it useful.

I’ve written a followup post on asynchronouse streams which discusses using higher order techniques in the browser window to do something a little more ‘real’ than the examples shown here.

Javascript heresy 5

Posted by Piers Cawley Wed, 29 Oct 2008 08:16:00 GMT

So, remind me, what’s the rationale for always using the optional ; when you’re writing Javascript? The only reasons I can think of, off the top of my head are, “Because you’ll break minifiers without it” and “Because Douglas Crockford doesn’t like it”. Well, broken minifiers that can’t parse the language correctly can kiss my fat hairy arse and argument from authority cuts little ice here.

Gareth Rushgrove pointed me at an article, which suggested that it’s because Javascript will insert a semicolon after the return in:

return
{ key: "value" }

But that’s not exactly surprising, and falls squarely into the “Don’t do that then” category of bugs, or putting it another way, the Dominus Doctrine (“You can’t just make shit up and expect the computer to know what you mean, retardo!”) applies.

Ruby also has an optional semicolon, but good style is avoid using them and we seem to survive. In fact, the Ruby parser is rather less capable than Javascript’s:

jQuery('.class')
  .addClass('whatever')
  .html('New inner HTML')


is legal Javascript, but, in Ruby you have to write:

jQuery('.class') \
  .addClass('whatever') \
  .html('New inner HTML')


because if you don’t the compiler throws its toys out of the pram and, for bonus points, the resulting parse error implies that the parser knows what you meant but decided to throw the error anyway. Ho hum.



Is there something I’ve missed? Or should I make a preemptive stand against incompetent minifiers and start writing my Javascript without semicolons?

Updates

In the comments, “James” offers a succinct piece of code using Prototype that demonstrates the problem rather neatly. In the absence of semicolons, code like:

var foo = 3
, bar = 2 + foo

[foo, bar].each(function (i) { console.log(i) })


gets parsed as

var foo = 3
, bar = 2 + foo[foo, bar].each(...)


Which isn’t exactly what you want. If I were feeling churlish, I might argue that such problems are one reason why the jQueryish way:

var foo = 3
, bar = 2 + foo
$.each([foo, bar], function () {...})


is a better way of iterating over things, but I’m not entirely sure that it is.

Looks like I’ll keep taking the semicolons.

Work with us 2

Posted by Piers Cawley Thu, 03 Jul 2008 06:00:00 GMT

If the last post about our Javascript issues didn’t put you off, then you might be interested to know that we’re hiring. If you’re an experienced, test infected Ruby on Rails programmer with some Javascript and a real world consumer website or two under your belt, and you’re happy to work in Newcastle upon Tyne, then we definitely want to hear from you. I’d probably be interested in at least hearing from you if you’re an experienced dynamic language programmer who has only recently made (or is considering making) the switch to Ruby and Rails. It’s only syntax after all.

The money’s decent, the work is interesting, the people (well, apart from me, obviously) are great, and Newcastle’s a fantastic city. Drop me a line, ping me on AIM/gTalk/Twitter or just send your CV to the jobs@amazing-media.com.

Usability testing (throws) rocks 4

Posted by Piers Cawley Thu, 03 Jul 2008 04:55:00 GMT

Usability testing is wonderful. But wow, its humiliating.

I’ve spent the last few weeks working on the Amazingtunes in page player. Amazingtunes is a music site, so we need to play music. However, we don’t like the way that most music sites work; either the music stops as you go from one page to another, or the player is a huge Flash app running in its own window. There has to be a better way. There needs to be a popup window if you want to eliminate stop/start behaviour, but there’s surely no reason not to keep the controls on the main page.

So, we set about writing somthing that did just that. We settled on using Jeroen Wijering’s excellent flvPlayer, which handles the media formats we need and has good Javascript control and communications. This sits in the child window and we use Javascript cross-window communication to have a player controller in the main window that looks something like:

Piers Cawley on amazingtunes.com
Uploaded with plasq’s Skitch!

This is all done in HTML and and Javascript, the progress bar does the Safari trick of running behind the tune data links, the buttons do their AJAX magic and the whole thing is rather slick, though I say so myself.

At least, we thought it was slick until we pointed the usertesting.com legions at it. Without exception, they ignore the in page player, foreground the popup and use the teeny weeny controls on the flash player. Originally, the popup window didn’t even display any transport controls, it just had a picture of some speakers and some text asking the user not to close it because it was playing the tunes. We added transport controls as a stopgap while we made the in page player work properly.

I sound like I’m whinging don’t I? It’s certainly a blow to the ego to see something we spent so much time and attention on being ignored by our sample users. On at least one occasion, while watching the screencasts I found myself boggling at the things the users did, and if I didn’t shout “Just play some bloody music!” at the screen, then I came worryingly close.

It would be easy to retreat into a state of denial: “They’re not our target users! They’re stupid! They’re American! If they would only magically intuit the way we think they should use the site!”. And maybe it would be comforting to do so, for a while. The right thing to do is to suck it up – take away from those videos the sure and certain knowledge that bits of the site don’t work and do something about it.

We may dislike the ‘popup window for transport controls’ model of controlling music playback, but users are cool with it. And it’s not as if the work we did on making the in page player work is going to be wasted – widget is straightforwardly event driven so it’ll work just as well in the popup window, and the communication protocol will be much simpler. Having the player in its own window means we’ll be able to extend its interface in ways that would be hard when the player had to share window space with the rest of the page. In the end, it’s all good.

But… damn that in page player was sweet. I learned Javascript as I wrote it (mostly by pretending it was Perl with odd syntas) and I’m bloody proud of it. I’ll happily replace it with the next iteration (which I’m already working on), but it’ll be with a pang of remorse all the same.

A quick Javascript formatting tip 10

Posted by Piers Cawley Wed, 16 Apr 2008 04:01:00 GMT

IE’s a pain. The particular pain I want to write about is its pickiness about Javascript object literals. Consider the following Javascript object:

{ success: function () {...},
  failure: function () {...},
}

If you’re used to programming in Perl or Ruby, that trailing comma’s perfectly fine, in fact leaving it there is sometimes considered good practice because it makes it easy to extend the hash, just add a new row and leave another trailing comma.

The trouble is, it’s not strictly legal to do that in Javascript. Pretty much every implementation out there will allow it though.

Except IE.

So, I’ve taken a leaf out of Damian Conways Perl Best Practices and writing my object literals as:

{ success: function () {...}
, failure: function () {...} 
}

By sticking the comma at the beginning of the line, I’m never going to make an object that breaks IE, and adding a new line to the hash is straightforward too. Just stick the cursor in front of the }, type my leading comma, space, attribute name, and hit return when I’m finished.

I’ve also started using the same practice pretty much everywhere else that I’ve got a comma separated list of things:

var foo
  , bar
  , baz
  ;
$.each( anArray
      , function () { ... }
      );

It looks weird at first, but trust me, it grows on you.

Update

In the comments, I make reference to tweaking Steve Yegge’s excellent js2-mode to handle leading comma style a little more gracefully. Since then, I’ve made it work and attached a diff to this issue on the project’s issue tracker.

Javascript scoping makes my head hurt 13

Posted by Piers Cawley Thu, 20 Mar 2008 11:41:00 GMT

Who came up with the javascript scoping rules? What were they smoking. Here’s some Noddy Perl that demonstrates what I’m on about:

my @subs; 
for my $i (0..4) {
  push @subs, sub { $i }
}

print $subs[0]->(); # => 0;

Here’s the equivalent (or what I thought should be the equivalent) in Javscript:

var subs = [];
for (var i in [0,1,2,3,4]) {
  subs[i] = function () {
    return i;
  }
}
alert subs[0]() // => 4

What’s going on? In Perl, $i is scoped to the for block. Essentially, each time through the loop, a new variable is created, so the generated closures all refer to different $is. In Javascript, i is scoped to the for loop’s containing function. Each of the generated closures refer to the same i. Which means that, to get the same effect as the perl code, you must write:

var subs = [];
for (var shared_i in [0,1,2,3,4]) {
  (function (i) {
    subs[i] = function () {
      return i;
    };
  })(shared_i);
}
subs[0]() // => 0

Dodgy Ruby scoping

I had initially planned to write the example “How it should work” code in Ruby, but it turns out that Ruby’s for has the same problem:

subs = [];
for i in 0..4
  subs << lambda { i }
end
subs[0].call # => 4

Which is one reason why sensible Ruby programmers don’t use for. If I were writing the snippet in ‘real’ Ruby, I’d write:

subs = (0..4).collect { |i|
  lambda { i }
}
subs[0].call # => 0

My conclusion

Javascript is weird. Okay, so you already know this. In so many ways it’s a lovely language, but it does have some annoyingly odd corners.

Getting to grips with Javascript

Posted by Piers Cawley Fri, 18 Jan 2008 02:30:00 GMT

I’ve been busily adding AJAX features to the work website, and I got bored of writing Form handlers. I got especially bored of attaching similar form handlers to lots of different forms on a page, so I came up with something I could attach to document.body and then plug in handlers for different form types as I wrote them.

So, I wrote FormSender and set up my event handler like so:

FormSender.onSubmit = function (e) {
  if (canDispatch(e)) {
    YAHOO.util.Event.stopEvent(e);
    YAHOO.util.Connect.setForm(e.target);
    YAHOO.util.initHeader('Accept', 'application/javascript, application/xml');
    YAHOO.util.Connect(e.target.method.toUpperCase(), e.target.action,
    callbackFor(e));
  }
};

jQuery(document.body).each(function () {
  YAHOO.util.Event.addListener(this,  "submit", FormSender.onSubmit);
});

I decided to mark any Ajax dispatchable forms using a class of ‘ajax’, the idea being that it would be a simple thing to check
jQuery(e.target).hasClass('ajax'), but there was a snag. We had two sorts of forms on our pages, forms built using form_for(..., :class => 'ajax') and forms built using button_to(..., :class => 'ajax'), and they attached their classes in different places. In the form_for case, the class was on the form tag, but in the button_to case, it was on the generated form’s submit field. One option would be to monkey patch button_to, or roll my own ajax_button_to, but I ended up writing canDispatch like so:

function canDispatch(e) {
  jQuery(e.target).find(':submit').andSelf().hasClass('ajax');
}

This uses jQuery to build a list of the form, and its submit button, and then checks to see if any member of that list has the class ‘ajax’.

So, we can now tell if the source of a submit event is a form we should be doing AJAX dispatch with. The next trick is to work out what needs to be done with the results of sending the form. One option is the Prototype trick of simply evaluating the returned javascript, but it often makes sense to keep the behaviour clientside and just have the server return a datastructure. I decided that the way to do this would be by adding a second class to a form which used a none default handler, and then keep a hash of callback constructors keyed by class. This made callbackFor look like:

function callbackFor(e) {
  var candidates = candidateClasses(e.target);
  for (var i = 0; i < candidates.length; i++) {
    if (FormSender.callbacks[candidates[i]]) {
      return new FormSender.callbacks[candidates[i]](e);
    }
  }
  return new FormSender.callbacks.ajax(e);
}

candidateClasses is, again, a little more complex than I’d like, by virtue of the differences between button_to and form_for differences, but still reasonably straightforward, thanks to jQuery:

function candidateClasses(element) {
  return
  jQuery(element).find(':submit').andSelf() 
    .filter('.ajax').attr('className')
      .replace(/ajax/, '').trim().split(/ +/);
}

JQuery gets the form and its submit button, then selects the tag that has the ‘ajax’ class and pulls out the full className string. The replace gets rid of ‘ajax’, trim chops any useless whitespace off either end, and split(/ +/) turns it into an array of classnames. The replace -> trim -> split pipeline has the feel of something that must already exist in some DOM interface somewhere, but I’m not sure where, so I rolled my own.

Once we have a list of classes it’s easy to just cycle through the candidates until we find one that matches a callback constructor, falling back to the default where nothing matches.

For completeness, I’ll show you my current default handler, which I expect to be extending to deal with a couple more media types and, in the case of the failure handler, more failure statuses.

FormSender.callbacks.ajax = function (e) {
    var form = e.target;
    this.scope = form;
};

FormSender.callbacks.ajax.prototype.success = function (o) {
    switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
    case 'application/javascript':
    case 'application/x-javascript':
    case 'text/javascript':
        eval(o.responseText);
        break;
    default:
        YAHOO.log("Can't handle AJAX response of type " + o.getResponseHeader['Content-Type']);
    }
};

FormSender.callbacks.ajax.prototype.failure = function (o) {
    switch (o.status) {
    case 401:
        Authenticator.loginThenSubmit(this);
        break;
    default:
        switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
        case 'application/javascript':
        case 'application/x-javascript':
        case 'text/javascript':
            eval(o.responseText);
            break;
        default:
            YAHOO.log("Can't handle AJAX failure response of type " + o.getResponseHeader['Content-Type']);
        }
    }
};

You’ll notice a reference to Authenticator.loginThenSubmit in the 401 handler, but that’s something I’ll save for another day.

A note on namespacing

Although I’ve been showing the various FormSender helper functions as if they were in the global namespace, in the real code they’re wrapped in a function call:

var FormSender = (function () {
  var candidateClasses = function (element) {...};
  var callbackFor = function (e) {...};
  ...
  var onSubmit = function (e) {...};
  return {onSubmit: onSubmit, callbacks: {}};
})();

I love the (function () {...})() pattern – it’s a great way of keeping your paws out of the global namespace until you really, really need to.

FormSender Benefits

Aside from the obvious benefit of drastically reducing the number of onSubmit event handlers registered with the browser, I found that using FormSender has simplified some of my response handlers. For instance, one form would get a chunk of html back from the server and would use that to replace the div that contained the form. But the new div also contained a form that needed to have Ajax behaviour, so a chunk of the handler code was concerned with reregistering onSubmit handlers for the new form (or forms). No fun. By switching to a single, body level, form handler, that problem simply disappears – so long as the new forms have the right class, they automatically get the appropriate behaviour. Result.

Obviously, FormSender is unobtrusive javascript, which is nice, and its pluggable nature means it’s easy to extend just by writing new response handlers and registering them with the FormSender object.

Future Directions

One obvious extension to FormSender is to pull out the meat of the onSubmit method into the callback object to allow for forms that don’t simply send themselves to the server. Another is to wrap my head around the workings of Javascript’s object model to make it easy to build handlers that don’t duplicate the behaviour of the default handler through the medium of copy and pasting code…

Your comments please?

I’m still very new to Javascript as a programming language and I’m sure I’m doing plenty of boneheaded things here. Please let me know if there’s things I can do to improve this, or point me at any libraries that already cover this ground.

So... I was wondering 3

Posted by Piers Cawley Sat, 17 Nov 2007 12:54:23 GMT

Has anyone written an Atompub client in JavaScript yet?

Finally learning JavaScript 10

Posted by Piers Cawley Wed, 16 May 2007 17:17:00 GMT

For years I’ve managed to dodge learning JavaScript. People have told me it’s a fine language with some dodge implementations. Friends have built an entire business model in the language, and I’ve continued to treat as if it were The Sound of Music (I’m 39 now and I have managed to completely avoid watching that film).

Except, today I finally had a technical need that I couldn’t dodge by writing another .rjs file, so with the help of the Prototype docs, a helpful article or two and healthy dose of cribbing from others, I was off.

What do you know, it really is a fine language. I doubt I’m writing idiomatic code though – I’m leaning heavily on closures and private member functions, and I haven’t written a single for loop yet, but it seems to accommodate my style.

Now, if I can just get my head round the innards of keyboard events I shall be a great deal happier.


Just A Summary