CL-BEANSTALK, an Implementation of the beanstalk v1.4.2 protocol in Common Lisp

CL-BEANSTALK implements the version 1.4.2 of the Beanstalk queuing service protocol. It allows splitting programs into "worker" parts that execute units of work asynchronously, by fetching them from the beanstalk daemon.

Installing CL-BEANSTALK

CL-BEANSTALK requires three CL libraries to run:

  1. usocket
  2. flexi-streams
  3. split-sequence

All of these libraries are available via clbuild and other packaging services.

Once you have these libraries, git clone the cl-beanstalk repository and symlink the cl-beanstalk.asd file to your asdf:*central-registry* directory:

git clone git://github.com/antifuchs/cl-beanstalk.git
ln -sf `pwd`/cl-beanstalk/cl-beanstalk.asd /your/asdf-central-registry-dir/

Using CL-BEANSTALK

First, all protocol functions reside in the BEANSTALK: package. Their names are the same as in the beanstalk protocol documentation.

Second, there are several error conditions specified in the protocol document, and all of these map to a specific lisp condition. All error conditions are subtypes of BEANSTALK:BEANSTALK-ERROR. There are several such error conditions, for an exhaustive list see the file package.lisp.

Connecting to the queue server and a little example

To start using CL-BEANSTALK, you need to connect to a beanstalk daemon. To do this, it provides functions BEANSTALK:CONNECT and BEANSTALK:DISCONNECT, and the macro BEANSTALK:WITH-BEANSTALK-CONNECTION:

(beanstalk:with-beanstalk-connection (conn "localhost" 11300)
  ;; Put jobs into a specific tube:
  (beanstalk:use conn "URLs-to-fetch")
  ;; Put two pieces of work into the tube, both at priority 100, 
  ;; no delay, with 180 seconds maximum running time:
  (beanstalk:put conn 100 0 180 "http://boinkor.net/")
  (beanstalk:put conn 100 0 180 "http://planet.lisp.org/"))

Getting work from the queue server

Workers need to get their units of work from somewhere, and they need to tell the queue server that they have performed (or failed to perform that work). Here's a simple (meaning, untested) worker implementation that fetches URLs from the tube in the last example and handles them in some way:

(beanstalk:with-beanstalk-connection (conn "localhost" 11300)
      ;; Get jobs from a specific tube:
      (beanstalk:watch conn "URLs-to-fetch")
      (loop 
        ;; reserve the next job:
        (multiple-value-bind (url id status) (beanstalk:reserve conn :timeout 1)
              ;; If no new jobs arrived within 1 second, quit:
              (when (eql status :timed-out) (return))
              (handle-response (drakma:http-request url))
              ;; The job is done, remove it from the queue:
              (beanstalk:delete conn id))))

Note that this code can run on several worker processes (indeed, worker machines) at once, and perform several thousands units of work in parallel.

For a complete reference, see the beanstalk protocol documentation.

Conditions and errors

Whenever it makes sense to do so, a function signals a non-error condition to highlight an untypical event: For example, if a job is buried instead of put on the "ready" queue because the server can't reverve enough memory, that results in a condition being signaled via WARN or SIGNAL.

These functions will signal out-of-band information from the queue server: