Loose Bits

Book Announcement - Backbone.js Testing

2013-06-13T11:30:00Z

Backbone.js Testing Book

I am very pleased to announce that I am in the last stages of publishing Backbone.js Testing - a book on Backbone.js test architecture and development due in final form around the end of June.

A brief abstract of the book’s content is available from the Packt Publishing website:

Frontend web applications are soaring in popularity and the Backbone.js library is leading this charge with a modular, lightweight approach for organizing JavaScript web applications. At the same time, testing client-side JavaScript and Backbone.js programs remains a difficult and tedious undertaking.

Backbone.js Testing brings sensible practices and current techniques to the challenges of Backbone.js test development. The book introduces fundamental testing concepts, comprehensive test infrastructure design, and practical exercises to easily and systematically test modern JavaScript web applications.

The book progresses from Mocha test suites and Chai assertions to advanced test mocks and stubs with Sinon.JS. The requisite libraries and utilities are introduced with in-depth examples and best practices for integration with your applications. The book guides you through the test planning and implementation processes for your application models, views, routers, and other Backbone.js components.

Backbone.js Testing gives you the tools, examples, and assistance to test your Backbone.js web applications thoroughly, quickly, and with confidence.

To provide a better idea of the book’s structure, the tentative chapter titles are as follows:

Setting up a Test Infrastructure
Creating a Backbone.js Application Test Plan
Test Assertions, Specs, and Suites
Test Spies
Test Stubs and Mocks
Automated Web Testing

All of the code examples in the book are provided online at the website backbone-testing.com and as an open source GitHub repository.

The Road to my First Book

Application testing is difficult, and frontend JavaScript testing can often be a downright nightmare. After years of working through difficulties in frontend testing practices and technologies, I have gained a lot of experience through the various trials and tribulations bringing me to my current role in architecting (and testing) large Backbone.js applications.

Packt Publishing approached me towards the end of last year about authoring a book on the subject and I signed on. My hope with the project is that I will be able to provide a resource to reduce the pain in frontend tests with a structured programming guide and practical examples using various (exciting and interesting) modern testing libraries.

It has been a stressful, hectic, and wonderful experience writing the book, and I’m really looking forward to completing the project. And, I will be happy to be able to reclaim my nights and weekends that have been devoted to the book over the past couple of months.

After the book is published, I plan on writing a series of posts about authoring a technical book, with all of the ups and downs that the process entails. In the meantime, I just have to proof each chapter one final time and it should be off to the presses, so to speak!

Node.js in Production

2013-05-10T11:30:00Z

Node.js in Production

Curiosity Media runs the world’s largest Spanish learning website, SpanishDict.com, backed by many different Node.js applications. Since rolling out our first Node.js applications over a year ago, we have collected a fair amount of good and bad operational experiences.

I created a talk about our journey with Node.js for the Seattle Node.js Meetup group’s May 8th, 2013 event. My slides are available at the following locations:

GitHub Site: A live, navigable reveal.js website. (Note: use the space bar to advance slides and arrow keys to navigate.)
SlideShare: More traditional format, with download (PDF) links.

And, here’s an embedded format:

Five Node.js Production Tips

My talk discussed the following five topics:

Know when to Node: Should you use Node.js?
Keep up with Node: Stay up to date with Node.js and libraries.
Design for failure: Fail and recover at multiple levels.
Isolate services: Separate resource and failure classes.
Analyze everything: Data drives problem discovery and action.

There is, of course, much more to running Node.js in production, but hopefully my talk and slides provide a bit of insight into some of the considerations to keep in mind when taking a Node.js application to production.

Sphinx Bootstrap Theme 0.2.0 - Now with Bootswatch!

2013-04-10T11:30:00Z

Bringing Bootswatch to Sphinx

The Bootstrap Theme for Sphinx documentation continues to evolve, and I’m pleased to announce the v0.2.0 release. The theme integrates Twitter’s Bootstrap library into Sphinx, and v0.2.0 adds new support for Bootswatch CSS extensions, which provide a lot of great new UI possibilities for your documentation. We’ve also internally updated to Bootstrap 2.3.1.

The project’s demonstration site now uses the ”united” Bootswatch theme to give a clean and well… orange-ish feel to the site. Here is a screenshot of the demonstration site in a desktop viewport:

The project is available for download from GitHub and PyPi. You are encouraged to update and try out some Bootswatch goodness!

Adding Bootswatch

As a preliminary note, the use of Bootswatch themes is entirely optional, and disabled by default - the project starts with a vanilla Bootstrap appearance, which you can customize to your heart’s content. To enable a specific Bootswatch theme, set the following option in your ”conf.py” file:

html_theme_options = {
    # Bootswatch (http://bootswatch.com/) theme.
    #
    # Options are nothing with "" (default) or the name of a valid theme such
    # as "amelia" or "cosmo".
    #
    # Note that this is served off CDN, so won't be available offline.
    'bootswatch_theme': "united",
}

The list of Bootswatch themes you can now try out include:

As a final note, the Bootstrap CSS files are served off of CDN, which means that if you are offline, your site will have a non-themed, vanilla Bootstrap appearance. We are considering refactoring the Bootswatch code to be available offline as well.

Sphinx Bootstrap Theme 0.1.6 - Bootstrap and Other Updates

2013-02-12T11:30:00Z

Sphinx Bootstrap Theme

The Sphinx Bootstrap Theme is an extension for the Sphinx documentation tool, used for Python-based API documentation and static website authoring. The theme integrates Twitter Bootstrap, which is a wildly popular frontend framework. The theme GitHub project provides a demonstration site using the theme.

The theme project continues to move along, and as it has been a while since my last update post, here are some of the changes in version 0.1.6 (since 0.1.0):

Update to Bootstrap v2.3.0.
Enable configuration of the depth of the Global TOC (“Site” tab).
Add footer divider and various other styling tweaks.
Add Bootstrap styling for tables and inline code snippets

In addition, much of the underlying rendering JavaScript code has been revised. For example, Twitter Bootstrap v2.3.0 handles behavior like dropdown menu clicks in mobile much better than previous versions, enabling removal of some hacks and patches in the theme.

Theme Options

Here is the current list of options you can set for the theme in the “conf.py” configuration file, as used by the demo site in ”conf.py”:

html_theme_options = {
    # Global TOC depth for "site" navbar tab. (Default: 1)
    # Switching to -1 shows all levels.
    'globaltoc_depth': 2,

    # HTML navbar class (Default: "navbar") to attach to <div> element.
    # For black navbar, do "navbar navbar-inverse"
    'navbar_class': "navbar navbar-inverse",

    # Fix navigation bar to top of page?
    # Values: "true" (default) or "false"
    'navbar_fixed_top': "true",

    # Location of link to source.
    # Options are "nav" (default), "footer" or anything else to exclude.
    'source_link_position': "nav",
}

Sphinx Bootstrap Theme Updates - Mobile, Dropdowns, and More

2012-11-19T11:30:00Z

Sphinx Bootstrap Theme Updates!

Almost a year ago, I created the Sphinx Bootstrap Theme project to bring together two of my favorite open source technologies - Sphinx and Twitter Bootstrap. Sphinx is a widely-used Python-based authoring tool for creating static websites and API documentation. Twitter Bootstrap is a frontend JavaScript / CSS framework that offers a consistent UI experience, typography, grid systems, navigation / button / other components, and responsive features for mobile design.

Since its introduction, I now use Sphinx with the Bootstrap Theme for my open source Python projects and we use the theme for all internal technical documentation at work. Coming up on the project’s anniversary, it seemed appropriate to add some tweaks and enhancements to increase the usability and extensibility of the theme.

Improvements now available in version 0.1.0 include:

Upgraded the Bootstrap libraries to version 2.2.1.
Added new theme options to control UI appearance and behavior.
Navigation bar menus now have hierarchical sub-menus for a more intuitive experience.
Reworked the underlying layout code to support fully responsive design, allowing any site to look great on mobile devices like an iPhone or iPad.
Created a PyPI package to permit installation via pip install sphinx_bootstrap_theme.

So let’s see how everything looks and works with the new v0.1.0 theme.

Theme Options

The theme has several new options that you can set in the “conf.py” configuration file.

Bootstrap Theme Demo Site

Let’s start with the Bootstrap Theme’s own demonstration website.

The demo site has the following theme option settings in ”conf.py”:

html_theme_options = {
    # HTML navbar class (Default: "navbar") to attach to <div>.
    # For black navbar, do "navbar navbar-inverse"
    'navbar_class': "navbar navbar-inverse",

    # Fix navigation bar to top of page?
    # Values: "true" (default) or "false"
    'navbar_fixed_top': "true",

    # Location of link to source.
    # Options are "nav" (default), "footer".
    'source_link_position': "nav",
}

Notably, this gives us a dark navigation bar that is fixed to the top of the page during any scrolling.

Django Cloud Browser

Django Cloud Browser is a reusable Django application for browsing and downloading files stored in various cloud datastores (e.g., Amazon Web Services S3, Rackspace Cloud Files). The API documentation uses the Bootstrap Theme with (mostly) the default options.

The only real tweak is setting 'source_link_position': "footer" to move the documentation source code link down from the navigation bar to the bottom of each page.

Navigation Menu Dropdowns

The navigation bar menus are now hierarchically arranged into dropdowns and sub-menus for easier navigation. This looks a lot better than the previous versions (which just used hacked-up CSS whitespace padding).

Mobile UI Support

One of the most popular aspects of the Bootstrap library is its great support for responsive mobile design. These responsive features are integrated into version 0.1.0 of the theme.

The Bootstrap Theme now renders a collapsed, clickable navigation bar in smaller viewports (like a mobile phone and some tablet orientations). Viewing the theme demo site in an iPhone gives us a slightly more terse UI.

Clicking on the right navigation bar button gives us a basic first-level dropdown menu.

And we can further click through sub-menus to navigate to the desired section or page of a website.

With these and other responsive Bootstrap features, the Sphinx Bootstrap Theme now gives your API documentation or website real usability for mobile devices.

Installation

The Bootstrap Theme supports installation via download as detailed in my previous blog post and now adds full PyPI installation support as a Python package.

To get the theme from PyPI, use pip to install it:

$ pip install sphinx_bootstrap_theme

In your Sphinx “conf.py” configuration file, import the theme module at the top:

import sphinx_bootstrap_theme

Then, configure the HTML theme values in “conf.py”, using the module to get values for html_theme_path:

# Activate the theme.
html_theme = 'bootstrap'
html_theme_path = sphinx_bootstrap_theme.get_html_theme_path()

And, that’s pretty much it. The theme should now be available to Sphinx for documentation building.

Conclusion

The Bootstrap Theme brings Twitter Bootstrap to Sphinx sites and with version 0.1.0 adds even more useful Bootstrap features and options. So try it out, see if Bootstrap is right for your Sphinx website, and please pass on any feedback (issue reports, pull requests, etc.).

Shared JavaScript Code with AMD/RequireJS

2012-11-15T11:30:00Z

AMD/RequireJS

The Asynchronous Module Definition (AMD) provides a framework for predictable loading of JavaScript modules and dependencies. AMD is an extension of (and possible transport for) the CommonJS specification.

The RequireJS library implements AMD with loading, bundling and other functionality. It brings sensible imports (with define and require) to frontend JavaScript, instead of the usual array of script includes into the global namespace. Interestingly, as Node.js implements CommonJS-style require’s, there are also support modules to bring Node.js code under AMD.

Take both of things together and we have the real potential for shared JavaScript code across the browser and server. As we use this exact approach at work for our new Backbone.js frontend / Express Node.js backend web application, I created a short talk that I presented at the November 11, 2012 Node.DC meetup, entitled “Shared Code with AMD/RequireJS”.

Source Code and Demo

The source for the presentation is available on GitHub. The repository further includes all of the demonstration code I used to create a shared JavaScript library and then expose it via the frontend (as straight JavaScript) and the backend (as a REST service).

The talk and the source give a good overview and run-through of how to implement simple shared code, but to give a better sense of the final result, here is a RequireJS-compliant code module (for shuffling elements in a string) that has an underscore dependency, and works in both the browser and as a standard module in Node.js:

// Node.js hook boilerplate.
// This adds the `define` function to Node.
if (typeof define !== 'function') {
  var define = require('amdefine')(module);
}

// Define wrapper gives us `require`
define(function(require) {
  // Looks like a normal `require` for Node.js.
  // Note: Both browser and Node.js should use
  // the **same** version of libraries.
  var _ = require('underscore');

  // Slice a string into a list of items
  // and return the shuffled values.
  var shuffle = function (val) {
    val = val || "";
    return _.shuffle(val.split(/[\s,]+/));
  };

  // Export our shuffle function by returning.
  return shuffle;
});

We add some boilerplate at the top of the library for Node.js, then use the define function to make a require import function available to both the browser and Node.js, and we’re off and code sharing!

AMD, RequireJS, and shared code are topics that have a lot more depth, features, and challenges. Hopefully this provides the start to a journey into better JavaScript dependency management and more extensible full stack web application development.

Deck.js Starter

2012-11-05T10:00:00Z

Deck.js Starter

Deck.js is a modern presentation framework for creating interactive HTML-based interactive slide decks. The framework provides:

A basic boilerplate.
CSS themes.
Extensions including a menu, “go to” button, transitions, etc.

To create a presentation, simply edit the HTML files and host a static website and you have an auto-magic slide deck, available to the world.

I have now done a couple of deck.js-based presentations, and find it incredibly useful, but noticed I was re-implementing the same scaffolding enhancements. I wrapped up these common enhancements in a GitHub project, Deck.js Starter. The notable enhancements the starter kit has are:

Jade templates with Markdown support for faster slide authoring.
Executable JavaScript code snippets using CodeMirror.
Executable CoffeeScript code snippets via some specific tweaks I made to the CodeMirror deck.js extension.

Essentially, you can download and install the starter kit, and just start authoring new slides. The starting deck provides a short demo and is available at:

Deck.js Introduction

Getting up to speed deck.js is pretty straightforward: check out a tutorial / exemplary demo or dive right into the docs or wiki. After the basics are mastered, there are plenty of avenues for further exploration / additions:

Themes: Deck.js includes three by default and there are several more available as open source projects.
Extensions: Deck.js includes a menu, navigation and other controls by default. There is a fantastic open source ecosystem around deck providing all manner of additional extensions. To review a few:
- Remote control a presentation.
- Add a laser pointer to the presentation online.
- Easy “presenter” views.
- … and many more!
Tools: Several open source projects offer meta-level enhancements like markdown support, etc.

Working with Deck.js

I have used deck for two of my presentations to date. My basic workflow was to fork the repository from GitHub, then make my edits and add in extensions. Finally, I pushed the static web project to GitHub Pages for free hosting.

For a better sense of what the end products look like, here are the two live presentation decks.

Getting Started with Node.js in the Cloud

My talk to the Nova Node meetup on using Node.js in the cloud.

5 Things I Like About CoffeeScript

A simple CoffeeScript survey I presented at the Node.DC meetup.

A Starter Kit

After writing these presentations and looking to my next one, I realized I was writing the same boilerplate again, and decided to finally wrap everything up in a concise, opinionated starting point: Deck.js Starter

To get started, checkout the online demo, and then review the README. The “tl;dr” version is:

First, clone the repository:

$ git clone https://github.com/ryan-roemer/deck.js-starter.git my-presentation

Next, make sure Node.js is installed, and install all the necessary support packages.

$ cd my-presentation
$ npm install

Once you are ready to author, start the “watch” task that will automatically build ”layout.jade” and ”index.jade” into the “index.html” static page. You can then preview your slide deck at “index.html”:

$ npm run-script watch

Authoring is pretty easy. To add themes, extensions, or extra scripts, edit ”layout.jade”. For slides, edit ”index.jade” and add section.slide elements following the boilerplate as a guide. Make sure to take a look at the following possibilities:

Executable JavaScript:

Executable CoffeeScript:

After exploring the basics of the starter kit, feel free to add in any other Deck.js extensions or other useful components or tweaks. You’ll end up with an interactive, visually appealing presentation that you can easily author and host online.

As a parting thought, the Deck.js Starter was written in a couple of hours over the weekend. I do plan to enhance it as I author real presentations in the future (particularly with an eye towards adding more themes). But any feedback or enhancements are most welcome.

Responsive Web Design with Twitter Bootstrap

2012-09-30T18:37:07Z

Bootstrap

Twitter’s Bootstrap library is one of the most popular front-end frameworks, with an amazing adoption rate by various web sites, from small to large. Bootstrap provides:

Responsive Features: Enables seamless transitions between desktop, tablet, and mobile view sizes, with intuitive look-and-feel switching and custom single-view overrides.
Layout Support: Fixed and fluid grid systems for web design.
Style Components: Really nice looking form helpers, fonts, tables, etc.
UI Components: Dropdown menus, buttons, accordions, alerts, progress bars and more, with JavaScript support.

Essentially, Bootstrap makes it easy to design a responsive website (that looks good in desktop or mobile views) and add user interface components that keep with the overall user experience.

Moving Loose Bits to Bootstrap

Loose Bits (this blog) previously had a non-responsive design - the main heading bar was far too wide when viewed on a mobile phone (like my iPhone). I had been itching to make this site mobile-friendly, and finally decided to bite the bullet and integrate Bootstrap.

I downloaded the full bootstrap build from source, so that I could customize parts of the framework and only add in what I needed. As this blog is hosted as an open source project on GitHub, the source code and build system is available for checkout or download at my Loose Bits repository.

While I did customize some colors and variables for Bootstrap and spent some time inserting Bootstrap-compatible CSS classes in my HTML layout, I have to say the overall experience was quite easy, required minimal changes, and I am pleased with the final website.

The different layout effects for various viewports can be viewed simply by resizing a browser window on a desktop computer. In desktop view, I’ve retooled the main header and navigation bar to look like:

My layout uses the basic “inverse” (dark) Bootstrap theme and integrates the fantastic Font Awesome icons for each navigation bar menu item.

Loose Bits on Mobile!

Moving on to the mobile experience, the header and full navigation bars are replaced by a title-only heading with a clickable navigation menu button. Resizing a browser window down to mobile phone size (or if already on a mobile device), we get:

I’ve used the simple navigation collapse feature to have a clickable menu of all of the previous navigation bar items, so as to more effectively use the reduced real estate in a mobile view:

The rest of the website has a few other adjustments in mobile view mode, nearly all of which auto-magically happen without any further tweaks to Bootstrap configuration or JavaScript hacking.

At the end of the day, I’m pretty happy with the Bootstrap development experience and the end result for my website. Like any web framework under active development, there are still a few gotcha’s - like some loss of browser compatibility and weird JavaScript bugs (e.g., my navigation buttons sometimes disappear). But, ultimately, for anyone searching for the fastest path to a reliable and intuitive responsive web site, I would definitely recommend looking at Twitter Bootstrap.

5 Things I Like About CoffeeScript

2012-08-16T11:30:00Z

CoffeeScript

CoffeeScript is self-described as “a little language that compiles into JavaScript”. CoffeeScript implements a subset of the full JavaScript language and adds a little bit of common boilerplate code to take out some of the less savory parts of JavaScript. CoffeeScript works both in Node.js on the backend, as well as in the browser.

I gave a short survey talk at the August 15, 2012 Node.DC meetup, “5 Things I Like About CoffeeScript”.

The talk’s source is available on GitHub, and uses the awesome deck.js presentation framework with the CodeMirror plugin to enable editable and runnable code samples. I further hacked up the CodeMirror plugin to additionally make the CoffeeScript code examples executable (CodeMirror only does JavaScript by default).

Five Cool Things in CoffeeScript

For the impatient, the five topics I covered are:

Syntax
Functions
Classes
Existential Operator
Loops, Comprehensions

The talk just skims the surface of the powerful (yet slim) CoffeeScript language. For further reading / real introductory tutorials, I would suggest:

CoffeeScript Documentation: The official API.
The CoffeeScript Cookbook: Lots of “recipes” for common programming problems and projects.
Smooth Coffeescript: Interactive online eBook with live code execution and editing.

Better Data Slinging with Node.js Readable/Writable Streams and Pipes

2012-08-02T14:00:00Z

Node.js Streams

Node.js provides an extensible and fast platform for web servers, proxies, and middle-tier services. Node.js applications often utilize some transformation from one data format (e.g., a database or cloud store) to another (e.g., an HTML or JSON page).

Most folks are familiar with the callback-style of hooking together various JavaScript data components in a Node.js program. However, an often overlooked and very powerful data binding abstraction for Node.js is found in the stream class.

Streams are an abstract interface for data objects in Node.js which can be readable and/or writable. They can be hooked from one to another in a similar style to Unix pipes – in fact, the stream operation we’ll mostly focus on here is the not-coincidentally-named pipe(). Some potential advantages of stream pipes over other binding styles include:

Often much less code for the actual binding (can just call pipe()).
Streams can handle pausing / resuming of data flows. Implementing classes, however, have to implement this logic internally if supported.
Don’t have to set specific callbacks or listeners for intermediate data events – just pipe() the stream and forget it.
Avoiding buffering by processing data and re-emitting it directly to another stream (unless all of the data is required in one chunk).
Compatible with the many Node.js core modules that already implement streams, including HTTP, HTTPS, and file and process I/O.

For a brief example, we can create a download client to retrieve a web page and write it to a file as follows (ignoring error handling):

// Get Google's home page.
require('http').get("http://www.google.com/", function(response) {
  // The callback provides the response readable stream.
  // Then, we open our output text stream.
  var outStream = require('fs').createWriteStream("out.txt");

  // Pipe the input to the output, which writes the file.
  response.pipe(outStream);
});

Notice that we didn’t set any explicit ”data” listeners or buffer any of the data, even if it came in as chunks. We simply pipe()‘ed it with our two stream objects: response and outStream. The output of response is hooked to the input of outStream and we’re done.

More importantly (as we’ll get to below), we can add many more pipe() calls to do other transformations / data-slinging inline to our chained call. Ultimately, it just takes a little bit of glue code to hook together data flows in a terse and efficient manner with streams.

The Stream Interfaces

So how do we do this for our own classes and objects?

The Node.js streams documentation offers the full rundown of how to implement the interfaces, but we’ll give an abbreviated version here to get going. All of the code examples discussed in this post are available as a GitHub gist.

Readable Streams

Readable Streams must emit ”data” events whenever they have data to be read and ”end” when the data stream is finished. The implementing constructor should also set this.readable = true. The interface additionally provides a lot more implementer flexibility for things like pausing and resuming a stream, as well as resource management and cleanup.

Writable Streams

Writable Streams must implement the write() method to accept new data chunks into the stream and the end() method to instruct the stream that the data is finished. The implementing constructor should also set this.writable = true.

Putting it Together

Let’s take a look at a simple example of a custom-implemented readable and writable stream that simply passes through data – data input is simply emitted unchanged as output.

// Set both readable and writable in constructor.
var NopStream = function () {
  this.readable = true;
  this.writable = true;
};

// Inherit from base stream class.
require('util').inherits(NopStream, require('stream'));

// Extract args to `write` and emit as `data` event.
NopStream.prototype.write = function () {
  args = Array.prototype.slice.call(arguments, 0);
  this.emit.apply(this, ['data'].concat(args))
};

// Extract args to `end` and emit as `end` event.
NopStream.prototype.end = function () {
  args = Array.prototype.slice.call(arguments, 0);
  this.emit.apply(this, ['end'].concat(args))
};

This is essentially the bare minimum for a readable and writable stream class. Not too much work! And for more complicated streams, we can simply augment write/data and end/end to do whatever data transformations we want.

We can now take the web scraping example from above and add the pass-through stream in the middle with the same effect – we still get a file written to output.

// Download the same page again, but with the NOP stream
// in the middle.
require('http').get("http://www.google.com/", function(response) {
  var outStream = require('fs').createWriteStream("out.txt");

  response
    .pipe(new NopStream())
    .pipe(outStream);
});

Checking the output file (“out.txt”), we can see the download results are the same as our original example.

And in fact, we could even reuse the pass-through stream multiple times to illustrate repeated pipe() data flows (although there’s absolutely no practical sense to the following):

response
  // Wow, that's a lot of nop's!
  .pipe(new NopStream())
  .pipe(new NopStream())
  .pipe(new NopStream())
  .pipe(new NopStream())
  // OK, finally write out to file.
  .pipe(outStream);

So that’s the basics. Let’s look at creating something a tad more useful.

Let’s Upper-case Some Data!

The UpperCaseStream class takes a data source (in string or Buffer format) and converts string data into upper case letters. Not ultimately that useful or extensible, but it’s definitely a data transformation that can illustrate the ease of creation and use of a custom stream.

We mostly take our simple pass-through stream above and add a custom _transform helper method to transform the data in either a write() or end() call, then re-emit the upper-cased data in a data event.

/**
 * A simple upper-case stream converter.
 */
var UpperCaseStream = function () {
  this.readable = true;
  this.writable = true;
};

require("util").inherits(UpperCaseStream, require("stream"));

/**
 * Handle various params and upper-case string data.
 *
 * Signature can be in format of:
 *  - string, [encoding]
 *  - buffer
 *
 * Our example implementation hacks the data into a simpler
 # (string) form -- real implementations would need more.
 */
UpperCaseStream.prototype._transform = function (data) {
  // Here, we'll just shortcut to a string.
  data = data ? data.toString() : "";

  // Upper-case the string and emit data event with transformed data.
  this.emit("data", data.toUpperCase());
};

/**
 * Stream write (override).
 */
UpperCaseStream.prototype.write = function () {
  this._transform.apply(this, arguments);
};

/**
 * Stream end (override).
 */
UpperCaseStream.prototype.end = function () {
  this._transform.apply(this, arguments);
  this.emit("end");
};

To test things out, we can take an input file (“input.txt”), read it in, upper case all text, then write it out to “out.txt” using three streams.

var fs = require("fs"),
  input = fs.createReadStream("input.txt"),
  output = fs.createWriteStream("out.txt"),
  upperCase = new UpperCaseStream();

// Open our read input, uppercase it, then write out.
input
  .pipe(upperCase)
  .pipe(output);

The resulting output file is now uppercased! All in all, not that amazing, but considering the ease of our implementation, other (more useful) read/write stream applications could include:

XML to JSON conversion.
Unzipping zipped data.
Image resizing.
… any other transformations you’d like to use with your existing streams.

Conclusion

Streams provide a great means of binding together lots of data in a sane and manageable way. Beyond the core library documents, there are a lot of great stream introductions for further reference:

Getting Started with Node.js in the Cloud - Presentation

2012-03-24T22:08:21Z

Getting Started with Node.js in the Cloud

NovaNode had its first meetup event on March 20, 2012 at SpanishDict’s offices. I gave a talk titled “Getting Started with Node.js in the Cloud” covering Node.js application development and deployment to the cloud.

My talk goes through a first simple “hello world” application and builds up to a realtime chat application using Redis and websockets. We deploy the applications to Heroku and cover everything from logs and monitoring to add-on services and scaling. The talk concludes with an assessment and use cases for deploying on a Platform-as-a-Service provider like Heroku versus an Infrastructure-as-a-Service provider like Amazon Web Services.

The presentation is available live on GitHub, and uses the deck.js presentation framework. I have also posted the full demo source code to GitHub, including the chat server that we used during the meetup. Try it out!

Return-Oriented Programming - Systems, Languages, and Applications

2012-03-24T16:40:56Z

Return-Oriented Programming

Return-oriented programming is a software exploit technique to take over a program by diverting control flow without injecting any code. At UCSD, I did most of my graduate research around this specific attack, working with Erik Buchanan, Hovav Shacham, and Stefan Savage. Now three years later, the ACM Transactions of Information and System Security (TISSEC) journal has published our full article, ”Return-Oriented Programming: Systems, Languages, and Applications” in the March 2012 issue.

For a great introduction to return-oriented programming attacks, see Hovav’s Black Hat presentation, ”Return-oriented Programming: Exploitation without Code Injection”.

By way of a little background, software exploit techniques such as buffer overflows traditionally injected code into a vulnerable buffer, and then pointed control to that injected code whereby the attacker executed their own instructions. The original attack was succinctly described in Aleph One’s ”Smashing The Stack For Fun And Profit”.

Software vendors responded to injected code attacks with various defenses, one popular being “write XOR execute” (“W⊕X”), whereby memory is either writable (e.g., data structure storage) or executable (e.g., library code), but not both. The logic behind the defense was that an attacker could inject arbitrary data in writable memory but could not execute it, thus preventing the classic type of buffer overflow attack.

However, preventing code injection is not sufficient to prevent arbitrary computation. Following a long line of W⊕X exploit research, return-oriented programming works by taking existing executable code (e.g., loaded libraries) and using small chunks of it in unintended ways by hijacking control flow via compromising the stack or writable memory.

Return-Oriented Programming: Systems, Languages, and Applications

Erik and I became interested in return-oriented programming when we read Hovav’s paper that introduced the technique, ”The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86).”. Hovav focused on the x86 architecture and hypothesized that the attack would possibly be mitigated on a RISC architecture.

Fortunately, Erik and I were both TA’s for the SPARC (a RISC architecture) assembly programming course for undergraduates, and figured we could apply the attack further than originally anticipated. We were extremely lucky to get Hovav on board with our research and to later succeed in porting the attack to the SPARC architecture. We presented our findings in a paper at the 2008 ACM Conference on Computer and Communications Security (CCS). We were later invited to turn our conference paper into a full journal paper for TISSEC.

We essentially coalesced both the x86 and SPARC research to be the definitive statement on return-oriented programming, and submitted our draft in early 2009. Academic publishing does not always move at the most rapid pace, and it took about three years to finally get “Return-Oriented Programming: Systems, Languages, and Applications” published.

I think the paper is a good statement of the original development and first level extensions of the attack (and I’m obviously biased in recommending it, as I’m an author). But, it is worth pointing out that a lot has happened in the time since we submitted our draft, and the state of the art in return-oriented programming has advanced significantly past that described in our paper. Since our original research, the attack has found its way into voting machines, routers, mobile devices and other places. It is fascinating (and scary) to see the ways in which the technique has now entered the mainstream, but I’m at least happy to have (in a small way) helped expose the problem to software vendors early to give them a chance to build up defenses against return-oriented programming.

NovaNode Inaugural Meetup - Getting Started with Node.js in the Cloud

2012-03-06T15:01:00Z

Node.js in NoVA!

We recently launched the NovaNode meetup group focusing on the Node.js event-driven JavaScript framework. We have a great set of talks lined up for our inaugural meetup event on March 20, 2012 at 6:30 pm. We’re hosting things at SpanishDict’s offices in Arlington, VA near the Clarendon metro stop.

Our talks are:

“Getting Started with Node.js in the Cloud”, which I’ll present.
“Everyauth: OAuth for Busy Nerds” by Jason Bond Pratt, Co-founder of Tixelated. OAuth is the standard interface for web services integration for user authentication, and is your starting point for linking an application to Google, Facebook, Twitter, etc. Jason’s talk will take us through Everyauth’s basic architecture and integration with other data stores like MongoDB.

Getting Started with Node.js in the Cloud

Node.js provides a solid platform for scalable and responsive web applications. Development in Node.js further has a lower learning curve compared to many highly-concurrent / asynchronous frameworks as it uses JavaScript, which is very familiar territory for web developers.

The cloud complements Node.js by providing the means to deploy applications rapidly and scale up with no upfront investment costs. The “big” cloud providers such as Amazon Web Services and Rackspace easily support an infrastructures for hosting Node.js applications. Moreover, there has been a burst of Platform-as-a-Service contenders that provide even easier Node.js application hosting, including: Heroku, NodeJitsu and even Microsoft’s Azure platform.

In my talk, I’ll walk through a basic Node.js application that we develop, deploy and then scale easily in the cloud. I’ll discuss best practices and what opportunities and pitfalls lie ahead when taking your Node.js applications to the cloud.

Hope to see you at the meetup!

Async.js Presentation for Joint Node.DC / DC jQuery Meetup

2012-02-21T20:54:38Z

Async.js, a JavaScript Control Flow Library

On February 20, 2012, I gave a talk on Async.js to the joint meetup of Node.DC and DC jQuery.

Async.js is a control flow library that provides great abstractions for writing serial, parallel, and combined asynchronous code without ending up in the dreaded trenches of nested callback insanity.

My presentation is available live on GitHub, and uses Impress.js for Prezi-like functionality in the browser. As a consequence, it might not render in older browsers and IE. My talk covers the basic abstractions:

Series
Waterfalls
Parallel
Collections

and more, so check it out and feel free to ping me with any questions or comments!

Authenticated, Static Web Sites on Google App Engine

2012-01-08T14:00:00Z

Static HTML Web Site Hosting with Google App Engine

Google App Engine is a platform-as-a-service (PAAS) product that provides scalable, cloud-hosted web applications using Google’s massive engineering infrastructure. While App Engine is primarily used by web developers (e.g., programming in Python or Java), it offers three features which make it uniquely helpful for static site hosting:

Arbitrary static file handling.
Extensible authentication support.
Very inexpensive (most likely free) for hosting static content.

In this post, we’ll walk through uploading a static HTML site to App Engine, and configuring it such that it requires users to log in to via a Google Apps domain account before viewing any content. This is a common situation for organizations already using Google Apps to manage email, documents, etc. that want to host a web site without having extra configuration hassles. App Engine essentially takes care of all the authentication / user management, and you just have to upload the static web site.

The assumption throughout the rest of this post is that you already have a domain name managed by Google Apps (e.g., “example.com”). We will create an App Engine application and restrict it to users of the specific Google Apps domain, requiring a login of a user “@example.com”. Then we’ll upload the static site and verify that authentication works as expected.

Create an Authenticated App Engine Application

The first step is to create an App Engine web application. You’ll need to sign up for an App Engine account, download the App Engine Python SDK (make sure you get the Python one!), and you should read the ”getting started” documentation. Also, if given the option to “install command symlinks”, make sure you choose to do this (which will give us an appcfg.py executable in our path for use later).

Once App Engine is setup and installed, we can create the actual application. The important thing to point out here is that you must select the authentication method you want for your web site at creation time, as it cannot be changed later. (Although, you can create a new application with different authentication and delete your original application).

Take a moment to review the App Engine authentication article, as we’re basically going to follow these steps. Go to the create application web page. Fill in the following information:

Application Identifier: Choose a descriptive name for your site (e.g. “internal-docs”. There cannot be an existing matching identifier, so have some alternates handy. This will result in a domain name of “internal-docs.appspot.com” (or whatever your identifier is) for your finished website.
Application Title: A free text description of your website. Feel free to put anything in that provides a simple title for your site.
Authentication Options (Advanced): You’ll need to click the “edit” link which then gives us three options for authentication: (1) “Open to all Google Accounts users (default)”, (2) “Restricted to the following Google Apps domain:”, or (3) “(Experimental) Open to all users with an OpenID Provider”. Click the button for “Restricted to the following Google Apps domain:” and enter your Google Apps-managed domain (e.g., “example.com”). I should point out again that you already should have Google Apps set up for the domain name you are entering.

From there you can click “Create Application” and the application should be created. Make sure to keep your application identifier handy.

To enable Google Apps domain authentication for the new application, we need to follow the instructions in the App Engine authentication article. Basically, you need to open a web browser to: “http://www.google.com/a/YOUR DOMAIN” and click on the “Dashboard” tab. Go to “Service settings” and click on the “Add more services” link. In the “Other services” section, there will be a place to add an App Engine service. Type in your application identifier code here and click “Add it now”. This will hook up your specific Google Apps Domain with the App Engine service.

Configure and Upload Static Web Site

The next step is to gather your static web site files, add an application configuration, and upload all the content to the App Engine application. We will place all of our application content in a directory called “my_site” (or something else of your choosing). You are best off keeping this directory under a source control management system (e.g., git), so that you can monitor, track and revert changes to all of your files.

Configuration

We need an application configuration file call “app.yaml” in the root of our project directory. This file controls various aspects of the application, including how the application routes URLs to handlers. We’ll use a configuration that handles all static file types (including HTML), and just simply serves them.

There are various other posts out there discussing configurations for static web sites on App Engine, but the best configuration that I found was a gist by GitHub user ”darktable”. However, this configuration didn’t including authentication, so I forked the gist and added authentication attributes to produce our final app.yaml file that you should download to “my_site/app.yaml”. You can also view a basic Readme file and other information at the GitHub gist page.

Here’s a snippet of the “app.yaml” file that you’ll need to slightly modify:

application: you-app-name-here
version: 1
runtime: python
api_version: 1

default_expiration: "30d"

handlers:
- url: /(.*\.(appcache|manifest))
  mime_type: text/cache-manifest
  static_files: static/\1
  upload: static/(.*\.(appcache|manifest))
  expiration: "0m"
  login: required

# ... OTHER CONTENT SNIPPED ...

# site root
- url: /
  static_files: static/index.html
  upload: static/index.html
  expiration: "15m"
  login: required

After downloading to “my_site/app.yaml”, update the application: you-app-name-here directive with the specific App Engine application identifier you chose in the application creation section above.

Static Content

Now that we have a configuration file, create a folder named “my_site/static” which will house actual static web site. As we want to check that the authentication works first before uploading potentially sensitive information, I would recommend creating a test HTML page that just contains the content “It worked!” and adding that as “my_site/static/index.html”.

Now, we should have a project layout that looks like:

my_site/
  app.yaml
  static/
    index.html

At this point we can upload the full site to our static server using appcfg.py. Make sure that we have appcfg.py available:

$ which appcfg.py
/usr/local/bin/appcfg.py

If you don’t get an executable path back (any path is fine as long as something is returned by the which command), then review the App Engine ”getting started” documents for installation of the runtime.

Assuming we do have appcfg.py available, change directory in your terminal to the directory containing the “my_site” project folder and upload the static site with the following command:

$ appcfg.py update my_site

You will have to enter your Google credentials here. After the upload finishes, you should be able to open a web browser to: ”<your application identifier>.appspot.com”. If you are authenticated to your Google Apps domain, you should see the “It worked!” test page. If not, you should be prompted to login to your Google Apps domain. A good way to test the authentication works is to open a new Google Chrome Incognito window. It should always force a new Google Apps login if you have configured things properly. If the authentication doesn’t work quite right, review the App Engine authentication page for tips and pointers, or leave a comment below on this post.

Assuming authentication does work correctly, then you can now remove the test “index.html” file and upload your real site content to the “my_site/static” directory. Every time you change the content, make sure to re-upload the project with appcfg.py and enjoy your static web site!

Twitter Bootstrap Theme for Sphinx

2011-12-09T16:00:00Z

Bringing Twitter Bootstrap to Sphinx

Bootstrap is an open source CSS/JS framework from the folks at Twitter that provides base typography, layout (grids), forms / form inputs, tables and navigation, and has been gaining wide popularity among developers. In my personal experience with a couple of sites and projects, Bootstrap really provides a great starting point for a frontend that provides good extensibility over time.

Sphinx is the most popular Python documentation generator tool. It has fantastic Python module docstring / comment parsing in addition to a sensible static pages framework, providing a great final web site built from both inline code documentation and outside web pages.

In the process of starting up a number of new Sphinx projects, I decided that I would really like to use Bootstrap as the underlying UI framework. Fortunately, Sphinx supports custom theme plugins, so I spent a couple of days hacking, and created a Sphinx Bootstrap Theme, which is now available on GitHub.

To show all of the coolness of Bootstrap and Sphinx, I’ve put up a simple demo, which provides a skeleton Sphinx site with the project README file rendered as content. Here is a screenshot:

Installation

Sphinx themes can be installed either as a directory of files or a zip file. For simplicity’s sake, we’ll go with a pre-packaged zip file installation here.

First, set up a themes directory, which typically is something like “_themes” located in your Sphinx source directory:

$ cd /path/to/source
$ mkdir -p _themes

Next, download the most current ”bootstrap.zip” archive from GitHub. Note that older zipped theme files are available at the downloads page, with git hash suffixes.

$ cd /path/to/source/_themes
$ wget https://github.com/downloads/ryan-roemer/sphinx-bootstrap-theme/bootstrap.zip

As an alternative, you can just clone the theme repository, and place the “bootstrap” directory in your themes directory. The “bootstrap.zip” file should be the most up-to-date version, but getting the directory from source will remain the authoritative current version.

Finally, add the themes directory path and theme name to your source Sphinx “conf.py” file:

# Activate the theme.
sys.path.append(os.path.abspath('_themes'))
html_theme_path = ['_themes']
html_theme = 'bootstrap'

# Optional. Use a shorter name to conserve nav. bar space.
html_short_title = 'Demo'

Rebuild your Sphinx documentation, and you should now have the Bootstrap theme up and running! See the README for more information on customizing and hacking on the theme.

Brief Tour

The Bootstrap Sphinx theme basically rolls up the site-wide / page-level table of contents, page navigation, and search form into the main navigation top bar.

Site Navigation

The first dropdown tab is the “Site” button (expanded and highlighted below). Internally this wraps up the globaltoc template using the toctree directive.

For both the site and page navigation dropdowns, the Bootstrap theme uses JavaScript hackery to unpack the internal list structure and convert it to an ordered flat list with hierarchy displayed using extra left padding corresponding to levels. The natural levels of table of content displayed normally by Sphinx should be preserved in the dropdown menus.

Page Navigation

The second dropdown tab is the “Page” navigation for table of contents link within a single page. This wraps the localtoc template using the toc directive.

Search

The right “Search” box in the navigation top bar connects to the built-in, client-side search that Sphinx provides. Here we search for the term “bootstrap”:

… which gives us the following page matches:

Final Thoughts

The Bootstrap Theme provides an easy means for Twitter Bootstrap integration for my current Sphinx sites. But, as I’ve just put together the theme this week, it is probably still rough around the edges. Any issue reports, pull requests, and feedback are most welcome.

Sunny.js Presentation and Demo Server

2011-10-20T16:00:00Z

Sunny.js Presentation

I gave a lightning talk at last night’s Node.js Meetup introducing Sunny.js, a multi-cloud library for Node.js. For more background on Sunny, see my previous blog post to get started with the library.

Instead of the usual PowerPoint deck, I used Prezi, which I have been wanting to try out for some time. Here is the Sunny.js prezi I gave at the meetup:

Sunny.js Meetup Presentation

Although the Prezi took some time to put together (especially being my first time using the service), I was very happy with the appearance and flow of the presentation.

Simple Cloud/Web Proxy Server

As part of my presentation, I demo’ed a simple proxy service using Sunny and Node. I took the v0.0.1 HTML documentation for Sunny, and uploaded all of it to an Amazon S3 bucket, being careful to preserve paths from the original documentation files when naming S3 keys. (Actually, this wasn’t hard at all – I just used the reliable CyberDuck S3 client).

I then wrote a really simple 40-line web server. It basically takes credentials from the process environment, checks if a specified container exists, and then translates web request GET paths to blob names in the container I set up. The result is that I could serve my entire documentation site straight from by Amazon S3 bucket!

The source for the project, sunny-proxy, is available on GitHub, and is fairly straightforward to setup for yourself using the Readme instructions.

For the impatient, here’s the entire relevant code for the web server:

var http = require('http'),
  mime = require('mime'),
  sunny = require('sunny'),
  conn = sunny.Configuration.fromEnv().connection,
  ADDR = process.env.ADDRESS || "0.0.0.0",
  PORT = process.env.PORT || 2000,
  CONTAINER = process.env.SUNNY_PROXY_CONTAINER;

// Get our container and create server inside to get blobs.
conn.getContainer(CONTAINER, { validate: true })
  .on('error', function (err) {
    console.log(err);
    throw err;
  })
  .on('end', function (results) {
    // We have a valid container, so let's create the server now.
    http.createServer(function (req, res) {
      var path = req.url.replace(/\/$/, "/index.html").replace(/^\/*/, ""),
        status = 200,
        logResult = function () {
          console.log("[%s] %s", status, path); },
        stream;

      // Header based on MIME type (re-write on error).
      res.writeHead(status, { 'content-type': mime.lookup(path) });

      // Get blob and pass through error or pipe to response.
      stream = results.container.getBlob(path);
      stream.on('error', function (err) {
        status = err.statusCode || 500;
        res.writeHead(status, { 'content-type': "text/html" });
        res.end("<h1>" + status + ": " + err.message + "</h1>");
        logResult();
      });
      stream.on('end', logResult);
      stream.pipe(res);
      stream.end();
    }).listen(PORT, ADDR);
    console.log("Server running at http://%s:%s/", ADDR, PORT);
  })
  .end();

We use the magic of stream.pipe(res) to handle all of the transfer details of passing through our blob data chunks to the HTTP response.

I even took it a step further and set up a simple Heroku app to run the application. All it took was a quick review of the Node app instructions and a “Procfile” file (mine is available in the GitHub repo) and I was able to get my node proxy server up and running on the web in under a half hour! Moreover, I was pleasantly surprised at how fast the web application was running – especially considering the web server doesn’t use any caching logic or any intelligent optimizations that a real web server would.

The Sunny-based proxy server certainly isn’t anything amazing or revolutionary – it just serves blobs as if they were static files for a web server – but it does show just how easily Node.js and libraries like Sunny can glue cloud / service-based components together for fast and scalable applications.

Sunny.js, a Cloud Library for Node.js

2011-10-16T16:00:00Z

Sunny.js

Node.js provides a great environment for cloud applications and development. Node’s asynchronous design and great HTTP/REST support provide the building blocks for architecting non-blocking and scalable applications in the cloud.

There are a lot of great Node cloud clients out there for cloud datastores. Amazon Web Services S3 has: knox, node-sissy, node-s3, Node.js-Amazon-S3, and Rackspace Cloud Files has: node-cloudfiles.

After reviewing these libraries, I found a few features variously lacking that I would like in an ideal cloud datastore client:

Event-based interface.
“One-shot” requests wherever possible.
Able to create / delete arbitrary containers (buckets).
Configurable request headers, cloud options and metadata.
SSL support.

Additionally, it would be nice to have a library that worked across multiple clouds like jclouds for Java and Libcloud for Python.

With these goals in mind, I put together a basic multi-cloud datastore client called Sunny.js. Sunny initially supports Amazon Web Service’s Simple Storage Service (S3) and Google Storage for Developers (version 1). Sunny hits all the points above, and also has:

A really well-documented API and user guide. (At least I think so!)
Full unit tests for all cloud operations and providers.

Here are some basic resources to get started with Sunny, which we’ll dive into a little deeper in the subsequent sections of this post:

Sunny.js Documentation: Guides and API documentation.
- User Guide: Guide for programmers using Sunny in your applications and library.
- Developer’s Guide: Guide for developing, extending, and hacking on the Sunny.js source code.
- API: Full API.
Sunny.js GitHub Page: Source repository and issue tracker.
Sunny.js NPM Page: NPM page and history.

Getting Started

First, install sunny from npm:

$ npm install sunny

Once Sunny is installed, we need to set up some configuration information for our Node program. We can provide cloud account and option information to Sunny either via the process environment or a JavaScript object / file. For simplicity, we’ll just export some environment variables (see the user guide for more on configuration and using files).

$ export SUNNY_PROVIDER=("aws"|"google")
$ export SUNNY_ACCOUNT="ACCOUNT_NAME"
$ export SUNNY_SECRET_KEY="ACCOUNT_SECRET_KEY"
$ export SUNNY_SSL=("true"|"false")

From here, we can create our Node program, and get a live cloud datastore connection as follows:

var sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection;

Cloud Operations

Now that we have our cloud connection object (conn), we can perform all of our cloud operations on containers and blobs. For those used to Amazon Web Services parlance, a Sunny container is equivalent to an S3 “bucket”, and a Sunny blob is an S3 “key”.

Sunny cloud operations are asynchronous and event-based. The cloud methods return either a request or a stream object, that is then used to set listeners and callbacks on, before calling end() and starting the underlying real cloud network operation.

Requests

Sunny request objects are not true Node request objects, but approximate a subset of a Node HTTP request object events. The most common request events are:

“error”: The underlying real cloud operation failed.
“end”: The operation finished and we have results.
“data”: A GET request returned a data chunk.

Let’s perform a basic asynchronous operation to get a container named “sunny” from our cloud store.

var request = conn.getContainer("sunnyjs", { validate: true });
request.on('error', function (err) {
  console.log("We received error: %s", err);
});
request.on('end', function (results, meta) {
  var container = results.container;
  console.log("Retrieved container \"%s\".", container.name);
});
request.end();

which gives us the output:

Retrieved container "sunnyjs".

Breaking the code down, we call getContainer with the container name and the option validate: true. The validate option means we actually perform a cloud request to check the container exists before continuing. This is a good check to start with before launching in to other code. But, Sunny allows the programmer to choose to not validate, and wait for the first blob request to actually perform any network operation, which is faster.

The getContainer method returns a request object, which we then set our listeners on for “error” and “end” (when we have results). Finally, the call to request.end() initiates the actual network operation.

Our “end” callback takes a results parameter which contains a container method for further use with blob methods, and a meta object with information from the actual cloud call (metadata, HTTP headers, etc.) See the getContainer documentation for more options, results and uses.

The other request-based Sunny methods work in a similar fashion, and include the following:

Connection: Get a list of containers. Get/create/delete a single container.
Container: Get/create/delete a single container. Get a list of blobs. Head/delete a single blob. Get/put blob to/from a file.
Blob: Head/delete a single blob. Get/put blob to/from a file.

See the API for full method details.

Streams

Sunny returns stream objects for data-based cloud operations (PUT or GET), which are real implementations of Node streams. A GET blob method returns a Readable Stream object, and a PUT blob method returns a Writable Stream object.

Let’s take the container object we received from our successful getContainer request above, and perform a PUT blob operation with a simple string. (Note: programmatically, this would be within the “end” callback of the get container request). Data can be written with any number of write() calls and a single end() call (which starts the data transfer and ignores all subsequent writes).

var writeStream = container.putBlob("my-blob.txt");
writeStream.on('error', function (err) {
  console.log("We received error: %s", err);
});
writeStream.on('end', function (results, meta) {
  var blob = results.blob;
  console.log("Put blob \"%s\".", blob.name);
});
writeStream.write("My data string.");
writeStream.end();

which produces:

Put blob "my-blob.txt".

Now, in a later operation (say, within the putBlob “end” callback, after we know the blob was written), we can retrieve the data:

var readStream = container.getBlob("my-blob.txt");
readStream.on('error', function (err) {
  console.log("We received error: %s", err);
});
readStream.on('data', function (chunk) {
  console.log("Data chunk: \"%s\"", chunk);
});
readStream.on('end', function (results, meta) {
  var blob = results.blob;
  console.log("Get blob \"%s\".", blob.name);
});
readStream.end();

which gives us:

Data chunk: "My data string."
Get blob "my-blob.txt".

Note that we’re getting the data in raw chunks from the “data” event, which is somewhat tedious to cobble together (as strings if encoded or Buffer objects otherwise). Given that we have full read / write stream implementations, Sunny really shines with the availability of pipe()‘ing data. Essentially, you can simply connect a Sunny read / write stream to another read / write stream, call pipe() and let Node and Sunny take care of the rest.

For example, Sunny provides helper methods for getting / putting blobs to and from files. Looking at the source code, the real gist of, for example, getting a blob to a local file is the following:

var getStream = container.getBlob("name", /* ... */);
var writeStream = fs.createWriteStream(filePath, /* ... */);

// Start the pipe and call 'end()'.
getStream.pipe(writeStream);
getStream.end()

There’s a little more to it, but it is mostly this easy. Take a look at the source code for the blob convenience methods putFromFile and getToFile to see a full implementation that binds together file and cloud streams.

Node streams available for use with Sunny get / put pipe() calls include:

HTTP/HTTPS requests and responses. See my sunny-proxy project for a simple web server that proxies web requests to cloud blobs using Sunny blob get streams piped to HTTP responses.
File reads and writes – fs.ReadStream, fs.WriteStream

Cloud Headers and Metadata

Sunny abstracts common cloud provider header and metadata. For example, AWS S3 has a metadata header prefix of “x-amz-meta-“, while Google Storage has an analogous one of “x-goog-meta-“.

Sunny cloud operations return a meta object that looks like:

{
  headers: {
    /* HTTP headers. */
  },
  cloudHeaders: {
    /* Cloud-specific HTTP headers (e.g., "x-amz-"). */
  },
  metadata: {
    /* Metadata headers (e.g., "x-amz-meta-"). */
  }
}

with cloud-specific prefixes stripped out. Moreover, you can set cloud headers in the same manner by passing any of the fields above (“headers”, “cloudHeaders”, “metadata”) in the options object of a cloud request. In this manner, Sunny makes it easy to utilize metadata operations as part of your application, while remaining agnostic to your actual cloud provider. See the Sunny user guide for more information.

Error Handling

For programmatic ease, and to better abstract across cloud providers, Sunny wraps cloud-based errors with a custom class, that adds in some additional useful attributes. The CloudError class has the following members:

message: The error message (usually in XML). (from Error).
statusCode: HTTP status code, if any.

and error type methods:

isNotFound: Container / blob does not exist.
isInvalidName: Invalid name requested for container / blob.
isNotEmpty: Container cannot be deleted if blobs still exist within it.
isAlreadyOwnedByYou: Creating a container that is already owned by you.
isNotOwner: AWS and Google Storage have a global namespace, so if someone else owns the container name requested, your operations will fail.

Ultimately, Sunny “error” event listeners will get CloudError objects instead of raw Error objects on cloud method errors, which can be used to more easily programmatically react (especially since Sunny correctly handles different error codes and XML from cloud providers that correspond to the same type of error).

Putting it all Together, with a Little Help from Async.js

Now that we’ve seen how to create a Sunny connection, get a container and perform operations on blobs within the container, let’s put all of our code together in a single script. (Note: I’ve collapsed the error handling to a single utility function).

var sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection,
  errFn = function (err) { console.log(err); };

// Op 1: GET container.
var request = conn.getContainer("sunnyjs", { validate: true });
request.on('error', errFn);
request.on('end', function (results, meta) {
  var container = results.container;
  console.log("Retrieved container \"%s\".", container.name);

  // Op 2: PUT blob.
  var writeStream = container.putBlob("my-blob.txt");
  writeStream.on('error', errFn);
  writeStream.on('end', function (results, meta) {
    var blob = results.blob;
    console.log("Put blob \"%s\".", blob.name);

    // Op 3: GET blob.
    var readStream = container.getBlob("my-blob.txt");
    readStream.on('error', errFn);
    readStream.on('data', function (chunk) {
      console.log("Data chunk: \"%s\"", chunk);
    });
    readStream.on('end', function (results, meta) {
      var blob = results.blob;
      console.log("Get blob \"%s\".", blob.name);
    });
    readStream.end();
  });
  writeStream.write("My data string.");
  writeStream.end();
});
request.end();

Not too shabby! However, you’ll note that we have three serial cloud operations, each of which have to complete before the next starts, so we end up repeatedly nesting callbacks in “end” event listeners. Even with a mere three operations, the nesting makes the code hard to understand at first blush, and an indentation nightmare. Wouldn’t it be better if we could code our operations to match the general outline of what we’re trying to do? That is:

Get a container.
Once we have a container, put data to a blob.
Once we have put data to the blob, get the data back.

Fortunately, there are many asynchronous helper libraries for Node to help in just this situation. I like Async.js, which from the project page “provides around 20 functions that include the usual ‘functional’ suspects (map, reduce, filter, forEach…) as well as some common patterns for asynchronous control flow (parallel, series, waterfall…).”

In our simple three-cloud operation script, we just need async.series to serialize our operations for easier control flow. I am skipping over a lot of details about using Async.js, such as callbacks at the end of functions or errors, and passing state across serialized functions, but you can probably intuit most of how things are working here:

var async = require("async"),
  sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection,
  container,
  blob;

async.series([
  // Op 1: GET container.
  function (callback) {
    var request = conn.getContainer("sunnyjs", { validate: true });
    request.on('error', callback);
    request.on('end', function (results, meta) {
      container = results.container;
      console.log("Retrieved container \"%s\".", container.name);
      callback(null);  // Signal function "done" to async.series.
    });
    request.end();
  },

  // Op 2: PUT blob.
  function (callback) {
    var writeStream = container.putBlob("my-blob.txt");
    writeStream.on('error', callback);
    writeStream.on('end', function (results, meta) {
      blob = results.blob;
      console.log("Put blob \"%s\".", blob.name);
      callback(null);  // Signal function "done" to async.series.
    });
    writeStream.write("My data string.");
    writeStream.end();
  },

  // Op 3: GET blob.
  function (callback) {
    var readStream = container.getBlob("my-blob.txt");
    readStream.on('error', callback);
    readStream.on('data', function (chunk) {
      console.log("Data chunk: \"%s\"", chunk);
    });
    readStream.on('end', function (results, meta) {
      blob = results.blob;
      console.log("Get blob \"%s\".", blob.name);
      callback(null);  // Signal function "done" to async.series.
    });
    readStream.end();
  }
], function (err, result) {
  if (err) {
    console.log(err);
  } else {
    console.log("async.series is done!");
  }
});

which gives us our expected output:

Retrieved container "sunnyjs".
Put blob "my-blob.txt".
Data chunk: "My data string."
Get blob "my-blob.txt".
async.series is done!

Our code now appears in a serialized, contained order that matches the conceptual execution of the actual cloud operations, instead of a lot of callback nesting.

For more on developing with Async.js, please see the excellent Async.js Readme / tutorial. It is an investment well worth your time. Sunny internally uses Async.js extensively for unit tests. Explore the source tests directory for a lot of good use cases (parallel and serial).

Have Fun in the Sun

That wraps up our introduction to Sunny.js. Hopefully the library provides a useful interface for your cloud programming needs.

The project is still very much in an early development state. See the Sunny “to do” list to get a better idea of what features and fixes are coming down the pipeline. The big ticket items I would really like to see added are:

More cloud providers, notable Rackspace Cloud Files and OpenStack Storage.
A basic retry function wrapper to handle expected, occasional cloud operation failures and throttling.
Advanced cloud operation support for ACL’s, policies, etc.

Help, bug reports, and pull requests for the project are most welcome. Please review the developer’s guide, and contact me with any questions or comments.

Pivot Faceting (Decision Trees) in Solr 1.4.

2011-09-20T16:00:00Z

Solr Pivot Facets

Solr faceting breaks down searches for terms, phrases, and fields in the Solr into aggregated counts by matched fields or queries. Facets are a great way to “preview” further searches, as well as a powerful aggregation tool in their own right.

Before Solr 4.0, facets were only available at one level, meaning something like “counts for field ‘foo’” for a given query. Solr 4.0 introduced pivot facets (also called decision trees) which enable facet queries to return “counts for field ‘foo’ for each different field ‘bar’” – a multi-level facet across separate Solr fields.

Decision trees come up a lot, and at work, we need results along multiple axes – typically in our case “field/query by year” for a time series. However, we use Solr 1.4.1 and are unlikely to migrate to Solr 4.0 in the meantime. Our existing approach was to simply query for the top “n” fields for a first query, then perform a second-level facet query by year for each field result. So, for the top 20 results, we would perform 1 + 20 queries – clearly not optimal, when we’re trying to get this done in the context of a blocking HTTP request in our underlying web application.

Hoping to get something better than our 1 + n separate queries approach, I began researching the somewhat more obscure facet features present in Solr 1.4.1. And after some investigation, experimentation and a good amount of hackery, I was able to come up with a “faux” pivot facet scheme that mostly approximates true pivot faceting using Solr 1.4.1.

We’ll start by examining some real pivot facets in Solr 4.0, then look at the components and full technique for simulated pivot facets in Solr 1.4.1.

Pivot Faceting in Solr 4.0

Pivot facets were added to Solr in SOLR-792. A good introductory article is available on the Solr.pl site. To see the basic operation in action, let’s just use the “example” setup that comes with the Solr 4.0 distribution (located at “solr_4.0_path/solr/example”).

Let’s start the Solr process:

# Start Solr as non-daemon.
$ cd solr_4.0_path/solr/example
$ java -jar start.jar

Next, we want to upload a series of documents. We’ll take the provided “exampledocs/books.csv” file, tweak it slightly and update via a CSV handler. The CSV format is: first line is field names, other lines are data. Note that I have changed some field names in the original sample “exampledocs/books.csv” file. The following should be written out to a new file, which I am calling “sample_books.csv”.

Note that we use _s fields for simplicity, forcing string fields for what would ordinarily be text fields – Solr facets only return results on indexed, not stored terms, and string fields are identical for both. In a real deployment, you would use copyField directives to copy text fields to string fields for faceting.

# Create CSV file.
$ vim sample_books.csv
id,cat_s,name_s,price_f,inStock_b,author_s,series_s,sequence_i,genre_s
0553573403,book,A Game of Thrones,7.99,true,George R.R. Martin,A Song of Ice and Fire,1,fantasy
0553579908,book,A Clash of Kings,7.99,true,George R.R. Martin,A Song of Ice and Fire,2,fantasy
055357342X,book,A Storm of Swords,7.99,true,George R.R. Martin,A Song of Ice and Fire,3,fantasy
0553293354,book,Foundation,7.99,true,Isaac Asimov,Foundation Novels,1,scifi
0812521390,book,The Black Company,6.99,false,Glen Cook,The Chronicles of The Black Company,1,fantasy
0812550706,book,Enders Game,6.99,true,Orson Scott Card,Ender,1,scifi
0441385532,book,Jhereg,7.95,false,Steven Brust,Vlad Taltos,1,fantasy
0380014300,book,Nine Princes In Amber,6.99,true,Roger Zelazny,the Chronicles of Amber,1,fantasy
0805080481,book,The Book of Three,5.99,true,Lloyd Alexander,The Chronicles of Prydain,1,fantasy
080508049X,book,The Black Cauldron,5.99,true,Lloyd Alexander,The Chronicles of Prydain,2,fantasy

We’ll upload this file to Solr using curl:

# Upload to Solr.
$ curl http://localhost:8983/solr/update/csv \
  --data-binary @sample_books.csv \
  -H 'Content-type:text/plain; charset=utf-8'

# Commit the upload.
$ curl http://localhost:8983/solr/update \
  --data-binary '<commit/>' \
  -H 'Content-type:application/xml'

You should now be able to query the 10 sample documents at: http://localhost:8983/solr/admin/form.jsp.

Now that we have some documents to work with, let’s do a simple pivot query on price by genre:

# Pivot query.
$ curl http://localhost:8983/solr/select --data indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.pivot=price_f,genre_s

(Note that I’ve added line breaks and escapes to show the parameters more clearly).

This gives us decision tree results for the facet_pivot field:

{
  /* ... */,
  "facet_counts":{
    /* ... */,
    "facet_pivot":{
      "price_f,genre_s":[{
          "field":"price_f",
          "value":5.99,
          "count":2,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":2}]},
        {
          "field":"price_f",
          "value":6.99,
          "count":3,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":2},
            {
              "field":"genre_s",
              "value":"scifi",
              "count":1}]},
        {
          "field":"price_f",
          "value":7.95,
          "count":1,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":1}]},
        {
          "field":"price_f",
          "value":7.99,
          "count":4,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":3},
            {
              "field":"genre_s",
              "value":"scifi",
              "count":1}]}]}}}

Nice intuitive results, for a fairly straightforward facet query. However, now to the bigger question – can we approximate this in Solr 1.4.1, which doesn’t have the facet.pivot query option?

Pivot Faceting in Solr 1.4.1

Solr 1.4.1 has much more limited facet support than 4.0. The building blocks that we will use to cobble together a “faux” pivot query are:

facet.field: The normal facet field option.
facet.query: The normal facet query option.
fq: Basic field queries (for restrictions).
Local Params: We use a couple of Solr local parameters.
- tag: Tags a fq with an arbitrary name.
- key: Tags a facet field an arbitrary name (instead of field name).
- ex: Excludes tagged fq’s from being operative on a given facet field/query.

Note that either facet fields or facet queries can be used with this technique – I’ll only show fields, but everything applies equally to queries.

Setup

At this point, you should take a Solr 1.4.1 distribution and set it up exactly as we did above for Solr 4.0 and upload our simple 10-document CSV file to the running server. For simplicity here (and to keep my head on straight), I ended up running my Solr 1.4.1 server on port 8984, so that I could also keep the Solr 4.0 server running on port 8983. Here’s what I did:

# Start Solr as non-daemon.
$ cd solr_1.4.1_path/solr/example
$ java -Djetty.port=8984 -jar start.jar

# (Copy sample_books.csv)
# Upload to Solr.
$ curl http://localhost:8984/solr/update/csv \
  --data-binary @sample_books.csv \
  -H 'Content-type:text/plain; charset=utf-8'

# Commit the upload.
$ curl http://localhost:8984/solr/update \
  --data-binary '<commit/>' \
  -H 'Content-type:application/xml'

From here on, it is assumed you now have a populated Solr 1.4.1 server running on port 8984 (switch addresses / ports as appropriate for your actual setup).

Excluding Restrictions from Facets

The starting point for our pivot facets is excluding certain query restrictions for facets. A basic example is provided for tagging and excluding facets on the Solr wiki.

Let’s do a simple facet query on prices with a restriction of genre_s:scifi:

# Restricted facet query.
$ curl http://localhost:8984/solr/select --data \
\indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&fq=genre_s:scifi\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field=price_f

Looking to our results in facet_fields, we see that we only have 2 hits (numFound), and the facet counts also add up to 2 (which represent our 2 SciFi books).

{
  /* ... */,
  "response":{"numFound":2, /* ... */},
  "facet_counts":{
    /* ... */,
    "facet_fields":{
      "price_f":[
        "6.99",1,
        "7.99",1]},
    /* ... */,
}

For situations like a drill down, Solr developers often want to run a basic query with full restrictions for the returned records, but get more information for facets. In this case, Solr allows tagging of fq’s, and keys / excludes on facets to conditionally remove fq’s from a given facet only.

So, let’s tag our fq as “SCIFI_FQ” and exclude it from our facet counts with ex, and then rename the facet results to “PRICE_KEY” using the key option:

# Tag fq and exclude only from the field facet.
$ curl http://localhost:8984/solr/select --data \
\indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&fq={\!tag=SCIFI_FQ}genre_s:scifi\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field={\!key=PRICE_KEY\ ex=SCIFI_FQ}price_f

Note that I have to escape the exclamation points and other characters for a command line example here. Now, let’s look at the results:

{
  /* ... */,
  "response":{"numFound":2, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "PRICE_KEY":[
        "5.99",2,
        "6.99",3,
        "7.95",1,
        "7.99",4]},
    /* ... */,
}

We can first see that the exclusion of the tagged “SCIFI_FQ” field query did not affect the overall numFound, which is still 2. However, for the facet field we applied the exclusion to, we now have facet results for records in the whole set (which is the effective query after the exclusion). Finally, our facet field has been renamed “PRICE_KEY” instead of the field name (“price_f”).

Constructing a Pivot Query

With the basic tag/key/exclude technique in mind, let’s now return to our original goal – create a pivot query on price by genre using Solr 1.4.1. We will do this by performing two queries:

Perform a facet query for the top price results ordered by index.
Create fq tagged exclusions for each facet result, then create multiple keyed facets on genre to give us each of our decision tree “leaf” results.

The first query is a very basic facet field query:

# First level facet field query.
$ curl http://localhost:8984/solr/select --data indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field=price_f

Which gives us four individual facet results: “5.99”, “6.99”, “7.95”, “7.99”.

{
  /* ... */,
  "response":{"numFound":10, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "price_f":[
        "5.99",2,
        "6.99",3,
        "7.95",1,
        "7.99",4]},
    /* ... */,
}

We take each of those results and create specific fq tagged restrictions:

fq={!tag=FQ5.99}price_f:5.99
fq={!tag=FQ6.99}price_f:6.99
fq={!tag=FQ7.95}price_f:7.95
fq={!tag=FQ7.99}price_f:7.99

Each excludes one of the components we’ll want facet results for our next level field (genre) on. To then get the pivot facet result for each of our four facets, we will exclude all the fq’s above except the matching one for the facet. Translating into facet parameters, this is:

facet.field={!key=5.99_GENRE ex=FQ6.99,FQ7.95,FQ7.99}genre_s
facet.field={!key=6.99_GENRE ex=FQ5.99,FQ7.95,FQ7.99}genre_s
facet.field={!key=7.95_GENRE ex=FQ5.99,FQ6.99,FQ7.99}genre_s
facet.field={!key=7.99_GENRE ex=FQ5.99,FQ6.99,FQ7.95}genre_s

The key is that we can specify multiple exclusions using a comma. Thus, looking to the facet key “5.99_GENRE”, we exclude all the fq restrictions except “FQ5.99”, which means that the facet results for that facet field key will be the facet counts for ”fq=price_f:5.99” only. It’s kind of a twisted-double-negative logic, but it all works out.

Let’s put everything into our second-level query now:

# Second level pivot query.
$ curl http://localhost:8984/solr/select --data \
indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&fq={\!tag=FQ5.99}price_f:5.99\
\&fq={\!tag=FQ6.99}price_f:6.99\
\&fq={\!tag=FQ7.95}price_f:7.95\
\&fq={\!tag=FQ7.99}price_f:7.99\
\&facet.field={\!key=5.99_GENRE\ ex=FQ6.99\,FQ7.95\,FQ7.99}genre_s\
\&facet.field={\!key=6.99_GENRE\ ex=FQ5.99\,FQ7.95\,FQ7.99}genre_s\
\&facet.field={\!key=7.95_GENRE\ ex=FQ5.99\,FQ6.99\,FQ7.99}genre_s\
\&facet.field={\!key=7.99_GENRE\ ex=FQ5.99\,FQ6.99\,FQ7.95}genre_s

Which gives us the “leaves” of the decision tree with our result keys: “5.99_GENRE”, “6.99_GENRE”, “7.95_GENRE”, and “7.99_GENRE”.

{
  /* ... */,
  "response":{"numFound":10, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "5.99_GENRE":[
        "fantasy",2],
      "6.99_GENRE":[
        "fantasy",2,
        "scifi",1],
      "7.95_GENRE":[
        "fantasy",1],
      "7.99_GENRE":[
        "fantasy",3,
        "scifi",1]},
    /* ... */,
}

Looking at our original Solr 4.0 pivot query, we can cobble together our two Solr 1.4.1 queries to get an equivalent result. In the end, both produce the following decision tree for price by genre:

5.99: 2
- fantasy: 2
6.99: 3
- fantasy: 2
- scifi: 1
7.95: 1
- fantasy: 1
7.99: 4
- fantasy: 3
- scifi: 1

Victory!

Discussion and Practical Implications

Our “price by genre” example is a bit simplistic in that we can mostly get the same results with two standard Solr 1.4.1 facet field queries. But, the faux pivot facet technique really shines for a “foo by bar”-type query where there are large numbers of first (“foo”) level facet results. Say, the first level has 10 results, this would mean 11 queries (one for the top 10 “foo“‘s, then one each for the 10 second-level “bar“‘s for each “foo”). The faux pivot facet technique cuts this down to 2 queries total.

The method is generally applicable too. Although our examples here only use facet fields, the technique equally works for facet queries. And distributed search supports the approach as well.

Looking further, the technique can be applied to additional decision tree levels. In the Solr 4.0 world, this simply means adding another field like facet.pivot=price_f,genre_s,inStock_b to get further breakdowns for the “in stock” boolean field. For Solr 1.4.1, we would do a third query, with permutations of our previous tagged fq’s, as well as second-level fq’s. Then, we would have third-level keyed facet fields like: “6.99_fantasy_INSTOCK”, “6.99_scifi_INSTOCK”, etc. At this level, it certainly wouldn’t be pretty and would result in a beastly query, but shows that the technique only adds 1 more actual Solr query for each additional level in the faux decision tree.

However, on the topic of query complexity, it is fair to point out that this type of query hackery really should be done programmatically to ensure correctness, and definitely not via the manual queries I provided above using curl. It’s tough keeping track of just the 4 first-level pivots in our example above, let alone a larger first level group, or more than 2 levels deep of pivots. Another benefit is that you can collapse your tag / key names to integer or other simple keys, and then have your program match things up later for the final assembled decision tree result.

As a final performance note, the faux pivot facet approach doesn’t really lighten the Solr server load, it just collapses what would otherwise be multiple queries into one query.

Conclusion

Reflecting on the above method, pivot facets are possible in Solr 1.4.1 at a cost of n separate queries, where n is the number of levels in the decision tree. So, if reducing the number of round trips between a web application and Solr is the goal, and you need pivot facets in a pre-4.0 Solr, this may well be the ticket.

How I Learned to Stop Worrying and Love the Static Blog

2011-09-16T16:00:00Z

Farewell Blogger…

Blogger provides an easy, integrated blogging service, but I have not been posting with anywhere near the frequency I had hoped for, and have recently started thinking of the reasons why this is. The list I cam up with includes:

Page load times: To get syntax highlighting, a nice theme, etc., my posts have an incredible amount of download cruft.
Editing Interface: While blogger’s WYSIWYG editor is fairly intuitive and the HTML editor is the appropriate power tool, I don’t like the fact I have to (1) be online, and (2) find a lot of situations where I want something in between a WYSIWYG and raw HTML.
Offline Editing: Expanding on the above point, I travel a lot, and it would be great to be able to compose and view posts offline (like on a plane).
Versioning/Backups: Blogger doesn’t allow for easily versioning of posts, which would be nice.

I essentially run a programming blog, and as such, I don’t need a lot of frills, bells or whistles. What I really want is text-based, powerful and configurable blog engine. Enter Jekyll.

Hello Jekyll and GitHub!

After a decent amount of research, I settled on Jekyll. Jekyll is a static website generator, written in Ruby, supporting Markdown, Liquid templates, and custom Liquid extensions.

There are many stories of folks moving successfully to Jekyll, and the feature set really hit my pain points:

All Text: All files are text, and either configuration, markdown, template language, or whatnot.
Versioned: It’s all source, so place your source under Git, and you’re ready to go.
Offline: The whole site can be generated or locally served without and internet connection.
Markdown: There are other pre-processing options, but I just went with Markdown, and it’s really nice and easy to write posts now without jumping back and forth from a WYSIWYG editor to straight HTML. I stay in Markdown and everything (mostly) ends up looking correct.
Syntax Highlighting: Jekyll uses Pygments for source code highlighting, which is mostly pre-processed, and not an after-the-fact JavaScript processing step (which had previously been slowing down my Blogger site).
GitHub Support: GitHub’s default document generator is Jekyll, and (separately) has full website support with CNAMEs. This provides an easy means of both site storage / versioning as well as the actual serving.

However, this is not to say that a Jekyll site is everyone’s cup of tea. There are some drawbacks to this approach:

Static Site: The generated site is static, which means no database or dynamic pages for things like comments, etc. All dynamic content has to come in via JavaScript / JavaScript-based services (like Disqus for comments). And, you have to implement a manual RSS feed, etc.
Manual CSS, HTML, etc.: There are no built-in themes, etc., because you are crafting a site from scratch. Really, this isn’t too bad because other Jekyll-site authors often publicly version their source with permissive / public domain licenses at GitHub, so you can pull source for examples. Also, there are a lot of starter kits for static CSS / HTML sites. For me, it’s really a nice opportunity to get re-acquainted with site design.

Packing Up and Moving Out

Posts

The first step to migrating my Blogger site was to simply download all of the posts in a Jekyll-friendly format. I used a very simple script that a good job of downloading basic HTML from the Blogger feed URL.

I then took the HTML pages and further converted them to Markdown using the handy html2text script. From there, I had to manually edit the gremlins left in the conversion process, including redoing anchor tags, removing Blogger-specific extras, and getting local images into my source repository static media directory.

Jekyll has its own default (but customizable) format for blog post names. I like it, but it is slightly different than Blogger, which all my existing posts already use. Fortunately, each Jekyll post has a permalink attribute allowing the post to escape the normal URL processing rules and use a one-time path instead.

Comments

After getting my posts ported over, the next step was to get my comments out of Blogger and in to something that would work with the static Jekyll web site. Enter Disqus, which is inserted via JavaScript, and this is usable for my new static website.

Making things easier, Disqus has Blogger support. So, I first switched my Blogger site comments to Disqus, then used the Disqus blogger import tool, available at:

http://<DISQUS SHORT NAME>.disqus.com/admin/tools/import/platform/blogger/

I followed the instructions and started the Blogger comment import. I expected the import to finish quickly, as I only have a handful of comments at the time of the migration, but it ended up taking 6 or so hours. Searching the web, there are reports of the Disqus comment import taking up to 2-3 days, so the lesson is: be patient. Oh, and don’t get freaked out when your comments disappear entirely from Blogger in the meantime.

Once the import finished, I looked at my Blogger site and saw that all of the comments had successfully been transitioned over to Disqus. I then set up Disqus support on my post template using the Disqus ”universal” installation guidelines.

Before fully transitioning over to my new Jekyll/GitHub site, I wanted to make sure that everything was good to go with Disqus. Fortunately, Disqus allows you to specify a “developer mode” to have comments pulled down, even when running a localhost site. With these configuration variables in the Disqus JavaScript block, I can view the comments for this post on my laptop at localhost:

// Live page URL.
var disqus_url = 'http://loose-bits.com/2011/09/16/moving-to-jekyll.html';

// Developer mode for localhost.
var disqus_developer = 1;

RSS Feed

You have to specify your own “atom.xml” file for RSS feeds. Fortunately, there are many examples from other Jekyll sites, including Tom Preston-Werner’s site, as well as a good Google Groups thread.

Here is my finished atom.xml for this site for reference.

Post Excerpts

My Blogger site displays a few recents post excerpts on the index page, which I liked. However, there is no native “excerpt” functionality in Jekyll at the moment. Fortunately, a very clever hack enabled me to add post excerpts of my most recent couple of articles to my new index page.

And, We’re Up!

And that finishes my journey from Blogger to Jekyll and GitHub. As a final commentary Blogger does a great many things right and is a solid platform, but my use cases favor power and extensibility over many of the conveniences that Blogger provides.

Although it took some time to put my new site together, everything feels “right” to me. Even writing this first post for the new site has been considerably easier – I’ve been both online and offline composing the post in TextMate (which has Markdown syntax highlighting as well as spell check), and it has never been as easy or fast to write up posts, so here’s to hoping for a more consistent stream of posts in the future!

ConstantDict, Yet Another Python Enumeration Pattern

2011-06-13T13:30:00Z

Introduction

The Python language does not natively provide enumerations. Rejected PEP 354 proposed a string-constructor-based enumeration, later provided as the separate enum package. The web is rife with various cookbook recipes and ad-hoc solutions – see, e.g., ActiveState and Stack Overflow.

Here is ConstantDict, one of my preferred enumeration patterns.

`ConstantDict`

This pattern (like most Python enumeration recipes) is quite simple and straightforward, so I’ll actually start with the enumeration class, and then discuss the merits / drawbacks after.

The enumeration class is as follows:

class ConstantDict(object):
    """An enumeration class."""
    _dict = None

    @classmethod
    def dict(cls):
        """Dictionary of all upper-case constants."""
        if cls._dict is None:
            val = lambda x: getattr(cls, x)
            cls._dict = dict(((c, val(c)) for c in dir(cls)
                             if c == c.upper()))
        return cls._dict

    def __contains__(self, value):
        return value in self.dict().values()

    def __iter__(self):
        for value in self.dict().values():
            yield value

Essentially, we create an internal dictionary of all class variables that are upper-cased (well, and methods too), and wrap up a couple of instance built-in methods. Implementing an enumeration with the pattern is pretty straightforward. Here is a “days of the week” enumeration contrived by multiply inheriting two base enumeration classes:

class Weekdays(ConstantDict):
    """Weekday days."""
    MONDAY = 'Monday'
    TUESDAY = 'Tuesday'
    WEDNESDAY = 'Wednesday'
    THURSDAY = 'Thursday'
    FRIDAY = 'Friday'

class Weekend(ConstantDict):
    """Weekend days."""
    SATURDAY = 'Saturday'
    SUNDAY = 'Sunday'

class DaysOfTheWeek(Weekdays, Weekend):
    """Days of the week."""
    pass

And a simple use case example:

# Get enumeration instance.
DAYS = DaysOfTheWeek()

print(DAYS.MONDAY)        # "Monday"
print('Saturday' in DAYS) # True
print('Notaday' in DAYS)  # False
print([x for x in DAYS])  # ["Monday", "Tuesday", ...]

Discussion

The ConstantDict pattern shines in a couple of key areas:

It has a terse, clean, and extensible structure. The enumerations are inheritable, and even support multiple inheritance (as the above examples shows). Also, the enumerations can be enhanced with helper methods, etc., because they are real classes.
The structure allows arbitrary enumeration values. In my example I used strings, but anything (even including functions) are valid values.
The enumeration members are real class variables and not strings. For me this is a key point over a pure string-based solution, as I want to be able to run pylint and have it throw errors on invalid enumeration names.

However, ConstantDict probably isn’t the best choice for many situations. Some of the drawbacks of the pattern include:

Enumeration members are unordered. Because everything is shoved into a dictionary, there are no order guarantees.
Despite being called ConstantDict, the enumeration values aren’t actually enforced constants.
Enumeration values must be manually specified and are not guaranteed to be unique. As noted above arbitrary (including identical) enumeration values are allowed. However, the pattern could be easily extended to raise an exception on detection of duplicate values.
An enumeration class cannot have other upper-cased class variables or members. This should not be too big of a deal as the enumeration class is ideally used just as a small constant holder and not a multi-purpose class.
The caller has to instantiate an instance of the ConstantDict class. This is necessary for the __contains__ and __iter__ convenience method. In the past, I have used a different version of the ConstantDict class that is used only at the class variable / method level.

There are certainly other tweaks and enhancements that can make the ConstantDict pattern better for a given situation. But, in its most basic form as described above, it provides an easy, extensible thin-wrapper enumeration.

Extending Django Settings with Derived Attributes and Methods

2011-04-20T05:07:00Z

Django settings construction and configuration is fairly straightforward. Configure Django and installed application settings, maybe add your own project-specific settings, and let Django take care of the rest. As long as you import settings from django.conf all the options and defaults are properly resolved into the ultimate settings object.

However, every so often I run across a situation where I wish the Django settings could do just a little bit more. Rather than try and wade into the depths of Django runtime configuration and risk mucking things up, the approach I take is to use a thin wrapper class around the already configured settings and pass through all calls directly.

What we essentially want is “pass through” wrapper class like:

from django.conf import settings as _settings

class Settings(object):
    """Thin settings wrapper."""

    def __init__(self, default_settings):
        """Initializer."""
        self.default_settings = default_settings

    def __getattr__(self, name):
        """Get setting."""
        return getattr(self.default_settings, name)

settings = Settings(_settings)

Fortunately, Django already has a slightly more complex version of this in the django.conf.UserSettingsHolder class, so we’ll use that instead. I usually have a common application in my Django projects and insert the settings wrapper as the common.settings or common.conf module.

From there, we can add any number of “extra” settings variables or derived methods. For instance, if we wanted to add a settings method to tell if our database was SQLite or PostgreSQL using knowledge of the driver names, we could implement some fairly simple methods:

def is_sqlite(self):
    """Return ``True`` if default database is SQLite."""
    # Driver: 'sqlite3'
    return 'sqlite3' in self.DATABASES['default']['ENGINE']

def is_postgres(self):
    """Return ``True`` if default database is PostgreSQL."""
    # Drivers: 'postgresql_psycopg2', 'postgresql'
    return 'postgresql' in self.DATABASES['default']['ENGINE']

While perhaps too simplistic and brittle for production use, the above example illustrates how easy it is to add extra dynamic checks. To use the above methods if very easy – just import settings from our module (here common.settings instead of django.conf).

from common.settings import settings

print("Using SQLite?: %s" % settings.is_sqlite())
print("Using Postgres?: %s" % settings.is_postgres())

which outputs:

Using SQLite?: True
Using Postgres?: False

for a project configured with 'ENGINE': 'django.db.backends.sqlite3'.

In a typical Django project there are several file paths that are usually derived and hacked into the settings file – the Django project root directory, the Django admin media directory, and sometimes the path to the settings file itself. Here are methods for our wrapped settings class to intuit these paths:

def settings_path(self):
    """Return path to project settings."""
    from os import path
    from django.utils import importlib

    # Have to use real settings directly because UserSettingsHolder
    # sets SETTINGS_MODULE to ``None`` as a class variable.
    mod = importlib.import_module(
      self.default_settings.SETTINGS_MODULE)

    # Split off the extension to replace ".pyc" with ".py"
    return path.splitext(path.abspath(mod.__file__))[0] + ".py"

def project_root_path(self):
    """Return path to project root."""
    from os import path

    # We "cheat" here to find project root because we have know
    # this file is ``PROJECT_ROOT/common/settings.py``.
    return path.abspath(path.join(path.dirname(__file__), ".."))

def admin_media_path(self):
    """Return path to Django admin media directory."""
    from os import path
    from django.contrib import admin

    return path.abspath(
      path.join(path.dirname(admin.__file__), "media"))

And we can view our results with:

from common.settings import settings

print("Settings file path: %s" % settings.settings_path())
print("Project root directory path: %s" %
      settings.project_root_path())
print("Admin media directory path: %s" %
      settings.admin_media_path())

which outputs:

Settings file path: /Users/ryan/demo/settings.py
Project root directory path: /Users/ryan/demo
Admin media directory path: /Users/ryan/env/demo/lib/python2.6/site-packages/django/contrib/admin/media

Putting a full file together with our simple extension methods. In our case, the finished “common/settings.py” file will be:

from os import path
from django.contrib import admin
from django.conf import settings as _settings, UserSettingsHolder
from django.utils import importlib

class CustomSettings(UserSettingsHolder):
    """Thin settings wrapper.

    .. note:: "Real" settings object is available as
      ``.default_settings`` member.
    """

    def is_sqlite(self):
        """Return ``True`` if default database is SQLite."""
        # Driver: 'sqlite3'
        return 'sqlite3' in self.DATABASES['default']['ENGINE']

    def is_postgres(self):
        """Return ``True`` if default database is PostgreSQL."""
        # Drivers: 'postgresql_psycopg2', 'postgresql'
        return 'postgresql' in self.DATABASES['default']['ENGINE']

    def settings_path(self):
        """Return path to project settings."""
        mod = importlib.import_module(
          self.default_settings.SETTINGS_MODULE)

        # Split off the extension to replace ".pyc" with ".py"
        return path.splitext(path.abspath(mod.__file__))[0] + ".py"

    def project_root_path(self):
        """Return path to project root."""
        return path.abspath(
          path.join(path.dirname(__file__), ".."))

    def admin_media_path(self):
        """Return path to Django admin media directory."""
        return path.abspath(
          path.join(path.dirname(admin.__file__), "media"))

settings = CustomSettings(_settings)

While these contrived example method extensions are solving problems perhaps best solved elsewhere (like in the actual project settings file), they serve to illustrate a simple means of how to extend the Django settings module easily and without interfering with the runtime settings configuration process.

Django Cloud Browser

2011-03-10T05:44:00Z

Introduction

I recently pushed the v.0.1.2 release of my new Django application, django- cloud-browser, to PyPi. Cloud Browser is a simple web-based file browser for cloud-based datastores, which so far includes Amazon S3, Rackspace Cloud Files (including OpenStack) and local file system. The application can expose read-only cloud files to users and/or administrators with configurations for both normal and administrative deployments.

The basic reference points to get started include:

Background

At work, we have some containers with very large numbers of objects – over 5 million and counting. While objects ultimately are named in a flat namespace, we divide up the namespace with slashes (e.g., “path/to/cloud/object/file.txt”). Nearly all cloud providers (Amazon S3, Microsoft Azure, and Rackspace) include functionality in their object APIs to return results grouped around hierarchical “implied” or “pseudo-” directory objects based on a separator (like the slash that we use).

Despite the fact that Rackspace’s underlying REST and Python APIs both support implying directories from a delimiter like a slash, Rackspace’s own management console will only display objects using a completely flat namespace. As an example, I have an old side project where I stored US patent data in XML form in both Rackspace Cloud Files and Amazon S3, and broke up each object with a slash-delimited path. (I have obscured my bucket / container names in all of the screen shots in this post). Let’s view the objects in the Rackspace container using their web interface:

Thus, you have to wade through multiple pages of flat objects to find one you are looking for, and all of the work in sectioning out the object namespace has basically gone to waste.

By contrast, Amazon’s web dashboard for S3, will imply pseudo-directories, as we can see from the analogous view in S3:

Given the hassles with trying to browse our millions of objects in the Rackspace dashboard, I finally decided to bite the bullet and just code up a Django application to do the job (such that I could easily integrate it into our system at work). A few months later, I finally hacked together a first version of the django-cloud-browser application.

Django Cloud Browser

At a high level, Cloud Browser application exposes containers and objects in a Cloud datastore to users or administrators. Let’s look at a basic example:

Basic Deployment

On your system, first install the application:

$ pip install django-cloud-browser

Then, edit your Django “settings.py”. Here, I’m setting up a Rackspace Cloud Files account for browsing:

INSTALLED_APPS = (
    # ...
    'cloud_browser',
)

CLOUD_BROWSER_DATASTORE = "Rackspace"
CLOUD_BROWSER_RACKSPACE_ACCOUNT = "<my_account>"
CLOUD_BROWSER_RACKSPACE_SECRET_KEY = "<my_secret_key>"

Next, add the URLs to your “urls.py”:

urlpatterns = patterns('',
    # ...
    url(r'^cb/', include('cloud_browser.urls')),
)

And start up your Django project! You should be able to click through container and object navigation links, as well as page results when they are over the maximum number of objects per page (configurable via CLOUD_BROWSER_DEFAULT_LIST_LIMIT, but defaults to 20 objects). Here is my navigation into the same Rackspace object as before, but now the objects have implied pseudo-directories.

Objects that lead to ultimate files will be downloaded / displayed when clicked. Breadcrumb links are displayed along the top of the page as well.

The application even figures out and sets appropriate headers for content type and content encoding, which means, e.g., that compressed files will be uncompressed on the fly by browsers that support the compression scheme. At this point, it is worth noting that all access through Cloud Browser is read-only, although in the future we will look to enhancements like uploads, metadata retrieval / modification, etc.

Configuration Options

Other Datastores

In addition to Rackspace Cloud Files, the application supports Amazon S3 as well as the local file system (with a mock container / objects scheme imposed on real directories and files). Examples:

# AWS S3
CLOUD_BROWSER_DATASTORE = "AWS"
CLOUD_BROWSER_AWS_ACCOUNT = "<my_account>"
CLOUD_BROWSER_AWS_SECRET_KEY = "<my_secret_key>"

and

# Local filesystem
CLOUD_BROWSER_DATASTORE = "Filesystem"
CLOUD_BROWSER_FILESYSTEM_ROOT = "/usr/share/doc"

White / Black Lists

By default, all containers and all objects are browsable. However, the application supports basic white and black lists at the container level. If a whitelist is specified, only those containers will be browsable.

CLOUD_BROWSER_CONTAINER_WHITELIST = (
    'my_container',
    'more_stuff',
)

If a blacklist is specified, only those containers are excluded from browsing.

CLOUD_BROWSER_CONTAINER_BLACKLIST = (
    'secret_stuff',
    'archive',
)

If both are specified, access is effectively only what is allowed by the whitelist.

View Decorators

The settings variable CLOUD_BROWSER_VIEW_DECORATOR can be set to a function (e.g., a decorator) that wraps all browsing views. Setting this to Django’s login_required decorator permits only logged-in users to use the Cloud Browser.

Static Media

Cloud Browser contains a minimal amount of CSS and JavaScript. If you set up Django to statically serve these files, then set the CLOUD_BROWSER_STATIC_MEDIA_DIR variable to the relative path from settings.MEDIA_ROOT. The application has a fallback where if this is not set, the CSS / JavaScript will be included (inefficiently) in the HTML of each application view. See the documentation for further details.

Admin Deployment

Cloud Browser also allows for integration with the Django admin. Here, we just make a simple change to our “urls.py”:

urlpatterns = patterns('',
    # ...
    # Admin URLs. Note: Include ``urls_admin`` **before** admin.
    url(r'^admin/cb/', include('cloud_browser.urls_admin')),
    url(r'^admin/', include(admin.site.urls)),
)

It is also a really good idea to manually set the “staff required” decorator in “settings.py” to mirror the other admin view restrictions:

from django.contrib.admin.views.decorators import staff_member_required
CLOUD_BROWSER_VIEW_DECORATOR = staff_member_required

And here’s the same view as above in the admin:

The administrative views also have a closable container element (click the +/- box in the upper right hand corner to toggle):

This work is at a very early stage and will likely change a lot, as there are many unresolved issues: links don’t appear on the dashboard because there are no models, there are no configurable admin permissions to the Cloud Browser views (either all or none), etc. But, it’s close enough for now.

Conclusion

Cloud Browser provides a very basic cloud file viewer for your Django projects, but it gets the job done, particularly for Rackspace Cloud Files. Any and all feedback and bugs are strongly encouraged – just open a ticket on the GitHub issues page. Happy browsing.

Browserless AJAX Testing with Rhino and Envjs, Part 2.

2011-02-16T15:17:00Z

Introduction

This is the second of two posts looking at browserless AJAX / JavaScript testing with Rhino and Envjs. Last time, we set up Rhino and Envjs to run a basic set of QUnit tests from the command line, exercising pure JavaScript code with no AJAX or DOM manipulation. Today, we will add tests for code that hacks on the DOM and makes AJAX calls.

Let’s start with the files we had last time (with links to previous blog post):

env.rhino.1.2.js: Envjs for Rhino.
qunit.css: QUnit styles. (Link to known working version).
qunit.js: QUnit code. (Link to known working version).
setup.js: Our custom hook Envjs to QUnit.
run-tests.js: Include our setup script and actually invoke the tests at on our test HTML page.
my-lib.js: An example JavaScript library (that we want to test).
my-tests.js: An example custom QUnit tests for my-lib.js.
test.html: Basic QUnit HTML page (see QUnit documentation) with script links to “my-lib.js” and “my-tests.js”.

New Files and Tests

Let’s add some new libraries and tests to our working set, to include DOM actions and AJAX:

jquery.js: The venerable jQuery library, which we’ll use for DOM and AJAX calls.
my-tests2.js: Our new tests, described in full detail below.
my-lib2.js: We’ll add two functions – one which does simply DOM manipulation with events, the other which does an AJAX call and inserts the text from another URL into a div element.

// my-lib2.js
/**
 * Add text and clickable elements to div.
 */
function addTextAndClicker($div, text) {
  $div.
    text(text).
    css('color', "rgb(0, 0, 0)").
    css('cursor', "pointer");
  $div.click(function () {
    $div.
      css('color', "rgb(255, 255, 255)").
      css('cursor', "default");
  });
}

/**
 * Dump text from URL into 'code' div.
 */
function dumpText($div, url, callback) {
  $.ajax({
    type: 'GET',
    url: url,
    dataType: 'text',
    success: function (data) {
      // Update the div.
      // Note: The "<pre/>" tag has some issues with Envjs.
      $("<code />").appendTo($div).text(data);

      // User callback.
      if (callback) {
        callback();
      }
    }
  });
}

Interestingly, I originally had the dumpText() function create a new ”<pre/>” tag, but something between Rhino and Envjs did not like it, and Rhino would error out on running this code. So, I switched to ”<code/>” and everything worked fine. I guess the point is that nothing absolutely substitutes for a real browser, but Envjs does a really great job notwithstanding!

Serving Our Files for AJAX

One important thing to note is that because we are doing AJAX calls now, we quickly will run into the same origin policy. To easily get around this for purposes of this demonstration, open a new terminal window, point it to the test directory we’re working in and serve all files in the directory directly from localhost with the python built-in web server (here I chose port 8001):

$ cd /path/to/test/code
$ python -m SimpleHTTPServer 8001 .

Check things out by navigating a real browser to: http://127.0.0.1:8001/test.html. Our QUnit tests from last time should run in the window.

Testing our New Code

Now that we have created our new library and are running a slim web server (so we can actually make AJAX requests), let’s write some test code in “my- tests2.js”. At the module level, we make use of the QUnit setup and teardown options to attach test div elements to the QUnit fixture element, so we can actually do some (temporary) DOM manipulation.

We use an asynchronous test pattern for the dumpText AJAX tests to handle latency between the call and response. The call to stop() stops QUnit execution to allow the code and tests to run. If the asynchronous callback doesn’t finish (and invoke start()) within the designated time period (here 1 second), then QUnit will raise an error. Assuming we have given ourselves enough time, once start() is invoked, then QUnit starts up again for the remaining tests and a final summary. For a good overview of QUnit testing with asynchronous examples, see ”How to Test your JavaScript Code with QUnit”.

Here’s everything together for “my-tests2.js”:

module("my-lib2", {
  /**
   * Add new elements before each test.
   */
  setup: function () {
    var self = this;

    // Store test page fixture.
    self.$fixture = $('#qunit-fixture');

    // Create new test div's.
    self.$addText = $('<div/>').
      attr('id', "add-text").
      appendTo(self.$fixture);

    self.$dumpText = $('<div/>').
      attr('id', "dump-text").
      appendTo(self.$fixture);
  },

  /**
   * Remove tests elements after.
   */
  teardown: function () {
    var self = this;

    // Remove test div's
    self.$addText.remove();
    self.$dumpText.remove();
  }
});

test("addTextAndClicker", function () {
  var self = this;
  equals(self.$addText.text(), "", "Div starts empty");

  // Add the text and clicker.
  addTextAndClicker(self.$addText, "Hi There!");
  equals(self.$addText.text(), "Hi There!", "Div has text");
  equals(self.$addText.css('color'),
         "rgb(0, 0, 0)", "Starting color");
  equals(self.$addText.css('cursor'),
         "pointer", "Changed cursor");

  // Click the clicker and check.
  self.$addText.click();
  equals(self.$addText.text(), "Hi There!","Div still has text");
  equals(self.$addText.css('color'),
         "rgb(255, 255, 255)", "Changed color");
  equals(self.$addText.css('cursor'),
        "default", "Changed cursor");
});


test("dumpText", function () {
  var self = this;
  equals(self.$dumpText.text(), "", "Div starts empty");

  // Stop QUnit execution to wait for AJAX call and tests.
  stop(1000);  // Wait up to 1 second.

  // Perform AJAX call.
  // Just dump our setup script as straight text.
  dumpText(self.$dumpText, "http://127.0.0.1:8001/setup.js",
    function () {
    // Now, run tests.
    var $code = self.$dumpText.find('code');
    equals($code.length, 1, "Verify code element.");

    // Find some known text strings.
    var find = function (text) {
      ok($code.html().indexOf(text) != -1,
         "Find string \"" + text + "\"");
    };
    find("beforeScriptLoad");
    find("afterScriptLoad");
    find("QUnit test runner loaded");
    find("QUnit.log");

    // Restart QUnit execution.
    start();
  });
});

Now, lets update our “test.html” file to run both our previous and new unit tests (and add the jQuery dependency):

<html>
  <head>
    <title>QUnit</title>
    <link rel="stylesheet" href="qunit.css" type="text/css" />
    <script type="text/javascript" src="qunit.js"></script>
    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-lib.js"></script>
    <script type="text/javascript" src="my-tests.js"></script>
    <script type="text/javascript" src="my-lib2.js"></script>
    <script type="text/javascript" src="my-tests2.js"></script>
  </head>
  <body>
    <h1 id="qunit-header">QUnit Test Suite</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture"></div>
  </body>
</html>

Finally, we need to tweak “run-tests.js” to point to the locally served test page instead of simply the local filesystem.

load('setup.js');

console.log("Starting QUnit tests...");
window.location = "http://127.0.0.1:8001/test.html";

And, let’s run the tests!

$ java -cp ~/rhino1_7R2/js.jar org.mozilla.javascript.tools.shell.Main -opt -1 run-tests.js
[  Envjs/1.6 (Rhino; U; Mac OS X x86_64 10.6.6; en-US; rv:1.7.0.rc2) Resig/20070309 PilotFish/1.2.13  ]
Starting QUnit tests...
* QUnit test runner loaded.
  * { addTwo }( 0 )[ PASS ]  Add nothing., expected: 0
  * { addTwo }( 1 )[ PASS ]  Add numbers., expected: 3
  * { addTwo }( 2 )[ PASS ]  Add negatives., expected: -3
  * { addTextAndClicker }( 3 )[ PASS ]  Div starts empty, expected: ""
  * { addTextAndClicker }( 4 )[ PASS ]  Div has text, expected: "Hi There!"
  * { addTextAndClicker }( 5 )[ PASS ]  Starting color, expected: "rgb(0, 0, 0)"
  * { addTextAndClicker }( 6 )[ PASS ]  Changed cursor, expected: "pointer"
  * { addTextAndClicker }( 7 )[ PASS ]  Div still has text, expected: "Hi There!"
  * { addTextAndClicker }( 8 )[ PASS ]  Changed color, expected: "rgb(255, 255, 255)"
  * { addTextAndClicker }( 9 )[ PASS ]  Changed cursor, expected: "default"
  * { dumpText }( 10 )[ PASS ]  Div starts empty, expected: ""
  * { dumpText }( 11 )[ PASS ]  Verify code element., expected: 1
  * { dumpText }( 12 )[ PASS ]  Find string "beforeScriptLoad"
  * { dumpText }( 13 )[ PASS ]  Find string "afterScriptLoad"
  * { dumpText }( 14 )[ PASS ]  Find string "QUnit test runner loaded"
  * { dumpText }( 15 )[ PASS ]  Find string "QUnit.log"

*****************
* QUnit Results *
*****************
* PASSED:  16
* FAILED:  0
* Completed  16  tests total in  2.981  seconds.

From the results, we can see that we now have all of our unit tests running from pure JavaScript code to our new DOM manipulation and AJAX calls.

Other Enhancements

Beyond the setup we discussed today, there are several really cool additions and improvements you can add to your browserless testing solution.

Use Mocks for AJAX Calls

Although running a simple python web server gets us through this demonstration pretty easily for our AJAX calls, in practice AJAX calls might rely on more than a simple static server can provide. At that point, you either have to deal with running your development web application server at the same time, or switch to mocking out the AJAX calls. A great solution is to use fantastic MockJax jQuery plugin, which overrides $.ajax() calls to substitute with a configurable mock response.

Add Code Coverage Reports

Amazingly, the JSCoverage code coverage library does work with this Rhino and Envjs setup. I hooked things up by running jscoverage-server in a similar fashion to how we did the command line python server above (which completely replaces the need for it), and then added the following to “setup.js” in the QUnit.done option:

QUnit.done = function (fail, total){
  // ... previous code ...

  // Code coverage report.
  if (window.jscoverage_report) {
    console.log("Writing coverage report.");
    jscoverage_report();
  }
};

It is important that the coverage report come at the very end of the QUnit.done handler to ensure that all of our tests truly are finished running. See the JSCoverage manual for more details about running the coverage server and writing reports.

Conclusion

Revisiting the original motivation for this post series, the original pain point was that frontend tests are complicated to write and a hassle to run. With the fully browserless testing environment we have explored here, it is possible to create a system out of Rhino, Envjs and QUnit that runs all tests with a single command line argument, making the former hassle into something so easy you could add it to your pre-/post- commit hooks in your version control software. And, by being able to write our tests in mostly straight QUnit (with a little extra setup help), test writing for developers remains relatively straightforward, even dealing with AJAX, etc.

Browserless AJAX Testing with Rhino and Envjs, Part 1.

2011-02-08T06:15:00Z

Updates - 5/27/2011

Since this original post, there has been a lot of good feedback surrounding getting an Envjs + QUnit + Rhino setup going, so please read the comments at the bottom of this post. Here are the summarized points:

QUnit: Newer versions of QUnit have issues with Envjs, adding a <pre/> to output that Envjs can’t handle. A comment from “Ryan” (not me – I’m “Ryan Roemer” below) also points out the QUnit has changed the function signature of the logging callbacks and links to an improved ”setup.js” file. I have inserted links to a known working version of QUnit.
jQuery 1.5: jQuery 1.5+ also has issues with Envjs due to calls to a couple of methods not currently available in either Envjs 1.2 or 1.3. There is a patch on GitHub for the issue for Envjs 1.3. I have backported the patch (and some other fixes) to a forked version of Envjs 1.2 on my GitHub account.
Envjs 1.3: Envjs 1.2 was a single file, with primarily Rhino support. Envjs 1.3 is massive enhancement project that supports a large number of JavaScript engines including: Node.js, SpiderMonkey, etc. It’s still in early development, but definitely worth taking a look at.

Introduction

This is the first of two blog posts focusing on browserless testing of AJAX / JavaScript using Rhino and Envjs. Today, we discuss getting browserless QUnit tests up and running from the command line. A follow-on post will discuss running QUnit tests which involve DOM manipulation and AJAX interaction.

Testing web applications can be an enormous pain. Tests take time to write correctly and usefully, the interaction between the front and back ends is complicated, and it is often a hassle just to get people to run the tests on a regular basis. At work as of late, we have been trying to kick start our JavaScript unit testing – our system has a Django/Python backend, and a JavaScript (mostly jQuery-based) frontend.

On the backend, we are in good shape, test-wise – the Django test case extensions to the Python unit test framework make backend unit testing easy and straightforward. Everything runs fairly quickly, and from the command line. In fact, we have actually made a (soft) requirement that all developers run a Fabric target called “precommit” that runs all of our Django unit tests (in addition to Python and JavaScript style checkers).

On the frontend, it’s a different story. Our previous test suite relied entirely on a browser and a running development server loaded with data in a known state. Our unit tests were written against QUnit, and for the most part worked fine for the code they did exercise. But, even where our frontend tests would catch newly introduced bugs, all too often the tests just weren’t run, and we would actually discover the bug several commit cycles later.

While we clearly need to integrate a fully automated infrastructure around browser-based tests (using great tools like Selenium or Windmill), my thoughts turned to the idea that the heavyweight nature of full end-to-end AJAX testing was getting in the way of us writing and running frontend tests. More specifically, the difficulties come down to two fundamental problems – our tests on the frontend are: (1) not easy to write and run, and (2) are not integrated into an automated, “one-touch” running scheme.

JavaScript Engines and the Pesky Browser Environment

Last week I decided to finally bite the bullet and push hard for the easiest and simplest JavaScript testing solution – execution of a full suite of QUnit tests that would execute without a browser from a single command line command.

The first stop is getting a JavaScript engine to actually run JavaScript (test and code) in lieu of a browser. Fortunately, there are lots of great open source options including:

Rhino: Mozilla’s Java JavaScript implementation.
SpiderMonkey: Mozilla’s C implementation of JavaScript.
V8: Google’s JavaScript virtual machine for Chrome.

A JavaScript engine only gets you pure JavaScript language support. The hard part is that any nontrivial web application with a good amount of JavaScript has a large percentage of code that manipulates the DOM and essentially needs a browser environment for a lot of functionality. So much so, that as general assertion, it is probably not worth trying to separate out pure and browser- dependent JavaScript code in development solely for this purpose.

Thus, the second step necessary for running QUnit tests is a simulation of the browser environment (e.g., window, console, etc.), just without the browser. This is a harder problem.

A few years ago, while working on some prototype code for a friend’s startup, I started researching browserless JavaScript testing, specifically looking for browser emulation options. At the time, the only serious emulation solution I found was in a blog post by John Resig (of jQuery fame), where he had successfully hacked together a pure JavaScript browser environment (called “env.js”) that was capable of AJAX, DOM manipulation, and even running the jQuery set of QUnit tests.

Unfortunately, my buddy’s side project eventually went on hiatus, and I never got around to rigging up a completely browserless test project. Working on purely backend server code for the past couple of years (C++, C#, python), it has taken me some time to get both the opportunity and desire to research a browserless JavaScript environment.

So years later, looking back into the issue, I was pleased to revisit Resig’s proof-of-concept browser simulation environment and find that it had evolved into a full open source project – Envjs – which provides an amazingly powerful browser-like environment, capable of DOM manipulation, AJAX interaction, cookie setting, etc.

Making Rhino, Envjs, QUnit, and JavaScript Tests Play Nicely Together

Rediscovering Envjs and finding it in good shape meant that I simply had to plug in a JavaScript engine, make sure it works with QUnit, and then fire up my test suites without a browser. I ended up choosing Rhino as my JavaScript engine as Envjs has specific support for it, but presumably other engines can be made to work with Envjs and get equivalent results as I will describe below.

So, let’s get started.

Rhino and Envjs Installation

In terms of extra tools / libraries, all we need is Rhino and Envjs:

Rhino is available from the Mozilla FTP site. I ended up going with rhino1_7R2 – the Envjs web site notes that it is best to go with a recent Rhino release. The package can be installed anywhere, but we are going to need to know the path to the file “js.jar” in the package to pass as a class path argument to Java.

After installing, let’s check that Rhino actually works:

$ cd /path/to/rhino
$ java -cp js.jar org.mozilla.javascript.tools.shell.Main -opt -1
Rhino 1.7 release 2 2009 03 22
js> print("hello world");
hello world
js> quit();

Next, install Envjs. I went with Envjs 1.2. (I did end up having to patch the library to get cookies working properly, but that’s a post for another day – and presumably fixed in trunk). Envjs can be installed anywhere the rest of your code can access it – I went with the same JavaScript directory as all our other third party library code.

Example Libraries and Test Code

Ignoring how I actually installed and placed files, for the rest of this discussion, let’s assume we have a simplified world view in a single working directory, where we’ll place all our third party, test and library code. None of these files really need (or should) be in the same directory, but it is easier to describe things without multiple directory traversal.

Here is the third party code:

env.rhino.1.2.js: The Envjs file we downloaded. If you are using jQuery 1.5+, you may want this patched Envjs instead.
qunit.css: QUnit styles. (Link to known working version).
qunit.js: QUnit code. (Link to known working version).

We are going to create two more files in a subsequent section.

setup.js: Hook Envjs to QUnit.
run-tests.js: Include our setup script and actually invoke the tests at on our test HTML page.

And, here is some library and test code files:

my-lib.js: An example JavaScript library (that we want to test).

function addTwo(x, y) {
    return x + y;
}

my-tests.js: An example custom QUnit tests for my-lib.js.

module("My Module");

test("addTwo", function () {
    equals(addTwo(0, 0), 0, "Add nothing.");
    equals(addTwo(1, 2), 3, "Add numbers.");
    equals(addTwo(-1, -2), -3, "Add negatives.");
});

test.html: Basic QUnit HTML page (see QUnit documentation) with script links to “my-lib.js” and “my-tests.js”.

<html>
  <head>
    <title>QUnit</title>
    <link rel="stylesheet" href="qunit.css" type="text/css" />
    <script type="text/javascript" src="qunit.js"></script>
    <script type="text/javascript" src="my-lib.js"></script>
    <script type="text/javascript" src="my-tests.js"></script>
  </head>
  <body>
    <h1 id="qunit-header">QUnit Test Suite</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture"></div>
  </body>
</html>

Once we’ve got all of these custom and third party files, we can just open test.html in a browser, and the QUnit tests should all run, pass and look something like:

Configuring Envjs and QUnit

We need to hook QUnit into Envjs, so we can display output to stdout (using console.log()). Fortunately, QUnit explicitly provides a lot of hooks to do everything we want, which I’ve implemented in a file named “setup.js”, placed alongside our other JavaScript libraries.

I mostly followed the excellent guide from the Envjs folks, and here’s the gist of my configuration for “setup.js”:

load('env.rhino.1.2.js');
load('qunit.js');

var starttime = new Date().getTime();

// Envjs/QUnit Bridge.
Envjs({
    // Straight from the Envjs guide.
    scriptTypes: {
        "": true,
        "text/javascript": true
    },
    // Straight from the Envjs guide.
    beforeScriptLoad: {
        'sharethis': function (script) {
            script.src = '';
            return false;
        }
    },

    // Hook QUnit logging to console.
    afterScriptLoad: {
        'qunit': function () {
            var count = 0, testName;

            console.log("* QUnit test runner loaded.");

            // Grab current test name.
            QUnit.testStart = function(name, testEnvironment) {
                testName = name;
            };
            // Override log to display to stdout.
            QUnit.log = function (result, message) {
                // Strip out HTML in results messages.
                message = message.replace(/<\/?.*?>/g, '');
                console.log("  * {%s}(%s)[%s] %s",
                    testName, count++,
                    result ? 'PASS' : 'FAIL', message);
            };
            QUnit.done = function (fail, total){
                var endtime = new Date().getTime();
                var pass = total - fail;
                console.log("\n" +
                    "*****************\n" +
                    "* QUnit Results *\n" +
                    "*****************\n" +
                    "* PASSED: %s\n" +
                    "* FAILED: %s\n" +
                    "* Completed %s tests total in %s seconds.\n",
                    pass, fail, total,
                    parseFloat(endtime-starttime) / 1000.0);
           };
        },

        // Straight from the Envjs guide.
        '.': function (script) {
            script.type = 'text/envjs';
        }
    }
});

Note that the Envjs guide actually went further than this example, as the guide’s setup code writes the QUnit HTML report out to a new file. The interesting thing about running JavaScript in Rhino is that code runs with full system permissions outside of the JavaScript sandbox, so you can also do normal file system manipulation. (And, as the Envjs documentation notes with big scary warnings, there are huge security issues with running outside code within an Envjs / Rhino environment).

Test Runner

The last file we have to compose is our actual test runner. Fortunately, this is quite simple, as shown in “run-tests.js”– just load the setup and point to the test page:

load('setup.js');

console.log("Starting QUnit tests...");
window.location = "test.html";

Test it Out!

Now that we have Rhino, Envjs and QUnit all hooked together, it’s time to actually run some QUnit tests. Assuming your file / directory setup matches the above contrived example, we should be able to run the tests by invoking Rhino with our test runner script:

$ java -cp ~/rhino1_7R2/js.jar org.mozilla.javascript.tools.shell.Main -opt -1 run-tests.js
[  Envjs/1.6 (Rhino; U; Mac OS X x86_64 1 ... ]
Starting QUnit tests...
* QUnit test runner loaded.
  * { addTwo }( 0 )[ PASS ]  Add nothing., expected: 0
  * { addTwo }( 1 )[ PASS ]  Add numbers., expected: 3
  * { addTwo }( 2 )[ PASS ]  Add negatives., expected: -3

*****************
* QUnit Results *
*****************
* PASSED:  3
* FAILED:  0
* Completed  3  tests total in  1.989  seconds.

Awesome. We have QUnit testing goodness for our library, running within Rhino, and log / results output to the console, all through a single bash command. We can add and include more libraries and test suites to test.html, and they automatically get picked up by the test suite.

Next Time – Browserless AJAX, Playing with the DOM and More!

Today we examined and set up basic browserless JavaScript unit tests with Rhino and Envjs. Despite the promises of the title of this post, we’ll have to wait until next time to extend out QUnit test suite to include actual / mocked AJAX testing and DOM manipulation with this set up. But, just removing the browser from the frontend test path is a great step towards more lightweight JavaScript unit tests – and situating us for the type of tests that developers will want to write and run.

Rackspace Cloud Files and Pseudo-Directories

2010-12-28T19:57:00Z

Cloud Providers and Listing Storage Objects

Rackspace’s Cloud Files provide a basic distributed blob / file abstraction. Like Amazon Web Services Simple Storage Service (AWS S3) and Microsoft Azure Blobs, Cloud Files are organized into a two level hierarchy consisting of:

Containers: A collection of file objects. In AWS S3 parlance this is called a “bucket”. For all three cloud providers, a container is a single collection unit that cannot be further subdivided.
Storage Objects: A single file object. In AWS S3, this is an “object”, while in Azure this is a “blob”. A storage object for all the providers similarly cannot be further nested / subdivided.

Thus, there are no further levels of hierarchy than container, then object, and for all three providers, the storage object namespace is completely flat.

As most of the world is used to nested file hierarchies much deeper than this, cloud providers allow wide leeway in naming storage objects, most notably allowing characters like a slash (”/”). To provide the illusion of a nested hierarchy, AWS S3 and Azure Blobs provide listing operations that take a delimiter character to treat as a nested directory delimiter. In this manner, calls to the list objects API will return results as if there are intermediate directories in the (really) flat storage object namespace.

The Old Way - Rackspace Cloud Files and Dummy Directory Objects

To much frustration, Rackspace Cloud Files did not originally provide a delimiter query API for treating a delimiter as a nested directory organizer. Instead of delimiter queries, Rackspace required that clients upload a storage object of type “application/directory” at each level in which a nested pseudo- directory was desired. Only then would the results start to resemble those from AWS / Azure with delimiter queries.

Here’s a quick example showing how the dummy directory objects work. First let’s create a container:

>>> import cloudfiles
>>> conn = cloudfiles.get_connection(ACCOUNT, SECRET_KEY)
>>> cont = conn.create_container("my_test_container")
>>> cont.list_objects()
[]

Now, put in some objects with internal slashes, and list all.

>>> cont.create_object("foo/bar/file1.txt").write("some text.")
>>> cont.create_object("foo/bar/file2.txt").write("some text.")
>>> cont.create_object("foo/diff/file3.txt").write("some text.")
>>> cont.create_object("baz/file4.txt").write("some text.")
>>> cont.list_objects()
['baz/file4.txt',
 'foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

We can use the “prefix” parameter to limit files to only “foo”, but the internal pseudo-directories are not inferred.

>>> cont.list_objects(prefix="foo")
['foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

The “path” parameter will infer directories, but not without internal dummy directory objects.

>>> cont.list_objects(path="foo")
[]

So, let’s create the dummy objects for “foo”.

>>> dummy = cont.create_object("foo")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

>>> dummy = cont.create_object("foo/bar")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

>>> dummy = cont.create_object("foo/diff")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

And, now we get the pseudo-directory results we expect for “foo” and children paths.

>>> cont.list_objects(path="foo")
['foo/bar',
 'foo/diff']

>>> cont.list_objects(path="foo/bar")
['foo/bar/file1.txt',
 'foo/bar/file2.txt']

>>> cont.list_objects(path="foo/diff")
['foo/diff/file3.txt']

While the dummy objects technically make nested pseudo-directories possible, the onus falls to the user to create and maintain all the dummy objects. This really becomes a pain when creating new objects that might contain nested delimiters for which there is not already a delimiter (say adding a nested pseudo-directory). The user then has to either check / scrub all existing files for missing dummy objects, or create duplicate dummy directory objects on each new object creation.

The New Way - Rackspace Cloud Files and Delimiter Queries

Fortunately, Rackspace finally enabled “real” delimiter queries in their list storage objects API. I couldn’t actually find the date that this was added to REST API and added server side (as the current API PDF doesn’t even note the change, although I expect it soon will).

I first noticed the new query functionality when perusing commits on the Python API GitHub page. On Dec. 3, 2010, a change was pushed that added the “delimiter” parameter to object listing queries for the Python API. The Python API was incremented to version 1.7.4 for the change, so make sure to pull the proper version of the library to test the new feature out:

$ pip install -U https://github.com/rackspace/python-cloudfiles/tarball/1.7.4

Interestingly, while the container list_objects() and list_objects_info() methods picked up a delimiter parameter, the get_objects() method did not. At any rate, this gets the job done, and we can remove our dummy objects now to retry our examples:

>>> cont.delete_object("foo")
>>> cont.delete_object("foo/bar")
>>> cont.delete_object("foo/diff")

Let’s check we don’t have the dummy objects anymore:

>>> cont.list_objects()
['baz/file4.txt',
 'foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

And now, let’s try equivalent queries using the prefix and delimiter parameters. (Note that you must append an extra slash (e.g. “foo/”, not “foo”) for the queries to work.)

>>> cont.list_objects(prefix="foo/", delimiter="/")
['foo/bar/',
 'foo/diff/']

>>> cont.list_objects(prefix="foo/bar/", delimiter="/")
['foo/bar/file1.txt',
 'foo/bar/file2.txt']

>>> cont.list_objects(prefix="foo/diff/", delimiter="/")
['foo/diff/file3.txt']

… and we finally get the results we want without having to track or create any extraneous objects.

At the time of this post, many clients do not take advantage of more convenient / advanced object querying. The CyberDuck client currently will only display pseudo-directories via the dummy directory objects method. In fact, Rackspace’s own control panel doesn’t display any pseudo-directories using either the old dummy directory or new delimiter methods. So, hopefully with the new delimiter query API support, some folks out there will make some New Year’s resolutions to improve pseudo-directory support in Cloud Files clients.

Celery Logging with Python Logging Handlers

2010-12-13T15:54:00Z

Logging in Celery

We use Celery as our backend messaging abstraction at work, and have lots of disparate nodes (and across different development, test, and production deployments). As each system deployment now contains a large (and growing) number of nodes, we have been making a heavy push towards consolidated logging through a central logging server sink, using syslog (specifically rsyslog).

So, ideally, we would just use Celery’s configuration to specify a syslog handler (or maybe use a pipe). Unfortunately, it seems there is just not a simple way of doing this straight from Celery, as the only logging configuration parameters (with possible values) are:

CELERYD_LOG_FILE = "/path/to/file.log"  # File logging.
                                        # (OR)
CELERYD_LOG_FILE = None                 # stderr.

which means you either get a file logger (filehandler) or a stderr logger (streamhandler).

Hints and guidance on the web in terms of getting arbitrary logging handlers (or syslog specifically) is sparse, as well as noisy. A lot of the code and discussion I found used Celery as the backend for a generic logging framework or library. But, I found very little in terms of getting Celery tasks and processes to actually log to syslog.

Bringing Arbitrary Logging to Celery

As nothing magically appeared as the “right” solution, I considered a couple of different ways to hook things up:

Watched Files: Have Celery log to a file per usual, then add extra configuration and scripts to watch the file for changes and submit the changes to syslog directly. I didn’t go with this approach, as I really prefer to have configuration for our project within generic Python settings, and not need extra, system-specific scripts and setup.
Patch Celery: The Celery logging hooks and code are fairly straightforward. I could have added the patch and submitted upstream. Unfortunately, for the project at work, we are moving to fast to wait for changes, and as we are looking forward to some upstream Celery releases, I’d rather not maintain a private custom patch set through all of that.
Monkey Patch Celery: In the same vein as the former option, the same hooks and changes could be applied as a monkey patch instead. This is what I eventually went with.

Patching Logging Handlers into Celery

Monkey patching is oft-controversial and generally discouraged practice, as getting things wrong is easy, and interactions within the patched library / code can get really messy. In our case, I reluctantly chose monkey patching, as it was a short patch, easy to disable, and I had no good other solutions.

Now that I’ve given the standard disclaimer, let’s get to the patch! The relevant Celery function we want to patch in celery.log._setup_logger():

def _setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):

    if logger.handlers:                 # already configured
        return logger

    handler = _detect_handler(logfile)
    handler.setFormatter(formatter(format, use_color=colorize))
    logger.addHandler(handler)
    return logger

As we can see, there is only one shot at a handler, and it comes from celery.log._detect_handler(), which either returns a stream or file handler. What I would ideally like is to define be able to define a Django settings variable (we use Django and not native Celery settings at work):

PATCH_CELERYD_LOG_HANDLERS = (
    lambda: logging.FileHandler("path/to/another_log.txt"),
    lambda: logging.handlers.SysLogHandler(),
)

and define a new _setup_logger():

import celery.log

from celery.log import ColorFormatter
from django.conf import settings

old_setup_logger = celery.log._setup_logger

def _setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):

    patched_logger = old_setup_logger(
        logger, logfile, format, colorize, formatter, **kwargs)

    for handler_fn in getattr(settings, "PATCH_CELERYD_LOG_HANDLERS", []):
        handler = handler_fn()
        handler.setFormatter(formatter(format, use_color=colorize))
        patched_logger.addHandler(handler)

    return patched_logger

I would have preferred a pure ”*args, **kwargs” version of the patched function, but unfortunately, we need to dig out the formatter parameter if set, so I copied the method signature of the original. Also, I used lambda’s in PATCH_CELERYD_LOG_HANDLERS and called the function in the new setup logger so as to create a new handler object each time for loggers. It is important to have different handler objects because formats attach to handler objects, and Celery uses different formats for general/default/process logging vs. task-specific logging. If a handler has last been set to a task logger format, that includes a format for among other things “task_name”, then you will get a KeyError on trying to issue a logging statement.

Other than that, the patched method is very easy to read – it iterates the custom handler function list and creates new handlers for each different logging object. Putting this in to a patched version, I also added a marker attribute to the logger object (to tell the patching is done), as well as the patching code to lock, switch and mark the methods:

import logging
import celery.log

from celery.log import ColorFormatter
from django.conf import settings

old_setup_logger = celery.log._setup_logger  # Store the real method.
def my_setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):
    """Replacement for :method:`celery.log._setup_logger`."""

    patched_logger = old_setup_logger(
        logger, logfile, format, colorize, formatter, **kwargs)

    # Check if not patched.
    if not getattr(patched_logger, "_LOG_PATCH_DONE", False):
        # Lock and patch.
        logging._acquireLock()
        try:
            handler_fns = getattr(settings, "PATCH_CELERYD_LOG_HANDLERS", [])
            for handler_fn in handler_fns:
                handler = handler_fn()
                handler.setFormatter(formatter(format, use_color=colorize))
                patched_logger.addHandler(handler)

            # Mark logger object patched.
            setattr(patched_logger, "_LOG_PATCH_DONE", True)
        finally:
            logging._releaseLock()

    return patched_logger

# Apply patches.
CELERY_MOD_PATCHED = False
if not CELERY_MOD_PATCHED:
    logging._acquireLock()  # Lock logging during patch.
    try:
        celery.log._setup_logger = my_setup_logger  # Patch old w/ new.
        CELERY_MOD_PATCHED = True
    finally:
        logging._releaseLock()

With the patch applied, any number of Python standard logging handlers can now be added to the celery task / process loggers.

Applying the Patch at Runtime

So we’ve defined the patch, but how do we apply it? Part of the problem is celery configuration. If you apply the patch too early, you get the following exception:

.../python2.6/site-packages/celery/loaders/default.py:53: NotConfigured:
No celeryconfig.py module found! Please make sure it exists and is available to Python.

So, we have to find a proper place in the initialization process, after Celery settings. The best solution I found for this is to create a faux Django “application” , that contains only the patch code. Let’s call the application “patch_celery”:

$ cd path/to/project
$ django-admin.py startapp patch_celery

Copy and paste all of the patch code above into the init.py. And then, we simply add the patch application to the Django settings.py installed apps:

INSTALLED_APPS = (
    # ...
    'djcelery',
    'patch_celery',
    # ...
)

The best place to put the patch is right after the djcelery application. Now, add your patch variable, PATCH_CELERYD_LOG_HANDLERS, to settings.py with any designated handlers (per previous example) and you’re off with any logging handlers for celery tasks that you want! You can disable the patch by simply removing the “patch_celery” application from installed applications in settings, which enables easily toggling per appropriate context without major code swaps.

Afterthoughts

The monkey patch to Celery certainly gets the job done for now. However, the fact that the full method signature for celery.log._setup_logger() is used makes the patch very brittle – if a parameter changes in the underlying method, the patched method may cease to work, or worse, give a wrong parameter to the underlying method.

Rackspace Cloud Files and Servicenet

2010-10-18T00:01:00Z

Cloud Files, Cloud Servers

At work, we use Rackspace Cloud Files for bulk blob storage of large sets of documents (in our case, patents). We use a farm of Rackspace Cloud Servers to process and serve blobs from Cloud Files. Rackspace’s offering has been very solid to date (modulo some idiosyncrasies of their particular APIs) and quite interestingly, the underlying software is being released as an open source project, OpenStack.

One of the nice things building a solution on Rackspace is that bandwidth is free between Cloud Servers and Cloud Files, if properly configured. However, as our recent experience bears out, there are some pitfalls and caveats towards getting everything set up correctly.

Public, Private Interfaces and Servicenet

Rackspace Cloud Servers are essentially Xen VMs that come with two network interfaces: (1) a public interface on eth0 with a world-routable IP address, and (2) a “private” interface on eth1 with an RFC 1918 address. (The private interface isn’t really “private” in the sense that any other VMs on the same network can route to your VM, so security-wise, it should be treated as a public, internet-facing interface). For more details, Rackspace has published a good article on the public and private interfaces.

For billing purposes, bandwidth between Cloud Files and a Cloud Server VM on the public interface is charged per normal, whereas bandwidth on the private interface is free. To enable Cloud Files communication on the private interface, you can either set an environment variable in your application like follows:

export RACKSPACE_SERVICENET=True

or programmatically specify the private network, as shown here using the Python Cloud Files client library:

import cloudfiles
conn = cloudfiles.get_connection("acct", "api_key", servicenet=True)

In both cases, “Servicenet” refers to the private interface network internal to a Rackspace data center.

Servicenet Gotchas

All seems pretty simple! However, when we finally enabled “servicenet=True” in our Cloud Servers, we immediately hit errors talking to Cloud Files and everything refused to run. After a lot of back and forth with the Rackspace folks, we finally found out the main issue: Servicenet only works between Cloud Files and Cloud Servers in the same data center.

Before we ran into this snag, we hadn’t even checked that our servers and blob storage were in the same data center. As it turned out, our Cloud Files location was in Texas (DFW) and our Cloud Servers were in Chicago (ORD). And, after much hand-wringing, we learned that: (1) new Cloud Servers will most likely be provisioned in Chicago, and (2) you can’t easily migrate Cloud Files accounts from Texas to Chicago.

So, how can one easily tell where your Cloud Servers and Cloud Files are, and whether or not Servicenet will be available? The answer for Cloud Servers is straightforward: in the Rackspace management dashboard, look at your server list and the “Datacenter” column. In our case, all servers list as “ORD1”, which means Chicago. Moving on to Cloud Files, one way to figure things out is to fire up a Python shell and issue some basic commands to see what we have. In all cases, looking at a connection object’s ”connection_args[0]” value will yield a string for public interfaces like:

storage<NUMBER>.<DATACENTER>.clouddrive.com

and on the private interface:

snet-storage<NUMBER>.<DATACENTER>.clouddrive.com

So, on to some example code using the Python client. Here is a Chicago Cloud Server talking to Chicago Cloud Files on the public interface:

>>> import cloudfiles

>>> chicago_pub_conn = cloudfiles.get_connection("chicago_acct", "1234567890", servicenet=False)
>>> chicago_pub_conn.connection_args[0]
'storage101.ord1.clouddrive.com'
>>> chicago_pub_conn.list_containers()
['foo1', 'bar1', 'baz1']

… and on the private interface:

>>> chicago_priv_conn = cloudfiles.get_connection("chicago_acct", "1234567890", servicenet=True)
>>> chicago_priv_conn.connection_args[0]
'snet-storage101.ord1.clouddrive.com'
>>> chicago_priv_conn.list_containers()
['foo1', 'bar1', 'baz1']

Now, switching to a Chicago Cloud Server talking to Texas Cloud Files on the public interface, we get:

texas_pub_conn = cloudfiles.get_connection("texas_acct", "1234567890", servicenet=False)
texas_pub_conn.connection_args[0]
'storage101.dfw1.clouddrive.com'
>>> texas_pub_conn.list_containers()
['foo2', 'bar2', 'baz2']

but, trying to talk on the private interface yields the following exception:

>>> texas_priv_conn = cloudfiles.get_connection("texas_acct", "1234567890", servicenet=True)
>>> texas_priv_conn.connection_args[0]
'snet-storage101.dfw1.clouddrive.com'
>>> texas_priv_conn.list_containers()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/cloudfiles/connection.py", line 410, in list_containers
    raise ResponseError(response.status, response.reason)
cloudfiles.errors.ResponseError: 404: Not Found

Confusingly, the exception raised is a 404 “Not Found”, when it really should be something more context-specific.

Final Thoughts

The last time I chatted with the Rackspace folks, they noted most new accounts will have both Cloud Servers and Cloud Files allocated in Chicago, so using Servicenet should be as simple as setting the appropriate variable. In our specific case, we got around our Texas-Chicago split by allocating an entirely new account that has both servers and files in Chicago, and ported over our Texas files. Somewhat tedious, but a worthwhile long-term investment.

But, if you are trying to use Servicenet and experience 404’s or other strange failures, you probably want to go through the above exercise and identify the data center where your servers and blobs reside. And, if you use Cloud Servers and Cloud Files and aren’t already using Servicenet, you definitely should because you are paying for bandwidth that should otherwise be free!

Distributed Task Locking in Celery

2010-10-10T20:20:00Z

Background

We use the Celery distributed task queue library at work, which is great for running asynchronous tasks across multiple processes and servers. Celery has both user-initiated and periodic (think cron replacement) tasks, and we have found in practice that the system distributes tasks quite nicely across our farm of celery servers.

One issue we have is that for several of our periodic tasks, we need to ensure that only one task is running at a time, and that later instances of the same periodic task are skipped if a previous incarnation is still running.

The Celery documentation has a cookbook recipe for this scenario: ”Ensuring a task is only executed one at a time”. The crux of the solution is to make a distributed lock using the Django cache (memcached in the example) with the following lambda’s:

lock_id = "something unique"
lock_expire = 60 * 5  # five minutes

acquire_lock = lambda: cache.add(lock_id, "true", lock_expire)
release_lock = lambda: cache.delete(lock_id)

Non-persistent Locks?

This approach works fine if the cache is shared across all celery worker nodes and the cache is persistent. However, if memcached (or some other non- persistent cache) is used and (1) the cache daemon crashes or (2) the cache key is culled before the appropriate expiration time / lock release, then you have a race condition where two or more tasks could simultaneously acquire the task lock. This “distributed cache lock” approach has been discussed in various posts, which all acknowledge the danger of relying on memcached for persistent data.

Distributed Locks with Redis

As noted by one of the links above, the simplest solution to this problem if you like using the cache for distributed locks is to switch to memcachedb which is not a caching solution per se, but rather a persistent key-value store that implements the memcached interface.

However, for our system, using memcached as our Django cache is great in its current non-persistent form for what it is – a cache. So, I investigated more generally to find a high performance, persistent key-value store (ideally with a decent Python interface). After reviewing a lot of neat and interesting systems, I finally settled on Redis. Redis provides a very high performance key-value store (data is maintained in memory) with persistence (using the append only file feature) and distribution / replication. As an added bonus, I found the server setup, installation and CLI interaction to be very straightforward.

So, back to locking… The Redis python client already has a lock class with ”with” operator support:

import redis

with redis.Redis().lock("my_key"):
    print("Got lock.")

However, the above example is a blocking lock, and for the “single task” issue, we want a non-blocking lock, that simply exits if the lock is not acquired:

import redis

have_lock = False
my_lock = redis.Redis().lock("my_key")
try:
    have_lock = my_lock.acquire(blocking=False)
    if have_lock:
        print("Got lock.")
    else:
        print("Did not acquire lock.")

finally:
    if have_lock:
        my_lock.release()

Beyond this simple example, the Lock class implements key expiration (via the Redis setnx command) to enable timeouts in the python client.

Enforced Single Celery Task

So, bringing this back to our celery tasks, we can use this distributed lock to have our tasks try to acquire a non-blocking lock, and exit if the lock isn’t acquired. Also, we want to set a lock timeout (lasting for a generous overestimate of task duration time) so that tasks will eventually be able to re-acquire the lock if some task / celery node hard crashes or goes in to an unresponsive state before releasing the lock.

All of this can be put together as a decorator around a Task.run() method:

import redis

REDIS_CLIENT = redis.Redis()

def only_one(function=None, key="", timeout=None):
    """Enforce only one celery task at a time."""

    def _dec(run_func):
        """Decorator."""

        def _caller(*args, **kwargs):
            """Caller."""
            ret_value = None
            have_lock = False
            lock = REDIS_CLIENT.lock(key, timeout=timeout)
            try:
                have_lock = lock.acquire(blocking=False)
                if have_lock:
                    ret_value = run_func(*args, **kwargs)
            finally:
                if have_lock:
                    lock.release()

            return ret_value

        return _caller

    return _dec(function) if function is not None else _dec

Note that this decorator preserves task return values. If your tasks don’t have return values, you can get rid of the ret_value code.

Using the decorator is easy – just annotate a task run() method:

from celery.task import Task

class SingleTask(Task):
    """A task."""

    @only_one(key="SingleTask", timeout=60 * 5)
    def run(self, **kwargs):
        """Run task."""
        print("Acquired lock for up to 5 minutes and ran task!")

… and your task will only ever have one running instance at any given time.