Loose Bits

Writing a Technical Book, Part 4 - Marketing & Beyond

2015-10-10T00:00:00+00:00

Marketing & Post-Publication

In my fourth and final post about my adventure writing Backbone.js Testing, I take a look back on what has happened since the book was published in July, 2013, discuss the marketing initiatives we used to drive book sales, and finally reflect on the awkward, tough, and wonderful experience of writing my first book.

As it’s now been over two years since my book’s publication and the first post in this series, I’ll just chalk up this tardy final post to something quite familiar in the book publishing world – deadlines slip. And as a refresher, here are some quick links to the previous posts in this series:

Marketing & Selling a Book

In my latest post on publishing the book, I discussed all of the legwork needed to translate the various chapter drafts into a book that you could purchase. On July 12th, 2013, the book was published, after which Packt (my publisher) turned our focus to marketing the book.

Technical Article & Silence

After notifying me of the formal publication announcement, my commissioning editor requested a technical article of about 5 pages for Packt to publish so that we could garner some search engine goodness and drive people to the book’s sale page. I was a little wary of yet more writing after the book, but the editor said that we could just grab a section of the book itself as the article content. I put together a recommended list of book sections to serve as the basis for the proposed article.

This ended up being a huge fiasco. The editor dropped the issue for over a week, then changed directions and said a technical committee would select the book portion for the article for me. After a last email on July 25th, all communication with the marketing / technical folks just went dead silent.

I wasn’t much too concerned, as it was nice to have a break from the back-and- forth of requested book work. I started doing some of my own book publicity via LinkedIn and Twitter and just generally took it easy.

A Marketing Executive & Actual Marketing

On Sept. 3rd, over a month after any communication, my commissioning editor sent over an email introducing me to a marketing executive. The marketing executive seemed to be a little more put together and wanted to launch various different initiatives for the book with:

The ill-fated article project
A book giveaway
Book reviews

Oh, the Article. It Hurts.

For the article, the marketing executive asked “why hasn’t this been published yet?” and I forwarded on the circuitous email chain of inaction. The commissioning editor finally chimed in again and blamed the editorial department. The email chains went in further circles and then the article project just got dropped due to no one really following through.

Free Books!

The book giveaway was really easy. I announced a contest based on a template from the publisher – offering five copies of the book to the best five blog comments about frontend testing. The contest got some good responses and we gave out some books.

Book Reviews

For book reviews, I solicited a lot of local area Seattle JavaScript developers that I new and respected, and reached out the local JavaScript meetup group. Packt provided free copies of the book in exchange for Amazon reviews. Fortunately, I received a good amount of interest and got about 15 potential reviewers that I hooked up with the marketing executive to receive free books.

About five reviews went live on Amazon within the first two weeks of our reviewing campaign, and my intuition is that these reviews probably had the most positive impact on sales of the book of anything we tried. Since the original push, I ended up with eight solicited reviews (with the standard disclaimer of “I received this book in exchange for reviewing”) and two organic reviews from people I don’t know. While I was originally nervous about the book’s reception, I was pleased to end up with nine 5-star and one 4-star reviews on Amazon.

After all the planned marketing campaigns finished, the marketing executive announced “we’re done!” and moved on to one of presumably many other book campaigns for Packt. He was fairly competent and focused on the task at hand, which was a welcome relief from some of the other mishaps along the way.

… and Everything After

After marketing push ended, my obligations to the book project mostly wrapped up as well. Every three months I get a royalty statement – the book continues to sell in the low hundreds, with the vast majority of purchases being ebooks. The marketing executive periodically drops a note to encourage me to tweet about Packt’s latest company-wide marketing campaign. And, I occasionally check my Amazon page for new reviews or sales rank.

So, was it worth it?

Definitely. The book writing, publishing, and marketing process was a positive experience on several levels:

I honed my technical writing skills and discovered that I could write an entire friggin’ book.

I became recognized (at least at some level) as an expert on JavaScript testing and Backbone.js. I’ve talked with other technical authors and we agree on the broad proposition that you are usually not an expert when selected to write a book, but by the time you are done with the book you will be.

I met and interacted with a good chunk of the Seattle area JavaScript community. My book writing period coincided with a move from DC to Seattle and in seeking feedback about the book, I made some really good connections to the development community in my new home. And, best of all, when giving a lightning talk about the book at a local meetup, I ended up meeting my future business partner and co-founder of Formidable Labs.

Would I do it again?

I’m not sure the future holds another book project for me. The enormous time commitments, editorial bumps and mishaps, and lack of any real money at the end of the road all factor against it.

Having one book under your belt is great. But I’m wary about the marginal value in community outreach, personal brand, etc. that I would receive from another one.

So, in the meantime, I’ve been focusing more on things like organizing conferences and local meetups, and writing open source software. And, you know what? Those things, coupled with not having book work on the horizon, is pretty nice.

Writing a Technical Book, Part 3 - Publication

2014-11-26T00:00:00+00:00

Publishing a Book

At long last, this is the third post in a series on my experiences writing Backbone.js Testing, published in July, 2013. In this post, I’ll dive into the publication process for my book and expand on the following takeaway points:

There is a good amount of boring, necessary work on your book to turn final drafts into correctly laid-out, ready-for-printing pages.
You need to keep a keen eye to everything about your book, including the cover / marketing language and especially your publisher’s book webpage.
This is your last chance to catch both small and extremely large errors.
Working though the final mile of publication with your publisher can be frustrating…
… but it feels wonderful to actually have the book done, published and in a very literal sense, “shipped”.

You’ve Written a Book. Now What?

In my previous post on authoring a book, I recount writing the individual chapters of the book, going through technical and editorial review, and submitting the “final” draft for publication at the end of April, 2013.

After this final submission, I asked the publisher’s project folks what to expect from that point to actual publication and was told that the process for each chapter was something along the lines of:

Technical editing.
Copy editing
Indexing key terms
Layout adjustments for actual, rendered book pages
Proofreading
“Prefinal” submission for the author’s final review
Finalization and clubbing

Each of those steps was forecast to take 1-2 days and run in parallel for all the chapters.

It was at that time that I also reminded Packt (my publisher) that we still needed to formally update the book’s title from “Backbone Testing” to “Backbone.js Testing” in all of the materials, something Packt repeatedly told me was easy to do throughout the process.

A Break + How Hard Can a Title Change Be?

As my publisher had pushed me quite hard the entire time I was writing my chapters to get everything done as fast as possible, I was surprised to hear mostly radio silence for around twenty days. Finally, I started poking around Packt’s website to see if they had started any pre-publication activity, and to my pleasant surprise, they had – putting up a web page for my book.

Unfortunately, they somehow managed to screw up my title change request and renamed the book to “Testing Backbone.js”. Issues aside about the difficulty of getting two simple words right and in the correct order, I was extremely disappointed – a publishing house should be able to correctly assign a book title. And, Packt had submitted the incorrect title for pre-publication book pages on Amazon, Barnes & Noble, and other online book retailers.

Somehow making things worse, the page that Packt put up for pre-publication was nonsensically worded in its description of the book and why prospective readers should buy it – with repeatedly copy-and-pasted sentences and gems like “Test your web applications using backbone’s tried and tested techniques and industry best practices”. (For those not familiar with Backbone.js, just trust me that the last statement is pure technical gibberish).

So, I emailed my commissioning editor, only to find out that they had been on break for some time (which probably explained the lack of editorial activity). I finally tracked down my previous commissioning editor and roped them in to fixing the mess.

It turns out that Packt had just been more or less guessing at the marketing and book cover metadata – bullet points about why to purchase the book, the title tagline, etc. So, I undertook reviewing all the book metadata and online content and provided my publishers with new content for all of it. Although time consuming, I really wanted something that would make sense from the point of view of an actual prospective reader, and hopefully make the book seem like a good purchase.

Through all of this mess, there were some actual silver linings. I got to actually choose a picture for the book cover, which I chose from one of my wife’s photographs from nearby our house. Here’s the cover again for reference:

And, I did learn about all of the details of the book cover and website marketing content because I was so paranoid about something else inaccurate or nonsensical being written on my behalf about the book.

Prefinal and Final Editing

After the diversion of the book title / marketing material, Packt finally got the first chapters back to me for prefinal review on May 30th, supposedly targeting a “to the printers” date of June 10th. When I asked about the delay and impending deadline, the publisher responded that they had actually been short-staffed the entire time after I submitted my final drafts. We jointly agreed to push back the actual final deadline.

The prefinal and final editing process was mostly boring but important work. I was sent PDFs of what would be close to the final layout and look for the chapters to review, with comments and suggestions from the publisher. The technical editors and proofreaders didn’t find many issues and sometimes mis-identified correct writing as errors. But, they did ultimately find a number of real errors we were able to correct.

It was also a good opportunity for me to re-read my chapters after about a month’s break. At that point, I didn’t really have a chance to make substantive changes (which would dramatically affect the layout work), but I was able to gauge the readability and substantive arch of the book as a whole. And, I also found a few bugs and errors on my own in the process.

Packt and I went back and forth either one or two times per chapter, circling around a narrowing group of sentences and words to discuss, and then finally arrived at the “good enough!” point. On July 3rd, I sent over the final approval of all the remaining chapters. And, on July 12th, I received confirmation that the book was uploaded to the printers and thus officially published / available for purchase.

It’s Published!

I will post a final article on the post-publication / marketing experience with my book, but the short version is that I did feel a huge relief actually getting the book out the door and curiosity as to what would happen next.

Up until that point, I didn’t really have a lot of time for reflection along the way as to how good or bad the book was, or whether or not folks would actually buy it once it went on sale. But after publication, and especially when I was helping marketing the book, I did feel some degree of nervousness and anticipation as to what developers out in the wild would think of something that I had dedicated a good chunk of the past half year writing.

But, more on that in the last post in this series…

Backbone.js with React Views

2014-11-22T00:00:00+00:00

Cross-posted from the Formidable Labs blog.

On Nov. 18, 2014, I presented a talk on “Backbone.js with React Views” at the inaugural Seattle ReactJS Meetup.

Backbone.js

Backbone.js powers some of the largest web applications on the Internet and we use the framework extensively at Formidable Labs. It provides simple and flexible abstractions that developers can use to organize and create large-scale applications.

At the same time, Backbone.js application development runs in to complexities with issues such as:

Heavy handed DOM wiping and reflows.
Preventing memory leaks and dreaded “Zombie Views”.
Composing and aggregating child views correctly.

React

Enter React. React is a UI (view) layer for web applications that uses virtual DOM rendering and a modular architecture for creating reusable, web components. It offers some exciting new possibilities as a replacement for Backbone.js views, while keeping the rest of the framework pieces that Backbone.js does very well.

Backbone.js with React!

In my talk, I walk through a simple application, “Notes” (live demo), that uses Backbone.js with React views. The full source is available from one of our GitHub repos.

The talk slides go in to more detail, but some of the interesting results we get with a React-based view layer include:

A slimmer overall minified bundle. With React as the view layer, we can remove dependencies on jQuery and Underscore.js.
Server-side rendering. This is a huge feature, and actually quite easy with React. Even better, if you have a single-page JavaScript application for your site, React can bootstrap the HTML of the page server-side and then startup the app without re-rendering any HTML!
… and some other pretty cool stuff!

To dive into the talk, hop on over to the presentation website. Or, for the more traditional folks, the talk SlideShare deck is available (with downloadable PDF).

Making JavaScript Tests Fast, Easy, & Friendly

2014-08-11T00:00:00+00:00

Cross-posted from the Formidable Labs blog.

A few weeks ago, I presented a talk on “Making JavaScript Tests Fast, Easy, & Friendly” at the CascasdiaJS 2014 conference in Portland, OR. The talk looks beyond the traditional technical details of wiring up test frameworks and focuses on creating processes and environments that help developers actually write and run application tests.

Making the Tests Faster, Easier, Friendlier

The motivation for the talk comes from our work with developer teams of various sizes at Formidable Labs, where we have observed the challenges, successes, and failures of test development as a part of the overall development workflow. Essentially, these experiences can be culled down to the following:

If tests aren’t easy to write, then developers won’t write them.
If tests aren’t fast to run, then developers won’t run them.
If the organizational culture isn’t test-friendly, then developers won’t write or run the tests (and they’ll probably be unhappy to boot).

The talk dives into some tips and tricks to address these three broad goals as follows:

Easy
1. Set a Foundation: Build an infrastructure everyone can use.
2. Lower Barriers to Entry: Help new developers get acclimated with code and mentorship.
3. Write it Down: Document everything test-related and keep things up-to-date.
4. Teach & Learn: Provide onboarding and small, introductory test assignments.
Fast
1. Know What to Look For: Focus on the application behaviors most likely to be slowing down your tests (e.g., network and waits).
2. Find the Slow Pokes: Use your test tools to identify what is slow during tests.
3. Fake It: Programmatically “fake” the parts of your application that are slow during the tests.
4. Offer Shortcuts: Provide developers with subsets of tests to run when the whole thing is too slow to regularly run.
Friendly
1. Find Champions: Encourage and promote developers who are considered project-wide testing leads.
2. Win the Higher-Ups Over: Learn to speak your managers’ language to get top-down support in your organization for developer time spent writing and running tests.

These are just some of the tips that we’ve found useful for making testing better in our client projects at Formidable Labs. But, there are surely other good techniques for building a strong culture of testing in other organizations. (And, we’d love to hear your experiences in our comments below!)

Media

The full video of my talk is available from YouTube. As an aside, the CascadiaJS YouTube Channel has videos from the other (fantastic) talks at this year’s conference.

For those just looking for the slides, you can try the presentation website (with live executable code samples!) or the more traditional SlideShare site, where the final PDF resides.

chai-jq - jQuery Assertions for Chai

2013-11-27T00:00:00+00:00

A New jQuery Plugin for Chai

Chai is a wonderful JavaScript test assertion library that I rely on extensively (usually using it in conjunction with the Mocha test framework library). Chai has an intuitive, natural-language approach to assertions that enables you tests to read almost like an English narrative. Chai also supports plugins which extend the base assertion API.

Chai is quite often used for frontend JavaScript testing, and specifically for testing jQuery elements in an application web page. There is an existing chai-jquery plugin for Chai, which is extensive and really quite neat. Unfortunately, it has a few issues with overriding built-in Chai assertions like have and length in ways that changes the underlying Chai API.

With that motivation in mind, I hacked together a quick jQuery plugin, chai-jq for Chai that has a separate $-prefixed namespace to avoid collisions with existing Chai assertions, and put the project up on GitHub.

The chai-jq Plugin

The chai-jq plugin has full documentation at the project website, including installation instructions. The plugin works in all of the following environments:

Browser: Via a standard <script> tag include.
Browser + AMD: Via an AMD library like RequireJS.
Node.js + JsDom: In Node.js using the JsDom browser environment emulator.

In terms of what chai-jq provides, here is a brief tour of the API (stolen from the project docs):

`.$visible`

Asserts that the element is visible.

expect($("<div>&nbsp;</div>"))
  .to.be.$visible;

See: http://api.jquery.com/visible-selector/

`.$hidden`

Asserts that the element is hidden.

expect($("<div style=\"display: none\" />"))
  .to.be.$hidden;

See: http://api.jquery.com/hidden-selector/

`.$val(string|regexp)`

Asserts that the element value matches a string or regular expression.

expect($("<input value='foo' />"))
  .to.have.$val("foo").and
  .to.have.$val(/^foo/);

See: http://api.jquery.com/val/

`.$class(string)`

Asserts that the element has a class match.

expect($("<div class='foo bar' />"))
  .to.have.$class("foo").and
  .to.have.$class("bar");

See: http://api.jquery.com/hasClass/

`.$attr(name, string)`

Asserts that the target has exactly the given named attribute, or asserts the target contains a subset of the attribute when using the include or contain modifiers.

expect($("<div id=\"hi\" foo=\"bar time\" />"))
  .to.have.$attr("id", "hi").and
  .to.contain.$attr("foo", "bar");

See: http://api.jquery.com/attr/

`.$prop(name, value)`

Asserts that the target has exactly the given named property.

expect($("<input type=\"checkbox\" checked=\"checked\" />"))
  .to.have.$prop("checked", true).and
  .to.have.$prop("type", "checkbox");

See: http://api.jquery.com/prop/

`.$html(string)`

Asserts that the target has exactly the given HTML, or asserts the target contains a subset of the HTML when using the include or contain modifiers.

expect($("<div><span>foo</span></div>"))
  .to.have.$html("<span>foo</span>").and
  .to.contain.$html("foo");

See: http://api.jquery.com/html/

`.$text(string)`

Asserts that the target has exactly the given text, or asserts the target contains a subset of the text when using the include or contain modifiers.

expect($("<div><span>foo</span> bar</div>"))
  .to.have.$text("foo bar").and
  .to.contain.$text("foo");

See: http://api.jquery.com/text/

`.$css(name, string)`

Asserts that the target has exactly the given CSS property, or asserts the target contains a subset of the CSS when using the include or contain modifiers.

expect($("<div style=\"width: 50px; border: 1px dotted black;\" />"))
  .to.have.$css("width", "50px").and
  .to.have.$css("border-top-style", "dotted");

See: http://api.jquery.com/css/

The chai-jq plugin is just getting off the ground and is likely still rough around the edges. Any issue reports, pull requests, and feedback are most welcome.

Writing a Technical Book, Part 2 - Authoring

2013-11-25T00:00:00+00:00

Writing a Book

This is the second in a series of posts on my experiences writing my first technical book, Backbone.js Testing, published in July, 2013. In this article, I reflect on the task of authoring the core of the book - from writing the first pages to finishing off the last round of technical edits and submission of the full draft of the book to my publisher.

A rough overview of salient tips and experiences from this part of the book-writing journey includes:

Make sure to have a reliable backup of your work - expect to lose all your files (and your computer).
Before writing anything, plan and outline everything.
The writing process will take much longer than you think it will.
The writing process is also tiring, stressful, and boring.
Technical review will bruise your ego and leave you with a much better book.
Once you get most of the way through the drafting process, you (the author) hold the power in your relationship with your publisher.

Where we Left Off

In my previous post on starting a book project, I discussed how I got roped into writing a technical book and the process from initial contact from the publisher to signing a book contract. As part of the negotiations, we agreed on the following ultimate chapter outline (with estimated page count and due dates):

Setting up a Test Infrastructure (10 pages) - Jan. 8, 2013
Creating a Backbone.js Application Test Plan (8 pages) - Jan. 16, 2013
Test Assertions, Specs, and Suites (20 pages) - Jan. 28, 2013
Test Spies (7 pages) - Feb. 4, 2013
Test Stubs and Mocks (15 pages) - Feb. 11, 2013
Headless Web Testing (12 pages) - Feb. 19, 2013
Appendix A: Other JavaScript Test Frameworks (8 pages) - Feb. 25, 2013

As a spoiler for the rest of this post - I came nowhere near making most of the chapter deadlines.

Starting on the Book

After signing the contract in mid-December, I found that I suddenly had my first chapter due Jan. 8 and subsequent chapters following quickly after that. Shortly thereafter, I dug in and started getting my authoring and coding tools set up, enhancing my book outline, and dove into writing the first chapter.

Nuts and Bolts - Storage and Backups

As a purely logistical matter, my book chapters were authored as Microsoft Word documents and have images in PNG format. The source code examples are in HTML, JavaScript, and CSS. And, being a consummate techie, I knew that I needed a reliable and flexible digital storage system. My main goals for this setup were:

Provide a disaster recovery solution if my laptop dies.
Provide “checkpoints” to go back to earlier versions of my work to find cut text and alternative previous drafts.

The source code samples were going to be released as an open source project and were pure, usable source code, so the storage and backup solution was quite obvious - Git and GitHub. You can find the ultimate home of the code samples at the book’s GitHub repository.

For the book’s textual sources however, Microsoft Word documents typically don’t play well with Git, and I the book’s text as an open source project. Nonetheless, after reviewing a lot of alternative storage systems (Dropbox, Amazon S3, etc.), I decided to go ahead and use Git with a private repository. My rationale was that Git still provided checkpoints to early MS Word document drafts, even if I couldn’t use all of the normal Git tools with the documents.

I cannot emphasize enough the importance of backups when writing a book, and preferably something that checkpoints drafts. There are a variety of bad things that can happen to both your computer (crashes, etc.) and documents (like MS Word files becoming corrupted). In my case, I had my laptop stolen in May during the middle of the authoring process - I didn’t lose any of my book progress (code or text) as I was able to easily restore my existing work on another computer with just a few downloads.

Planning and Outlining

Turning to the substantive end of kicking off the book, I began reworking and expanding on my detailed book outline that I had begun as part of the book scoping and negotiation process with Packt.

The original outline had about one page of bullet points per chapter. I turned this into a series of sections and sub-sections for Chapter 1. Then, I filled in as many substantive bullet points as I could for each section and sub-section to essentially write the structure and outline content for the entire first chapter before diving in and actually writing text.

I strongly recommend detailed outlines for any large writing project as it forces you to collect your thoughts, hone the overall message and protects you from writing lots of text in directions that will later be cut in the editorial process. Or, putting in software development terms, you don’t start a large web application by writing code top-to-bottom - instead, you create a general design and architecture first (say, model-view-controller or something), and then you plan and implement the parts. The same goes for writing, if you want your end result to be cogent and well crafted.

This was also a chance for me to revisit the vision for the book as a whole, as the first chapter will really start limiting and focus the world for the book. My book was about using testing technologies (frameworks, assertions, and fakes) for a specific web framework, Backbone.js (with models, views, controllers, and other framework components). The big decision made in the original outline was to split chapters by different test technology, and not by web framework component.

At the same time, I needed to weave each testing lesson back to the Backbone.js framework (the book title is, after all, Backbone.js Testing), so I decided to create a reasonably complex Backbone.js sample application to use throughout the book. The application is a simple note-taking web page, named Notes, and I tested different parts of Notes in each chapter using the specific chapter’s test technology.

I think this approach was ultimately successful in unifying the test concepts with Backbone.js application fundamentals, but it also required a lot of upfront work, as the Notes sample application essentially needed to be coded and tested very early on, before most of the chapters were written.

And Off We Go (The First Chapter)

With the Chapter 1 outline prepared and Git repositories ready to go, I started writing the first chapter, “Setting up a Test Infrastructure”. The chapter essentially comprised downloading all of the libraries needed to run and execute the book samples and create a “Hello World”-ish test infrastructure.

Chapter 1 came together reasonably easily on a fairly relaxed scheduled, as it wasn’t due until the first week of January. I had time to create the example application and tests that would be used throughout the entire book. I had a little bit of acclimating to formatting the document per required guidelines and generating acceptable screenshot image files, but it wasn’t too bad.

I turned in the chapter draft on the scheduled due date, and thus started my first milestone for actually writing a book!

Digging In (The Other Chapters)

Delivering the first draft of Chapter 1 on time and in reasonably good shape did feel like an accomplishment. However, even at that point I realized that the original negotiated schedule was going to end up being far too aggressive in light of having a (more than) full time “day job” and trying to keep some semblance of a life outside of technical-related matters.

As it turns out, an outside factor then ended up blowing the schedule away. During the time I was writing the first chapter and after signing on to the book project, my wife interviewed for, and accepted, a new position in Seattle. When everything was finalized, I suddenly was faced with a cross-country move from Washington, DC to Seattle, WA in mid-February.

I raised the issue with my publisher upon turning in my first chapter, and we agreed on a revised schedule, effectively spreading out the due dates for chapters 2 and 3 through the end of February. We left the rest of the chapter deadlines as more of a “cross that bridge when we come to it”-type thing.

Over the next few months, I dug in and cranked out the chapters in my nights and weekends (with a generous interruption for moving to Seattle). The writing was relatively straightforward in terms of subject matter and getting it down on paper, but the process itself was quite difficult and taxing.

If I were to categorize the toughest parts of writing out the initial chapter drafts during this big push, the top items would be:

Exhaustion: I was pretty tired when writing. I started working on the book many nights after 10:00 pm or before 4:00 am. It’s hard to think when you’re tired, and harder to write.
Scope Creep: The chapters ended up being longer than originally anticipated, making the work take longer. I tried to keep to the proposed outline’s estimates, but there were some topics that just couldn’t be covered in the expected number of pages.
Reviewing Work: Authoring a first draft is not the end of your work. After a chapter submission, the publisher would forward the chapter draft to the technical reviewers, who added comments or made suggested changes. Then, the publisher would present their work back to me for chapter revisions. There was no real schedule for this, and it wasn’t factored into the underlying schedule of chapter first drafts, yet ended up being a significant portion of work.
Time Zones: The majority of the publisher’s editorial staff was based out of India, meaning that communication almost always took a full day and was a half-day out of sync.

At the end of the day, writing a book is tedious work and not always that fun for these and other reasons. The most important traits and tendencies I found to counter these obstacles and difficulties include the following:

Write and Review Quickly: I can write reasonably well-organized text fairly quickly. Despite my lofty hopes of an erudite tome for the ages, the real goals for a technical book is something that is correct, accessible, and not horrendously dry. (The exception to this rule is Larry Wall’s “Programming Perl” book, better known as the “Camel Book”, which is probably the most humorous and well-written technical book I’ve ever read.) From my experience, when I could actually get the time to write, I was able to reliably crank out drafts and revisions to my chapters.
Don’t Get Bored Easily: The subject matter of technical books is fairly dry. And, given how long you will spend engrossed in a single topic, it’s bound to be quite boring at times. It thus helps to have really good focus on a single topic and being able to push through in a methodical manner. I do pretty well with the latter - the topic of my book wasn’t always engrossing, but at least in chapter-long increments, I was able to always move forward and continue.
Detailed-Oriented: My publisher didn’t have the absolutely best reputation for the textual quality of their books. Apparently repeated, grievous editing mistakes and oversights somehow find their way into at least a number of the final published books, if the Amazon book reviews are to be believed. In my case, I made it my duty to proofread the heck out of everything I wrote and again at every review opportunity. My personal preference for proofing was to print out paper versions of each chapter, and attack it with a red pen, old-school style. However you slice it, it is ultimately the author’s responsibility to keep the book in good shape. And, as the next series in the post will discuss, it is also important to keep an eye on your editor / publisher to make sure they’re not adding errors.

Technical Review

One of the benefits of working with a publisher, as opposed to self-publishing or the like, was the recruitment of some very capable and smart technical reviewers. Packt found two software developers well known in the community for Backbone.js and frontend testing generally.

After I submitted each first draft chapter to Packt, they would assign the chapters to the technical reviewers to add comments and make suggested text changes. Then, I would receive the in-draft and separate comments from the reviewers and rework each chapter into the second draft.

Going through technical review is a humbling process - not only did my reviewers point out all of the errors in the chapter text, they also gave qualitative ratings for each chapter, many of which were quite low for my first drafts. One reviewer in particular was often quite critical in overall assessment and the numerous comments he would pepper my drafts with. Yet, I couldn’t have been more happy with all of the ego-bruising / “this is horrible”-type criticism – the feedback was invaluable for correcting errors, getting a sense of my overall message, and knowing that at least two highly intelligent developers had worked through the text.

As far as work went, simple errors and omissions pointed out by the technical editors were easy to correct. However, some were quite extensive to revise. For example, a reviewer suggested that I should re-organize the entirety of my example code’s file structure. Evaluating the comment, I realized that he was right, and spent a good deal of time retooling everything in the code and in the book examples and text. I think the book ended up much better overall for the change, but at the time it was really a pain to change.

Getting Through the Editorial Process

I managed to get drafts of the first four chapters submitted by mid-March. After my submission of Chapter 4, my publisher notified me that my editor was changing roles in the company and that I would be assigned a new editor.

Right out of the gate, the new editor notified me that my chapters had deviated from the estimates in the outline and that I would have to cut down my previous chapters. The new editor presumably reviewed all the chapters submitted up until that point and figured it was time for house cleaning.

I replied rather strongly that to try and go back and retrofit already completed work at that time would be time-consuming, difficult, and likely make the overall book worse for the changes. At the time I was writing Chapter 5, and the notion of stopping everything to go back to old chapters was just dreadful.

As an aside here, it’s worth mentioning the power dynamics of the author and publisher. In my situation, Packt essentially provided no money upfront, a quite meager advance on the completion of the entire book, and was essentially relying on my good graces to finish the thing. Although technically they could take my partially finished work and have another author pick it up mid-stream if our relationship truly went south, the practical logistics of them being able to actually go through with that would have been very difficult.

My intuition is that many book projects similarly set up to mine fail midway through the process due to external constraints or pressures. Put another way, if something came up on my end, there was really no way for my publisher to force me to finish. And, once I was over halfway through my first draft of the book, the publisher had strong incentives to see me complete it.

So, when I pushed back on the new editor’s extensive rewrite proposal with an alternate plan, Packt was quite amenable and the editor backed down pretty easily.

To address with the overall size of the book, I cut the planned Appendix (a survey of related technologies to by book’s core subject matter) instead of revising existing chapters. The Appendix was already going to be a pain to research and write, and figured offering it up as a “sacrifice” to reclaim page count would be the best way to speed things along.

Submission!

I submitted the final drafts of all chapters at the end of April 2013. I also finished and submitted meta-level drafts of things like my biography and the preface to the book.

As a foreshadowing event, I also reminded my editors that they had agreed to change the title of the book from “Backbone Testing” to “Backbone.js Testing” (just adding the three characters “.js”).

The editors accepted my submission and gave me a rough outline of the process that would unfold from there on out to actually publish the book, a story that we will dive into in next post in this series.

Learn Frontend Testing

2013-10-18T00:00:00+00:00

Learn Frontend Testing

As a part of Formidable Labs’ series of development education events, I led a “Learn Frontend Testing” workshop on Oct. 16, 2013 for the SeattleJS meetup group. My slides are available at the following locations:

Web Presentation: A live, navigable reveal.js website. (Note: use the space bar to advance slides and arrow keys to navigate.)
PDF: A download-able PDF.

And, here’s an embedded format:

Workshop Topics

My presentation walks through testing some very basic JavaScript application code in the following steps:

Install all of the necessary test libraries and create a test driver HTML page to hook up the application libraries and run the whole test harness.
Create Mocha suites and tests.
Write assertions to verify application behavior with Chai.
Spy and stub out application behavior with Sinon.JS.
Automate all of our tests with PhantomJS to run them from the command line without a web browser.

The test stack the presentation uses is:

Mocha: Test framework for organizing and running our tests.
Chai: Natural language-focused test assertion library.
Sinon.JS: Test fake library providing spies and stubs for observing and mutating application behavior during tests.

The test concepts and technology stack is the same one I use in my book, Backbone.js Testing. You can see the additional test samples beyond those contained in the presentation at: http://backbone-testing.com/.

Try it Out!

All of the source code for the presentation (include our finished tests) is available at the workshop’s GitHub repo and can be installed using git:

$ git clone https://github.com/FormidableLabs/learn-frontend-testing.git

Packt Holiday eBook Sale

2013-10-16T00:00:00+00:00

Packt Holiday Discount

Packt Publishing is offering a pretty respectable 50% off all of their eBooks through Oct. 17th. They’ve got a Columbus Day special code of COL50 to enter during checkout. (For those not quite comfortable with Columbus Day, consider it a Bartolomé Day discount.)

Backbone.js Testing - Book Contest

2013-09-09T11:30:00+00:00

Win Free Copies of Backbone.js Testing

I’m pleased to announce that Packt Publishing is organizing a contest to give away copies of my book, Backbone.js Testing!

Five participants stand a chance to win digital copies of the book.

Book Overview

Backbone.js Testing covers Backbone.js test architecture and development, and in brief summary will help you:

Create comprehensive test infrastructures.
Understand and utilize modern frontend testing techniques and libraries.
Use mocks, spies, and fakes to effortlessly test and observe complex Backbone.js application behavior.
Automate tests to run from the command line, shell, or practically anywhere.

Enter the Contest, Win the Book!

All you need to do is leave a comment below with what interests you most about the book. You can get a good sense of the book’s content by looking at it’s product description on Packt’s book page.

Deadline: The contest will close in one week’s time. Winners will be contacted by Packt via email, so be sure to use your real email address when you comment! (Alternately, after leaving a comment you can instead email me at ryan@loose-bits.com if you don’t want your email address to publicly appear on this page.)

Sphinx Bootstrap Theme 0.3.0 - Bootstrap v3 and more!

2013-09-08T11:30:00+00:00

Bootstrap v3 comes to Sphinx

The Bootstrap Theme for Sphinx has hit a new milestone with version 0.3.0, bringing in version 3.0.0 of the Twitter Bootstrap library.

[![Sphinx Bootstrap Theme v0.3.0][img_sbt]][img_sbt] [img_sbt]: http://loose-bits.com/media/img/2013/09/08/sbt.png

A lot has changed since the last v0.2.0 blog post, the most conspicuous being that you can now specify to use either Bootstrap v2.3.2 or v3.0.0 via the theme options:

html_theme_options: {
    # Choose Bootstrap version.
    # Values: "3" (default) or "2" (in quotes)
    'bootstrap_version': "3",
}

Thanks to the awesome work by MiCHiLU, the Sphinx Bootstrap Theme can easily switch between Bootstrap v2.x and v3.x with full support for all of the features, including Bootswatch CSS extensions.

Changes: v0.2.0 - v0.3.0

It’s been a while since I have posted updates for the theme, so here is the laundry list of all of the goodness added to the theme since my last blog post.

v0.3.0

Add Bootstrap v3.0.0 with legacy option for v2.3.2. (@MiCHiLU)

v0.2.9

Add navbar_links theme option. (@newgene)
Add navbarextra block in “layout.html”. (@grncdr)

v0.2.8

Sphinx compatible Sidebars. (@russell)
Topnav sidebarrel can now be disabled. (@russell)
Topnav page nav menu can now be disabled. (@russell)

v0.2.7

Add custom nav bar links. (@russell)
Fix wrapping of line numbers in code includes. (@russell)
Truncate long page titles in navigation bar. (@aababilov)

v0.2.6

Use network path for Bootswatch (@nail)
Switch from distribute to setuptools. (Suggested by @thedrow)

v0.2.5

Search page styling. (@russell)

v0.2.4

Adjust the max width of field lists. (@russell)
Update to Bootstrap v2.3.2.
Navbar search box now uses bootstrap search-query class. (@russell)
Field-list tables now have an inherited width. (@russell)

v0.2.3

Put navbar within a container. (@inducer)
Add navbar_site_name for renaming site nav. tab. (Suggested by @inducer)

v0.2.2

Better literal markup handling for Bootstrap code formatting. (@russell)
Scroll window when jumping to an anchor. (@russell)

v0.2.1

Fix code styling collision for cross references and inline code blocks. (@russell)

Happy Theming

I’ve been humbled and excited by all of the great community work on the theme, adding much needed functionality and flexibility. Please keep the pull requests coming and help us continue to develop the friendship between Bootstrap and Sphinx!

Writing a Technical Book, Part 1 - Starting the Project

2013-08-04T11:30:00+00:00

Finding and Starting a Book Project

I published my first technical book, Backbone.js Testing, in July, 2013. Now that I am coming down off the high of actually getting the book out to the world, I thought that I would take a little bit of time to reflect on my journey and write a series of posts about the adventures and tribulations of writing a technical book.

In this first post, I will look back on how I got roped into becoming an author, planning and scoping a prospective book outline, and signing a contract to begin work on the book.

Why Write a Book?

Writing a book is an enormous time commitment, puts pressure on your personal and work obligations, and has little or no chance of being financially a worthwhile endeavor.

So, why do it?

The short answer is that I liked the proposed topic, I was already interested in writing, and Packt Publishing had some fortuitous timing in pitching the project.

A good, challenging topic

Frontend web application testing has been a topic of keen interest and frustration for me for quite some time. Historically, testing JavaScript in web pages has been an arduous task, with very few good options and enormous amount of pain in any solution. At the same time, there is enormous need for this type of testing as more and more logic is pushed from a traditional backend application (e.g., a Django or Rails app) to the frontend.

In my day-to-day work over the past few years, I’ve noticed that the biggest holes in application test coverage and sources of most actual bugs have been in frontend JavaScript code. Accordingly, I have now spent a good amount of time implementing and writing about testing on the client side. In 2011, I wrote two posts on an early frontend testing solution I came up with using Env.js and Rhino. The technologies were unreliable and hacked together, but at that point, at least it was something.

Fast-forwarding to today, with the rapid rise of Node.js on the backend and a whole host of new and exciting browser libraries, there are a lot more options for frontend testing. At Curiosity Media, we had a large Backbone.js web application that we were able to get a solid, modern test harness around using Mocha, Chai and Sinon.JS.

I missed writing

By way of a bit more background, software engineering is my second career. Before becoming a full-time geek, I was an intellectual property attorney, focusing on mostly computer software and hardware matters.

While some folks think that the life of a lawyer mostly involves standing up and arguing in court (e.g., “I object!”), the reality is that as a junior associate attorney, you spend nearly all of your time reading and writing legal documents of some type or another. Moreover, as a patent attorney, most of your time is spent writing patent applications and related legal documents. And, patent writing is essentially technical writing, albeit with a bit of a legal bent.

So, coming back to my current life as a software engineer, while I occasionally write blog posts and articles, it had been several years since any substantive writing project. And, I kind of missed it.

And, it seemed like the right time

With that background, Packt Publishing had the good timing to approach me with a book proposal for testing Backbone.js applications in late October of 2012.

I’m not sure the process was particularly selective in my case, as Packt has a reputation for somewhat aggressive recruitment strategies for book authors and reviewers (e.g., they send a lot of emails). But, the subject matter was a good fit for my frontend testing interests, and the book size seemed to make the project tractable.

Choosing a Publisher

I didn’t really “choose” a publisher, as I was not seriously considering a book project until Packt presented the packaged concept to me. As such, I didn’t go through the steps of deciding to write a book, then trying to choose a publisher and whether or not to self-publish - but the topic is one of great interest to folks I have chatted with incidentally about technical book authorship.

I will get more into the intricacies of working with my specific publisher throughout the rest of these posts. As to the broad question of working with a traditional publisher versus a self-publishing service like Leanpub, the things that my publisher helped with that were really valuable includes the following:

Technical Reviewers: Packt recruited two community JavaScript developers to review the technical content for accuracy and suggestions. The reviewers for my book were absolutely top notch, and provided enormously useful feedback and comments.
Content Editing: A normal publishing house will provide proofreaders and content editors to fix language and writing errors. Packt does not always have the best reputation in this regard, so in my authoring, I viewed the final responsibility for all editing being my own. At the same time, the editors and proofreaders did find several misspellings, etc. over the course of my drafts.
Layout and Formatting: A publishing house will take care of all of the formatting issues, particularly those that require manual editing after the automated processes finish producing a print-worthy PDF. I’m not sure how far along the self-publishing services are in this regard, and in any case I didn’t have to do any of this work - I just had to review and approve the print-ready PDFs.
Publishing and Distribution: Packt takes care of the publication in paper and eBook form. They also got the book listed on Amazon and other third party sites with proper metadata, etc. (although they needed a couple of rounds to get it right).
Marketing: Traditional publishing houses will also market your book via email, website articles, and other means to get the word out in ways you won’t always be able to do yourself. In my case, Packt is in the process of creating an article around some of the book content to publish on their site and drive interest. Then, they’ll launch and email and social media campaign. I have no idea how that will go, but it should be interesting to see if it actually drums up more interest.

From what I can intuit, the disadvantages of going with a traditional publisher include:

Control: Someone else is approving drafts and driving the entire process.
People and Resources: As an author, you are one of many to the publisher, and if they haven’t staffed your book properly, you can end up waiting a long time for the project to move forward. As one example in my case, after I finished all of the drafts for the book, Packt sat on the book for an entire month without progress as they had some internally shuffling of editors and staff.
Pressure: With a traditional publishing house, you are on someone else’s time line, and in Packt’s case, they drove really hard to getting drafts in. At the same time, some authors might enjoy having a third party provide incentive to keep with the project.

Would I self-publish my next book? I’m not sure. (I’m still not sure I’ll ever write another technical book.) But, were I to go down the authoring rabbit hole once again, I would strongly consider self-publishing given the advances and control you have over the entire process.

In any case, as far as my story here goes, I ended up with Packt. And I will say that in my case, the book was most likely finished much faster with Packt than with any of the other publishing options or companies.

The Book Proposal

My first contact with Packt was from an “author relationship executive”, which I think is something like an author recruiter. The tentative project outline Packt proposed was for a 90-page book with five chapters covering Backbone.js application testing concepts and tools.

Once I expressed some interest, the author relationship executive passed me on to a “commissioning editor”, who is the main editor for the book. I would later find out that there are various editor roles, as well as many different people iterating through those roles during the course of writing the book.

The commissioning editor answered my initial queries and communicated the basic logistics of the book - drafts are written in Microsoft Word, the loose drafting and editorial process, etc. The expected page count got ramped down to 80 pages, which Packt considers a “mini” book - a short, concise book on a specific topic.

The editor then sent me a helper “outline kit” for me to put together a target outline for the book, which added up to 60 pages of content. The goal was to fill in a per-chapter outline for each of the five proposed chapters. I did this, then proposed an additional chapter (on automated web testing) and an appendix. The Packt team liked the outline and my suggested additions, and bumped the allowed page count to 80 pages.

After a bit more back and forth, I finished a final draft of the book outline, which Packt approved, and then we moved forward to the contract stage. The takeaway from the proposal process was that we ironed out the chapter structure and length, which would then constrain the writing process and define what parts were going to be due.

The Book Contract

After approving the book proposal, Packt sent me a book contract for review. The contract most entailed expected details of obligations for the author in writing the book, what happens if the author fails to deliver chapters or stop working on the book, etc.

The legalese

Although sometimes scary and written in abstruse legalese, a contract is essentially a negotiation in paper form. An important thing to remember that any point is up for grabs, modifiable or removable. With that in mind, I dove into the contract and started dissecting some of the points that might need changes.

One of the biggest issues with the contract was that Packt put in a clause titled “Option on future Works”, which basically obligated me to give Packt the first right to publish my next two books after the one we were discussing.

The Author shall give the Publisher the first opportunity to read and consider for publication the next 2 works on any computer-related subject that the Author seeks to have published.

To me, this clause was clearly ridiculous - it is one thing to agree to a book contract when I’ve never written one before nor worked with Packt before, but it’s quite another to say the first three books I would write would be at Packt’s publishing discretion. So, I asked Packt to remove the clause entirely.

A separate issue that wasn’t directly addressed in the contract was the licensing of the code samples that accompanied the book. The code samples technically fall under the copyrights of the book generally. However, I wanted to release all of the code samples as an open source project, so that I could easily publish the code, get community improvements and bug fixes, and get the code out earlier than the book. So, I asked that Packt allow all of the code samples to be separately published to GitHub as an open source project under the MIT license (the most common license for the components I was using).

Packt agreed to both changes without any further issue or push back.

The time frame

Packt set up an incredibly aggressive time frame for completing the first drafts of the book - something short of two months. As I had planned a December vacation, I pushed back a little bit and got the following schedule (assuming a starting date of January) for my chapter drafts:

Setting up a Test Infrastructure - Jan. 8, 2013
Creating a Backbone.js Application Test Plan - Jan. 16, 2013
Test Assertions, Specs, and Suites - Jan. 28, 2013
Test Spies - Feb. 4, 2013
Test Stubs and Mocks - Feb. 11, 2013
Headless Web Testing - Feb. 19, 2013
Appendix A: Other JavaScript Test Frameworks - Feb. 25, 2013

In retrospect, this schedule was insane and was in no way feasible given that I had a full time job and was expecting to write on nights and weekends. But, the story of the how all of the deadlines (both mine and my publisher’s) slipped over the course of the book is for another post.

Suffice to say that I should have padded the draft chapter due dates in the contract by at least a factor of 1.5 from what I have here. Being a first time author, I really had no real idea how long things would actually take, and my intuition now tells me that drafting chapters is slower your first time.

The money

The contract also specified the remuneration details - advance and royalties. Packt’s royalty rate is 16% of all book revenue, and is explained in further detail on their royalties page.

The advance amount was pretty minimal - $600 - with one-fourth due on completion of all drafts, and the rest due on publication. As an advance, you get to keep this amount, but it cuts against any royalties from the book after publication until it is accounted for.

Unless you happen to have an amazing prolific topic, money should not be a large motivating factor in choosing to write a technical book as you will invariably be disappointed. You’ll note that money was nowhere in my discussion of motivations to write a book in the above sections.

Software developers are fairly well-paid; authors tend not to be. Put another way, I have never done the math as to how many hours I spent writing the book versus how much I could have made in side software consulting gigs, nor do I plan to in the future - the opportunity cost in terms of pure dollars would be really depressing.

At the same time, I did find so many benefits from writing a technical book - learning a subject in depth, working with development communities in depth, and just meeting random folks along the way who were interested in the topic or have even read the book or online code samples.

Off to Write a Book

Having considered everything and deciding to jump in, I signed the revised book contract with Packt in December, 2012 and officially became on the hook for writing a book. Packt then sent me an “author bundle” of documents, which included style and writing guidelines and templates for the draft chapters.

Packt uses Microsoft Word (or OpenOffice) documents for drafting and relies on custom styles to eventually reformat the book for publication. I would much have preferred to use a text-based document approach such as Markdown or LaTeX, because there is so much more control over the document content and structure, and code examples are much more likely to be properly preserved.

And with these reference materials and a loose outline of my topic in hand, I began writing the draft of my first chapter…

(And I’ll pick up the authoring adventure from this point on in the next post in this series.)

Book Announcement - Backbone.js Testing

2013-06-13T11:30:00+00:00

Backbone.js Testing Book

I am very pleased to announce that I am in the last stages of publishing Backbone.js Testing - a book on Backbone.js test architecture and development due in final form around the end of June.

A brief abstract of the book’s content is available from the Packt Publishing website:

Frontend web applications are soaring in popularity and the Backbone.js library is leading this charge with a modular, lightweight approach for organizing JavaScript web applications. At the same time, testing client-side JavaScript and Backbone.js programs remains a difficult and tedious undertaking.

Backbone.js Testing brings sensible practices and current techniques to the challenges of Backbone.js test development. The book introduces fundamental testing concepts, comprehensive test infrastructure design, and practical exercises to easily and systematically test modern JavaScript web applications.

The book progresses from Mocha test suites and Chai assertions to advanced test mocks and stubs with Sinon.JS. The requisite libraries and utilities are introduced with in-depth examples and best practices for integration with your applications. The book guides you through the test planning and implementation processes for your application models, views, routers, and other Backbone.js components.

Backbone.js Testing gives you the tools, examples, and assistance to test your Backbone.js web applications thoroughly, quickly, and with confidence.

To provide a better idea of the book’s structure, the tentative chapter titles are as follows:

Setting up a Test Infrastructure
Creating a Backbone.js Application Test Plan
Test Assertions, Specs, and Suites
Test Spies
Test Stubs and Mocks
Automated Web Testing

All of the code examples in the book are provided online at the website backbone-testing.com and as an open source GitHub repository.

The Road to my First Book

Application testing is difficult, and frontend JavaScript testing can often be a downright nightmare. After years of working through difficulties in frontend testing practices and technologies, I have gained a lot of experience through the various trials and tribulations bringing me to my current role in architecting (and testing) large Backbone.js applications.

Packt Publishing approached me towards the end of last year about authoring a book on the subject and I signed on. My hope with the project is that I will be able to provide a resource to reduce the pain in frontend tests with a structured programming guide and practical examples using various (exciting and interesting) modern testing libraries.

It has been a stressful, hectic, and wonderful experience writing the book, and I’m really looking forward to completing the project. And, I will be happy to be able to reclaim my nights and weekends that have been devoted to the book over the past couple of months.

After the book is published, I plan on writing a series of posts about authoring a technical book, with all of the ups and downs that the process entails. In the meantime, I just have to proof each chapter one final time and it should be off to the presses, so to speak!

Node.js in Production

2013-05-10T11:30:00+00:00

Node.js in Production

Curiosity Media runs the world’s largest Spanish learning website, SpanishDict.com, backed by many different Node.js applications. Since rolling out our first Node.js applications over a year ago, we have collected a fair amount of good and bad operational experiences.

I created a talk about our journey with Node.js for the Seattle Node.js Meetup group’s May 8th, 2013 event. My slides are available at the following locations:

GitHub Site: A live, navigable reveal.js website. (Note: use the space bar to advance slides and arrow keys to navigate.)
SlideShare: More traditional format, with download (PDF) links.

And, here’s an embedded format:

Five Node.js Production Tips

My talk discussed the following five topics:

Know when to Node: Should you use Node.js?
Keep up with Node: Stay up to date with Node.js and libraries.
Design for failure: Fail and recover at multiple levels.
Isolate services: Separate resource and failure classes.
Analyze everything: Data drives problem discovery and action.

There is, of course, much more to running Node.js in production, but hopefully my talk and slides provide a bit of insight into some of the considerations to keep in mind when taking a Node.js application to production.

Sphinx Bootstrap Theme 0.2.0 - Now with Bootswatch!

2013-04-10T11:30:00+00:00

Bringing Bootswatch to Sphinx

The Bootstrap Theme for Sphinx documentation continues to evolve, and I’m pleased to announce the v0.2.0 release. The theme integrates Twitter’s Bootstrap library into Sphinx, and v0.2.0 adds new support for Bootswatch CSS extensions, which provide a lot of great new UI possibilities for your documentation. We’ve also internally updated to Bootstrap 2.3.1.

The project’s demonstration site now uses the “united” Bootswatch theme to give a clean and well… orange-ish feel to the site. Here is a screenshot of the demonstration site in a desktop viewport:

[![Theme with Bootswatch][img_demo_desk_th]][img_demo_desk] [img_demo_desk_th]: http://loose-bits.com/media/img/2013/04/10/demo_desk_th.png [img_demo_desk]: http://loose-bits.com/media/img/2013/04/10/demo_desk.png

The project is available for download from GitHub and PyPi. You are encouraged to update and try out some Bootswatch goodness!

Adding Bootswatch

As a preliminary note, the use of Bootswatch themes is entirely optional, and disabled by default - the project starts with a vanilla Bootstrap appearance, which you can customize to your heart’s content. To enable a specific Bootswatch theme, set the following option in your “conf.py” file:

html_theme_options = {
    # Bootswatch (http://bootswatch.com/) theme.
    #
    # Options are nothing with "" (default) or the name of a valid theme such
    # as "amelia" or "cosmo".
    #
    # Note that this is served off CDN, so won't be available offline.
    'bootswatch_theme': "united",
}

The list of Bootswatch themes you can now try out include:

As a final note, the Bootstrap CSS files are served off of CDN, which means that if you are offline, your site will have a non-themed, vanilla Bootstrap appearance. We are considering refactoring the Bootswatch code to be available offline as well.

Sphinx Bootstrap Theme 0.1.6 - Bootstrap and Other Updates

2013-02-12T11:30:00+00:00

Sphinx Bootstrap Theme

The Sphinx Bootstrap Theme is an extension for the Sphinx documentation tool, used for Python-based API documentation and static website authoring. The theme integrates Twitter Bootstrap, which is a wildly popular frontend framework. The theme GitHub project provides a demonstration site using the theme.

The theme project continues to move along, and as it has been a while since my last update post, here are some of the changes in version 0.1.6 (since 0.1.0):

Update to Bootstrap v2.3.0.
Enable configuration of the depth of the Global TOC (“Site” tab).
Add footer divider and various other styling tweaks.
Add Bootstrap styling for tables and inline code snippets

In addition, much of the underlying rendering JavaScript code has been revised. For example, Twitter Bootstrap v2.3.0 handles behavior like dropdown menu clicks in mobile much better than previous versions, enabling removal of some hacks and patches in the theme.

Theme Options

Here is the current list of options you can set for the theme in the “conf.py” configuration file, as used by the demo site in “conf.py”:

html_theme_options = {
    # Global TOC depth for "site" navbar tab. (Default: 1)
    # Switching to -1 shows all levels.
    'globaltoc_depth': 2,

    # HTML navbar class (Default: "navbar") to attach to <div> element.
    # For black navbar, do "navbar navbar-inverse"
    'navbar_class': "navbar navbar-inverse",

    # Fix navigation bar to top of page?
    # Values: "true" (default) or "false"
    'navbar_fixed_top': "true",

    # Location of link to source.
    # Options are "nav" (default), "footer" or anything else to exclude.
    'source_link_position': "nav",
}

Sphinx Bootstrap Theme Updates - Mobile, Dropdowns, and More

2012-11-19T11:30:00+00:00

Sphinx Bootstrap Theme Updates!

Almost a year ago, I created the Sphinx Bootstrap Theme project to bring together two of my favorite open source technologies - Sphinx and Twitter Bootstrap. Sphinx is a widely-used Python-based authoring tool for creating static websites and API documentation. Twitter Bootstrap is a frontend JavaScript / CSS framework that offers a consistent UI experience, typography, grid systems, navigation / button / other components, and responsive features for mobile design.

Since its introduction, I now use Sphinx with the Bootstrap Theme for my open source Python projects and we use the theme for all internal technical documentation at work. Coming up on the project’s anniversary, it seemed appropriate to add some tweaks and enhancements to increase the usability and extensibility of the theme.

Improvements now available in version 0.1.0 include:

Upgraded the Bootstrap libraries to version 2.2.1.
Added new theme options to control UI appearance and behavior.
Navigation bar menus now have hierarchical sub-menus for a more intuitive experience.
Reworked the underlying layout code to support fully responsive design, allowing any site to look great on mobile devices like an iPhone or iPad.
Created a PyPI package to permit installation via pip install sphinx_bootstrap_theme.

So let’s see how everything looks and works with the new v0.1.0 theme.

Theme Options

The theme has several new options that you can set in the “conf.py” configuration file.

Bootstrap Theme Demo Site

Let’s start with the Bootstrap Theme’s own demonstration website.

[![Sphinx Bootstrap Theme Desktop View][img_sbt_desk]][img_sbt_desk] [img_sbt_desk]: http://loose-bits.com/media/img/2012/11/19/sbt_desk.png

The demo site has the following theme option settings in “conf.py”:

html_theme_options = {
    # HTML navbar class (Default: "navbar") to attach to <div>.
    # For black navbar, do "navbar navbar-inverse"
    'navbar_class': "navbar navbar-inverse",

    # Fix navigation bar to top of page?
    # Values: "true" (default) or "false"
    'navbar_fixed_top': "true",

    # Location of link to source.
    # Options are "nav" (default), "footer".
    'source_link_position': "nav",
}

Notably, this gives us a dark navigation bar that is fixed to the top of the page during any scrolling.

Django Cloud Browser

Django Cloud Browser is a reusable Django application for browsing and downloading files stored in various cloud datastores (e.g., Amazon Web Services S3, Rackspace Cloud Files). The API documentation uses the Bootstrap Theme with (mostly) the default options.

[![Django Cloud Browser][img_sbt_cb]][img_sbt_cb] [img_sbt_cb]: http://loose-bits.com/media/img/2012/11/19/sbt_cb.png

The only real tweak is setting 'source_link_position': "footer" to move the documentation source code link down from the navigation bar to the bottom of each page.

The navigation bar menus are now hierarchically arranged into dropdowns and sub-menus for easier navigation. This looks a lot better than the previous versions (which just used hacked-up CSS whitespace padding).

[![Navbar Menu Dropdowns][img_sbt_nav_dropdown_hl]][img_sbt_nav_dropdown_hl] [img_sbt_nav_dropdown]: http://loose-bits.com/media/img/2012/11/19/sbt_nav_dropdown.png [img_sbt_nav_dropdown_hl]: http://loose-bits.com/media/img/2012/11/19/sbt_nav_dropdown_hl.png

Mobile UI Support

One of the most popular aspects of the Bootstrap library is its great support for responsive mobile design. These responsive features are integrated into version 0.1.0 of the theme.

The Bootstrap Theme now renders a collapsed, clickable navigation bar in smaller viewports (like a mobile phone and some tablet orientations). Viewing the theme demo site in an iPhone gives us a slightly more terse UI.

[![Mobile Phone View][img_sbt_ios_th]][img_sbt_ios] [img_sbt_ios_th]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_th.png [img_sbt_ios]: http://loose-bits.com/media/img/2012/11/19/sbt_ios.png

Clicking on the right navigation bar button gives us a basic first-level dropdown menu.

[![Mobile Phone Navbar][img_sbt_ios_nav_hl_th]][img_sbt_ios_nav_hl] [img_sbt_ios_nav_th]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_th.png [img_sbt_ios_nav]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav.png [img_sbt_ios_nav_hl_th]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_hl_th.png [img_sbt_ios_nav_hl]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_hl.png

And we can further click through sub-menus to navigate to the desired section or page of a website.

[![Mobile Phone Navbar Menu][img_sbt_ios_nav_dropdown_hl_th]][img_sbt_ios_nav_dropdown_hl] [img_sbt_ios_nav_dropdown_th]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_dropdown_th.png [img_sbt_ios_nav_dropdown]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_dropdown.png [img_sbt_ios_nav_dropdown_hl_th]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_dropdown_hl_th.png [img_sbt_ios_nav_dropdown_hl]: http://loose-bits.com/media/img/2012/11/19/sbt_ios_nav_dropdown_hl.png

With these and other responsive Bootstrap features, the Sphinx Bootstrap Theme now gives your API documentation or website real usability for mobile devices.

Installation

The Bootstrap Theme supports installation via download as detailed in my previous blog post and now adds full PyPI installation support as a Python package.

To get the theme from PyPI, use pip to install it:

$ pip install sphinx_bootstrap_theme

In your Sphinx “conf.py” configuration file, import the theme module at the top:

import sphinx_bootstrap_theme

Then, configure the HTML theme values in “conf.py”, using the module to get values for html_theme_path:

# Activate the theme.
html_theme = 'bootstrap'
html_theme_path = sphinx_bootstrap_theme.get_html_theme_path()

And, that’s pretty much it. The theme should now be available to Sphinx for documentation building.

Conclusion

The Bootstrap Theme brings Twitter Bootstrap to Sphinx sites and with version 0.1.0 adds even more useful Bootstrap features and options. So try it out, see if Bootstrap is right for your Sphinx website, and please pass on any feedback (issue reports, pull requests, etc.).

Shared JavaScript Code with AMD/RequireJS

2012-11-15T11:30:00+00:00

AMD/RequireJS

The Asynchronous Module Definition (AMD) provides a framework for predictable loading of JavaScript modules and dependencies. AMD is an extension of (and possible transport for) the CommonJS specification.

The RequireJS library implements AMD with loading, bundling and other functionality. It brings sensible imports (with define and require) to frontend JavaScript, instead of the usual array of script includes into the global namespace. Interestingly, as Node.js implements CommonJS-style require’s, there are also support modules to bring Node.js code under AMD.

Take both of things together and we have the real potential for shared JavaScript code across the browser and server. As we use this exact approach at work for our new Backbone.js frontend / Express Node.js backend web application, I created a short talk that I presented at the November 11, 2012 Node.DC meetup, entitled “Shared Code with AMD/RequireJS”.

![AMD/RequireJS Talk][img_talk] [img_talk]: http://loose-bits.com/media/img/2012/11/15/nodedc-requirejs.png

Source Code and Demo

The source for the presentation is available on GitHub. The repository further includes all of the demonstration code I used to create a shared JavaScript library and then expose it via the frontend (as straight JavaScript) and the backend (as a REST service).

The talk and the source give a good overview and run-through of how to implement simple shared code, but to give a better sense of the final result, here is a RequireJS-compliant code module (for shuffling elements in a string) that has an underscore dependency, and works in both the browser and as a standard module in Node.js:

// Node.js hook boilerplate.
// This adds the `define` function to Node.
if (typeof define !== 'function') {
  var define = require('amdefine')(module);
}

// Define wrapper gives us `require`
define(function(require) {
  // Looks like a normal `require` for Node.js.
  // Note: Both browser and Node.js should use
  // the **same** version of libraries.
  var _ = require('underscore');

  // Slice a string into a list of items
  // and return the shuffled values.
  var shuffle = function (val) {
    val = val || "";
    return _.shuffle(val.split(/[\s,]+/));
  };

  // Export our shuffle function by returning.
  return shuffle;
});

We add some boilerplate at the top of the library for Node.js, then use the define function to make a require import function available to both the browser and Node.js, and we’re off and code sharing!

AMD, RequireJS, and shared code are topics that have a lot more depth, features, and challenges. Hopefully this provides the start to a journey into better JavaScript dependency management and more extensible full stack web application development.

Deck.js Starter

2012-11-05T10:00:00+00:00

Deck.js Starter

Deck.js is a modern presentation framework for creating interactive HTML-based interactive slide decks. The framework provides:

A basic boilerplate.
CSS themes.
Extensions including a menu, “go to” button, transitions, etc.

To create a presentation, simply edit the HTML files and host a static website and you have an auto-magic slide deck, available to the world.

I have now done a couple of deck.js-based presentations, and find it incredibly useful, but noticed I was re-implementing the same scaffolding enhancements. I wrapped up these common enhancements in a GitHub project, Deck.js Starter. The notable enhancements the starter kit has are:

Jade templates with Markdown support for faster slide authoring.
Executable JavaScript code snippets using CodeMirror.
Executable CoffeeScript code snippets via some specific tweaks I made to the CodeMirror deck.js extension.

Essentially, you can download and install the starter kit, and just start authoring new slides. The starting deck provides a short demo and is available at:

![Deck.js Starter][img_ds_demo] [img_ds_demo]: http://loose-bits.com/media/img/2012/11/05/deck_slides.png

Deck.js Introduction

Getting up to speed deck.js is pretty straightforward: check out a tutorial / exemplary demo or dive right into the docs or wiki. After the basics are mastered, there are plenty of avenues for further exploration / additions:

Themes: Deck.js includes three by default and there are several more available as open source projects.
Extensions: Deck.js includes a menu, navigation and other controls by default. There is a fantastic open source ecosystem around deck providing all manner of additional extensions. To review a few:
- Remote control a presentation.
- Add a laser pointer to the presentation online.
- Easy “presenter” views.
- … and many more!
Tools: Several open source projects offer meta-level enhancements like markdown support, etc.

Working with Deck.js

I have used deck for two of my presentations to date. My basic workflow was to fork the repository from GitHub, then make my edits and add in extensions. Finally, I pushed the static web project to GitHub Pages for free hosting.

For a better sense of what the end products look like, here are the two live presentation decks.

Getting Started with Node.js in the Cloud

My talk to the Nova Node meetup on using Node.js in the cloud.

![Node.js cloud presentation][img_pres] [img_pres]: http://loose-bits.com/media/img//2012/03/24/nodejs-cloud.png

5 Things I Like About CoffeeScript

A simple CoffeeScript survey I presented at the Node.DC meetup.

![CoffeeScript Talk][img_talk] [img_talk]: http://loose-bits.com/media/img//2012/08/16/nodedc-coffeescript.png

A Starter Kit

After writing these presentations and looking to my next one, I realized I was writing the same boilerplate again, and decided to finally wrap everything up in a concise, opinionated starting point: Deck.js Starter

To get started, checkout the online demo, and then review the README. The “tl;dr” version is:

First, clone the repository:

$ git clone https://github.com/ryan-roemer/deck.js-starter.git my-presentation

Next, make sure Node.js is installed, and install all the necessary support packages.

$ cd my-presentation
$ npm install

Once you are ready to author, start the “watch” task that will automatically build “layout.jade” and “index.jade” into the “index.html” static page. You can then preview your slide deck at “index.html”:

$ npm run-script watch

Authoring is pretty easy. To add themes, extensions, or extra scripts, edit “layout.jade”. For slides, edit “index.jade” and add section.slide elements following the boilerplate as a guide. Make sure to take a look at the following possibilities:

Executable JavaScript:

![Deck.js Starter][img_js_demo] [img_js_demo]: http://loose-bits.com/media/img/2012/11/05/deck_js.png

Executable CoffeeScript:

![Deck.js Starter][img_cs_demo] [img_cs_demo]: http://loose-bits.com/media/img/2012/11/05/deck_cs.png

After exploring the basics of the starter kit, feel free to add in any other Deck.js extensions or other useful components or tweaks. You’ll end up with an interactive, visually appealing presentation that you can easily author and host online.

As a parting thought, the Deck.js Starter was written in a couple of hours over the weekend. I do plan to enhance it as I author real presentations in the future (particularly with an eye towards adding more themes). But any feedback or enhancements are most welcome.

Responsive Web Design with Twitter Bootstrap

2012-09-30T18:37:07+00:00

Bootstrap

Twitter’s Bootstrap library is one of the most popular front-end frameworks, with an amazing adoption rate by various web sites, from small to large. Bootstrap provides:

Responsive Features: Enables seamless transitions between desktop, tablet, and mobile view sizes, with intuitive look-and-feel switching and custom single-view overrides.
Layout Support: Fixed and fluid grid systems for web design.
Style Components: Really nice looking form helpers, fonts, tables, etc.
UI Components: Dropdown menus, buttons, accordions, alerts, progress bars and more, with JavaScript support.

Essentially, Bootstrap makes it easy to design a responsive website (that looks good in desktop or mobile views) and add user interface components that keep with the overall user experience.

Moving Loose Bits to Bootstrap

Loose Bits (this blog) previously had a non-responsive design - the main heading bar was far too wide when viewed on a mobile phone (like my iPhone). I had been itching to make this site mobile-friendly, and finally decided to bite the bullet and integrate Bootstrap.

I downloaded the full bootstrap build from source, so that I could customize parts of the framework and only add in what I needed. As this blog is hosted as an open source project on GitHub, the source code and build system is available for checkout or download at my Loose Bits repository.

While I did customize some colors and variables for Bootstrap and spent some time inserting Bootstrap-compatible CSS classes in my HTML layout, I have to say the overall experience was quite easy, required minimal changes, and I am pleased with the final website.

The different layout effects for various viewports can be viewed simply by resizing a browser window on a desktop computer. In desktop view, I’ve retooled the main header and navigation bar to look like:

My layout uses the basic “inverse” (dark) Bootstrap theme and integrates the fantastic Font Awesome icons for each navigation bar menu item.

Loose Bits on Mobile!

Moving on to the mobile experience, the header and full navigation bars are replaced by a title-only heading with a clickable navigation menu button. Resizing a browser window down to mobile phone size (or if already on a mobile device), we get:

I’ve used the simple navigation collapse feature to have a clickable menu of all of the previous navigation bar items, so as to more effectively use the reduced real estate in a mobile view:

The rest of the website has a few other adjustments in mobile view mode, nearly all of which auto-magically happen without any further tweaks to Bootstrap configuration or JavaScript hacking.

At the end of the day, I’m pretty happy with the Bootstrap development experience and the end result for my website. Like any web framework under active development, there are still a few gotcha’s - like some loss of browser compatibility and weird JavaScript bugs (e.g., my navigation buttons sometimes disappear). But, ultimately, for anyone searching for the fastest path to a reliable and intuitive responsive web site, I would definitely recommend looking at Twitter Bootstrap.

5 Things I Like About CoffeeScript

2012-08-16T11:30:00+00:00

CoffeeScript

CoffeeScript is self-described as “a little language that compiles into JavaScript”. CoffeeScript implements a subset of the full JavaScript language and adds a little bit of common boilerplate code to take out some of the less savory parts of JavaScript. CoffeeScript works both in Node.js on the backend, as well as in the browser.

I gave a short survey talk at the August 15, 2012 Node.DC meetup, “5 Things I Like About CoffeeScript”.

![CoffeeScript Talk][img_talk] [img_talk]: http://loose-bits.com/media/img/2012/08/16/nodedc-coffeescript.png

The talk’s source is available on GitHub, and uses the awesome deck.js presentation framework with the CodeMirror plugin to enable editable and runnable code samples. I further hacked up the CodeMirror plugin to additionally make the CoffeeScript code examples executable (CodeMirror only does JavaScript by default).

Five Cool Things in CoffeeScript

For the impatient, the five topics I covered are:

Syntax
Functions
Classes
Existential Operator
Loops, Comprehensions

The talk just skims the surface of the powerful (yet slim) CoffeeScript language. For further reading / real introductory tutorials, I would suggest:

CoffeeScript Documentation: The official API.
The CoffeeScript Cookbook: Lots of “recipes” for common programming problems and projects.
Smooth Coffeescript: Interactive online eBook with live code execution and editing.

Better Data Slinging with Node.js Readable/Writable Streams and Pipes

2012-08-02T14:00:00+00:00

Node.js Streams

Node.js provides an extensible and fast platform for web servers, proxies, and middle-tier services. Node.js applications often utilize some transformation from one data format (e.g., a database or cloud store) to another (e.g., an HTML or JSON page).

Most folks are familiar with the callback-style of hooking together various JavaScript data components in a Node.js program. However, an often overlooked and very powerful data binding abstraction for Node.js is found in the stream class.

Streams are an abstract interface for data objects in Node.js which can be readable and/or writable. They can be hooked from one to another in a similar style to Unix pipes – in fact, the stream operation we’ll mostly focus on here is the not-coincidentally-named pipe(). Some potential advantages of stream pipes over other binding styles include:

Often much less code for the actual binding (can just call pipe()).
Streams can handle pausing / resuming of data flows. Implementing classes, however, have to implement this logic internally if supported.
Don’t have to set specific callbacks or listeners for intermediate data events – just pipe() the stream and forget it.
Avoiding buffering by processing data and re-emitting it directly to another stream (unless all of the data is required in one chunk).
Compatible with the many Node.js core modules that already implement streams, including HTTP, HTTPS, and file and process I/O.

For a brief example, we can create a download client to retrieve a web page and write it to a file as follows (ignoring error handling):

// Get Google's home page.
require('http').get("http://www.google.com/", function(response) {
  // The callback provides the response readable stream.
  // Then, we open our output text stream.
  var outStream = require('fs').createWriteStream("out.txt");

  // Pipe the input to the output, which writes the file.
  response.pipe(outStream);
});

Notice that we didn’t set any explicit “data” listeners or buffer any of the data, even if it came in as chunks. We simply pipe()‘ed it with our two stream objects: response and outStream. The output of response is hooked to the input of outStream and we’re done.

More importantly (as we’ll get to below), we can add many more pipe() calls to do other transformations / data-slinging inline to our chained call. Ultimately, it just takes a little bit of glue code to hook together data flows in a terse and efficient manner with streams.

The Stream Interfaces

So how do we do this for our own classes and objects?

The Node.js streams documentation offers the full rundown of how to implement the interfaces, but we’ll give an abbreviated version here to get going. All of the code examples discussed in this post are available as a GitHub gist.

Readable Streams

Readable Streams must emit “data” events whenever they have data to be read and “end” when the data stream is finished. The implementing constructor should also set this.readable = true. The interface additionally provides a lot more implementer flexibility for things like pausing and resuming a stream, as well as resource management and cleanup.

Writable Streams

Writable Streams must implement the write() method to accept new data chunks into the stream and the end() method to instruct the stream that the data is finished. The implementing constructor should also set this.writable = true.

Putting it Together

Let’s take a look at a simple example of a custom-implemented readable and writable stream that simply passes through data – data input is simply emitted unchanged as output.

// Set both readable and writable in constructor.
var NopStream = function () {
  this.readable = true;
  this.writable = true;
};

// Inherit from base stream class.
require('util').inherits(NopStream, require('stream'));

// Extract args to `write` and emit as `data` event.
NopStream.prototype.write = function () {
  args = Array.prototype.slice.call(arguments, 0);
  this.emit.apply(this, ['data'].concat(args))
};

// Extract args to `end` and emit as `end` event.
NopStream.prototype.end = function () {
  args = Array.prototype.slice.call(arguments, 0);
  this.emit.apply(this, ['end'].concat(args))
};

This is essentially the bare minimum for a readable and writable stream class. Not too much work! And for more complicated streams, we can simply augment write/data and end/end to do whatever data transformations we want.

We can now take the web scraping example from above and add the pass-through stream in the middle with the same effect – we still get a file written to output.

// Download the same page again, but with the NOP stream
// in the middle.
require('http').get("http://www.google.com/", function(response) {
  var outStream = require('fs').createWriteStream("out.txt");

  response
    .pipe(new NopStream())
    .pipe(outStream);
});

Checking the output file (“out.txt”), we can see the download results are the same as our original example.

And in fact, we could even reuse the pass-through stream multiple times to illustrate repeated pipe() data flows (although there’s absolutely no practical sense to the following):

response
  // Wow, that's a lot of nop's!
  .pipe(new NopStream())
  .pipe(new NopStream())
  .pipe(new NopStream())
  .pipe(new NopStream())
  // OK, finally write out to file.
  .pipe(outStream);

So that’s the basics. Let’s look at creating something a tad more useful.

Let’s Upper-case Some Data!

The UpperCaseStream class takes a data source (in string or Buffer format) and converts string data into upper case letters. Not ultimately that useful or extensible, but it’s definitely a data transformation that can illustrate the ease of creation and use of a custom stream.

We mostly take our simple pass-through stream above and add a custom _transform helper method to transform the data in either a write() or end() call, then re-emit the upper-cased data in a data event.

/**
 * A simple upper-case stream converter.
 */
var UpperCaseStream = function () {
  this.readable = true;
  this.writable = true;
};

require("util").inherits(UpperCaseStream, require("stream"));

/**
 * Handle various params and upper-case string data.
 *
 * Signature can be in format of:
 *  - string, [encoding]
 *  - buffer
 *
 * Our example implementation hacks the data into a simpler
 # (string) form -- real implementations would need more.
 */
UpperCaseStream.prototype._transform = function (data) {
  // Here, we'll just shortcut to a string.
  data = data ? data.toString() : "";

  // Upper-case the string and emit data event with transformed data.
  this.emit("data", data.toUpperCase());
};

/**
 * Stream write (override).
 */
UpperCaseStream.prototype.write = function () {
  this._transform.apply(this, arguments);
};

/**
 * Stream end (override).
 */
UpperCaseStream.prototype.end = function () {
  this._transform.apply(this, arguments);
  this.emit("end");
};

To test things out, we can take an input file (“input.txt”), read it in, upper case all text, then write it out to “out.txt” using three streams.

var fs = require("fs"),
  input = fs.createReadStream("input.txt"),
  output = fs.createWriteStream("out.txt"),
  upperCase = new UpperCaseStream();

// Open our read input, uppercase it, then write out.
input
  .pipe(upperCase)
  .pipe(output);

The resulting output file is now uppercased! All in all, not that amazing, but considering the ease of our implementation, other (more useful) read/write stream applications could include:

XML to JSON conversion.
Unzipping zipped data.
Image resizing.
… any other transformations you’d like to use with your existing streams.

Conclusion

Streams provide a great means of binding together lots of data in a sane and manageable way. Beyond the core library documents, there are a lot of great stream introductions for further reference:

Getting Started with Node.js in the Cloud - Presentation

2012-03-24T22:08:21+00:00

Getting Started with Node.js in the Cloud

NovaNode had its first meetup event on March 20, 2012 at SpanishDict’s offices. I gave a talk titled “Getting Started with Node.js in the Cloud” covering Node.js application development and deployment to the cloud.

![Node.js cloud presentation][img_pres] [img_pres]: http://loose-bits.com/media/img/2012/03/24/nodejs-cloud.png

My talk goes through a first simple “hello world” application and builds up to a realtime chat application using Redis and websockets. We deploy the applications to Heroku and cover everything from logs and monitoring to add-on services and scaling. The talk concludes with an assessment and use cases for deploying on a Platform-as-a-Service provider like Heroku versus an Infrastructure-as-a-Service provider like Amazon Web Services.

The presentation is available live on GitHub, and uses the deck.js presentation framework. I have also posted the full demo source code to GitHub, including the chat server that we used during the meetup. Try it out!

Return-Oriented Programming - Systems, Languages, and Applications

2012-03-24T16:40:56+00:00

Return-Oriented Programming

Return-oriented programming is a software exploit technique to take over a program by diverting control flow without injecting any code. At UCSD, I did most of my graduate research around this specific attack, working with Erik Buchanan, Hovav Shacham, and Stefan Savage. Now three years later, the ACM Transactions of Information and System Security (TISSEC) journal has published our full article, “Return-Oriented Programming: Systems, Languages, and Applications” in the March 2012 issue.

For a great introduction to return-oriented programming attacks, see Hovav’s Black Hat presentation, “Return-oriented Programming: Exploitation without Code Injection”.

By way of a little background, software exploit techniques such as buffer overflows traditionally injected code into a vulnerable buffer, and then pointed control to that injected code whereby the attacker executed their own instructions. The original attack was succinctly described in Aleph One’s “Smashing The Stack For Fun And Profit”.

Software vendors responded to injected code attacks with various defenses, one popular being “write XOR execute” (“W⊕X”), whereby memory is either writable (e.g., data structure storage) or executable (e.g., library code), but not both. The logic behind the defense was that an attacker could inject arbitrary data in writable memory but could not execute it, thus preventing the classic type of buffer overflow attack.

However, preventing code injection is not sufficient to prevent arbitrary computation. Following a long line of W⊕X exploit research, return-oriented programming works by taking existing executable code (e.g., loaded libraries) and using small chunks of it in unintended ways by hijacking control flow via compromising the stack or writable memory.

Return-Oriented Programming: Systems, Languages, and Applications

Erik and I became interested in return-oriented programming when we read Hovav’s paper that introduced the technique, “The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86).”. Hovav focused on the x86 architecture and hypothesized that the attack would possibly be mitigated on a RISC architecture.

Fortunately, Erik and I were both TA’s for the SPARC (a RISC architecture) assembly programming course for undergraduates, and figured we could apply the attack further than originally anticipated. We were extremely lucky to get Hovav on board with our research and to later succeed in porting the attack to the SPARC architecture. We presented our findings in a paper at the 2008 ACM Conference on Computer and Communications Security (CCS). We were later invited to turn our conference paper into a full journal paper for TISSEC.

We essentially coalesced both the x86 and SPARC research to be the definitive statement on return-oriented programming, and submitted our draft in early 2009. Academic publishing does not always move at the most rapid pace, and it took about three years to finally get “Return-Oriented Programming: Systems, Languages, and Applications” published.

I think the paper is a good statement of the original development and first level extensions of the attack (and I’m obviously biased in recommending it, as I’m an author). But, it is worth pointing out that a lot has happened in the time since we submitted our draft, and the state of the art in return-oriented programming has advanced significantly past that described in our paper. Since our original research, the attack has found its way into voting machines, routers, mobile devices and other places. It is fascinating (and scary) to see the ways in which the technique has now entered the mainstream, but I’m at least happy to have (in a small way) helped expose the problem to software vendors early to give them a chance to build up defenses against return-oriented programming.

NovaNode Inaugural Meetup - Getting Started with Node.js in the Cloud

2012-03-06T15:01:00+00:00

Node.js in NoVA!

We recently launched the NovaNode meetup group focusing on the Node.js event-driven JavaScript framework. We have a great set of talks lined up for our inaugural meetup event on March 20, 2012 at 6:30 pm. We’re hosting things at SpanishDict’s offices in Arlington, VA near the Clarendon metro stop.

Our talks are:

“Getting Started with Node.js in the Cloud”, which I’ll present.
“Everyauth: OAuth for Busy Nerds” by Jason Bond Pratt, Co-founder of Tixelated. OAuth is the standard interface for web services integration for user authentication, and is your starting point for linking an application to Google, Facebook, Twitter, etc. Jason’s talk will take us through Everyauth’s basic architecture and integration with other data stores like MongoDB.

Getting Started with Node.js in the Cloud

Node.js provides a solid platform for scalable and responsive web applications. Development in Node.js further has a lower learning curve compared to many highly-concurrent / asynchronous frameworks as it uses JavaScript, which is very familiar territory for web developers.

The cloud complements Node.js by providing the means to deploy applications rapidly and scale up with no upfront investment costs. The “big” cloud providers such as Amazon Web Services and Rackspace easily support an infrastructures for hosting Node.js applications. Moreover, there has been a burst of Platform-as-a-Service contenders that provide even easier Node.js application hosting, including: Heroku, NodeJitsu and even Microsoft’s Azure platform.

In my talk, I’ll walk through a basic Node.js application that we develop, deploy and then scale easily in the cloud. I’ll discuss best practices and what opportunities and pitfalls lie ahead when taking your Node.js applications to the cloud.

Hope to see you at the meetup!

Async.js Presentation for Joint Node.DC / DC jQuery Meetup

2012-02-21T20:54:38+00:00

Async.js, a JavaScript Control Flow Library

On February 20, 2012, I gave a talk on Async.js to the joint meetup of Node.DC and DC jQuery.

Async.js is a control flow library that provides great abstractions for writing serial, parallel, and combined asynchronous code without ending up in the dreaded trenches of nested callback insanity.

![Async.js presentation][img_pres] [img_pres]: http://loose-bits.com/media/img/2012/02/21/async-talk.png

My presentation is available live on GitHub, and uses Impress.js for Prezi-like functionality in the browser. As a consequence, it might not render in older browsers and IE. My talk covers the basic abstractions:

Series
Waterfalls
Parallel
Collections

and more, so check it out and feel free to ping me with any questions or comments!

Authenticated, Static Web Sites on Google App Engine

2012-01-08T14:00:00+00:00

Static HTML Web Site Hosting with Google App Engine

Google App Engine is a platform-as-a-service (PAAS) product that provides scalable, cloud-hosted web applications using Google’s massive engineering infrastructure. While App Engine is primarily used by web developers (e.g., programming in Python or Java), it offers three features which make it uniquely helpful for static site hosting:

Arbitrary static file handling.
Extensible authentication support.
Very inexpensive (most likely free) for hosting static content.

In this post, we’ll walk through uploading a static HTML site to App Engine, and configuring it such that it requires users to log in to via a Google Apps domain account before viewing any content. This is a common situation for organizations already using Google Apps to manage email, documents, etc. that want to host a web site without having extra configuration hassles. App Engine essentially takes care of all the authentication / user management, and you just have to upload the static web site.

The assumption throughout the rest of this post is that you already have a domain name managed by Google Apps (e.g., “example.com”). We will create an App Engine application and restrict it to users of the specific Google Apps domain, requiring a login of a user “@example.com”. Then we’ll upload the static site and verify that authentication works as expected.

Create an Authenticated App Engine Application

The first step is to create an App Engine web application. You’ll need to sign up for an App Engine account, download the App Engine Python SDK (make sure you get the Python one!), and you should read the “getting started” documentation. Also, if given the option to “install command symlinks”, make sure you choose to do this (which will give us an appcfg.py executable in our path for use later).

Once App Engine is setup and installed, we can create the actual application. The important thing to point out here is that you must select the authentication method you want for your web site at creation time, as it cannot be changed later. (Although, you can create a new application with different authentication and delete your original application).

Take a moment to review the App Engine authentication article, as we’re basically going to follow these steps. Go to the create application web page. Fill in the following information:

Application Identifier: Choose a descriptive name for your site (e.g. “internal-docs”. There cannot be an existing matching identifier, so have some alternates handy. This will result in a domain name of “internal-docs.appspot.com” (or whatever your identifier is) for your finished website.
Application Title: A free text description of your website. Feel free to put anything in that provides a simple title for your site.
Authentication Options (Advanced): You’ll need to click the “edit” link which then gives us three options for authentication: (1) “Open to all Google Accounts users (default)”, (2) “Restricted to the following Google Apps domain:”, or (3) “(Experimental) Open to all users with an OpenID Provider”. Click the button for “Restricted to the following Google Apps domain:” and enter your Google Apps-managed domain (e.g., “example.com”). I should point out again that you already should have Google Apps set up for the domain name you are entering.

From there you can click “Create Application” and the application should be created. Make sure to keep your application identifier handy.

To enable Google Apps domain authentication for the new application, we need to follow the instructions in the App Engine authentication article. Basically, you need to open a web browser to: “http://www.google.com/a/YOUR DOMAIN” and click on the “Dashboard” tab. Go to “Service settings” and click on the “Add more services” link. In the “Other services” section, there will be a place to add an App Engine service. Type in your application identifier code here and click “Add it now”. This will hook up your specific Google Apps Domain with the App Engine service.

Configure and Upload Static Web Site

The next step is to gather your static web site files, add an application configuration, and upload all the content to the App Engine application. We will place all of our application content in a directory called “my_site” (or something else of your choosing). You are best off keeping this directory under a source control management system (e.g., git), so that you can monitor, track and revert changes to all of your files.

Configuration

We need an application configuration file call “app.yaml” in the root of our project directory. This file controls various aspects of the application, including how the application routes URLs to handlers. We’ll use a configuration that handles all static file types (including HTML), and just simply serves them.

There are various other posts out there discussing configurations for static web sites on App Engine, but the best configuration that I found was a gist by GitHub user “darktable”. However, this configuration didn’t including authentication, so I forked the gist and added authentication attributes to produce our final app.yaml file that you should download to “my_site/app.yaml”. You can also view a basic Readme file and other information at the GitHub gist page.

Here’s a snippet of the “app.yaml” file that you’ll need to slightly modify:

application: you-app-name-here
version: 1
runtime: python
api_version: 1

default_expiration: "30d"

handlers:
- url: /(.*\.(appcache|manifest))
  mime_type: text/cache-manifest
  static_files: static/\1
  upload: static/(.*\.(appcache|manifest))
  expiration: "0m"
  login: required

# ... OTHER CONTENT SNIPPED ...

# site root
- url: /
  static_files: static/index.html
  upload: static/index.html
  expiration: "15m"
  login: required

After downloading to “my_site/app.yaml”, update the application: you-app-name-here directive with the specific App Engine application identifier you chose in the application creation section above.

Static Content

Now that we have a configuration file, create a folder named “my_site/static” which will house actual static web site. As we want to check that the authentication works first before uploading potentially sensitive information, I would recommend creating a test HTML page that just contains the content “It worked!” and adding that as “my_site/static/index.html”.

Now, we should have a project layout that looks like:

my_site/
  app.yaml
  static/
    index.html

At this point we can upload the full site to our static server using appcfg.py. Make sure that we have appcfg.py available:

$ which appcfg.py
/usr/local/bin/appcfg.py

If you don’t get an executable path back (any path is fine as long as something is returned by the which command), then review the App Engine “getting started” documents for installation of the runtime.

Assuming we do have appcfg.py available, change directory in your terminal to the directory containing the “my_site” project folder and upload the static site with the following command:

$ appcfg.py update my_site

You will have to enter your Google credentials here. After the upload finishes, you should be able to open a web browser to: “<your application identifier>.appspot.com”. If you are authenticated to your Google Apps domain, you should see the “It worked!” test page. If not, you should be prompted to login to your Google Apps domain. A good way to test the authentication works is to open a new Google Chrome Incognito window. It should always force a new Google Apps login if you have configured things properly. If the authentication doesn’t work quite right, review the App Engine authentication page for tips and pointers, or leave a comment below on this post.

Assuming authentication does work correctly, then you can now remove the test “index.html” file and upload your real site content to the “my_site/static” directory. Every time you change the content, make sure to re-upload the project with appcfg.py and enjoy your static web site!

Twitter Bootstrap Theme for Sphinx

2011-12-09T16:00:00+00:00

Bringing Twitter Bootstrap to Sphinx

Bootstrap is an open source CSS/JS framework from the folks at Twitter that provides base typography, layout (grids), forms / form inputs, tables and navigation, and has been gaining wide popularity among developers. In my personal experience with a couple of sites and projects, Bootstrap really provides a great starting point for a frontend that provides good extensibility over time.

Sphinx is the most popular Python documentation generator tool. It has fantastic Python module docstring / comment parsing in addition to a sensible static pages framework, providing a great final web site built from both inline code documentation and outside web pages.

In the process of starting up a number of new Sphinx projects, I decided that I would really like to use Bootstrap as the underlying UI framework. Fortunately, Sphinx supports custom theme plugins, so I spent a couple of days hacking, and created a Sphinx Bootstrap Theme, which is now available on GitHub.

To show all of the coolness of Bootstrap and Sphinx, I’ve put up a simple demo, which provides a skeleton Sphinx site with the project README file rendered as content. Here is a screenshot:

[![Sphinx Bootstrap Theme Demo][img_sbt_teaser_th]][img_sbt_teaser] [img_sbt_teaser_th]: http://loose-bits.com/media/img/2011/12/09/sbt_teaser_thumb.png [img_sbt_teaser]: http://loose-bits.com/media/img/2011/12/09/sbt_teaser.png

Installation

Sphinx themes can be installed either as a directory of files or a zip file. For simplicity’s sake, we’ll go with a pre-packaged zip file installation here.

First, set up a themes directory, which typically is something like “_themes” located in your Sphinx source directory:

$ cd /path/to/source
$ mkdir -p _themes

Next, download the most current “bootstrap.zip” archive from GitHub. Note that older zipped theme files are available at the downloads page, with git hash suffixes.

$ cd /path/to/source/_themes
$ wget https://github.com/downloads/ryan-roemer/sphinx-bootstrap-theme/bootstrap.zip

As an alternative, you can just clone the theme repository, and place the “bootstrap” directory in your themes directory. The “bootstrap.zip” file should be the most up-to-date version, but getting the directory from source will remain the authoritative current version.

Finally, add the themes directory path and theme name to your source Sphinx “conf.py” file:

# Activate the theme.
sys.path.append(os.path.abspath('_themes'))
html_theme_path = ['_themes']
html_theme = 'bootstrap'

# Optional. Use a shorter name to conserve nav. bar space.
html_short_title = 'Demo'

Rebuild your Sphinx documentation, and you should now have the Bootstrap theme up and running! See the README for more information on customizing and hacking on the theme.

Brief Tour

The Bootstrap Sphinx theme basically rolls up the site-wide / page-level table of contents, page navigation, and search form into the main navigation top bar.

The first dropdown tab is the “Site” button (expanded and highlighted below). Internally this wraps up the globaltoc template using the toctree directive.

[![Site Navigation][img_sbt_nav_site_th]][img_sbt_nav_site] [img_sbt_nav_site_th]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_site_thumb.png [img_sbt_nav_site]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_site.png

For both the site and page navigation dropdowns, the Bootstrap theme uses JavaScript hackery to unpack the internal list structure and convert it to an ordered flat list with hierarchy displayed using extra left padding corresponding to levels. The natural levels of table of content displayed normally by Sphinx should be preserved in the dropdown menus.

The second dropdown tab is the “Page” navigation for table of contents link within a single page. This wraps the localtoc template using the toc directive.

[![Page Navigation][img_sbt_nav_page_th]][img_sbt_nav_page] [img_sbt_nav_page_th]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_page_thumb.png [img_sbt_nav_page]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_page.png

Search

The right “Search” box in the navigation top bar connects to the built-in, client-side search that Sphinx provides. Here we search for the term “bootstrap”:

[![Search Navigation][img_sbt_nav_search_th]][img_sbt_nav_search] [img_sbt_nav_search_th]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_search_thumb.png [img_sbt_nav_search]: http://loose-bits.com/media/img/2011/12/09/sbt_nav_search.png

… which gives us the following page matches:

[![Search Results][img_sbt_search_th]][img_sbt_search] [img_sbt_search_th]: http://loose-bits.com/media/img/2011/12/09/sbt_search_thumb.png [img_sbt_search]: http://loose-bits.com/media/img/2011/12/09/sbt_search.png

Final Thoughts

The Bootstrap Theme provides an easy means for Twitter Bootstrap integration for my current Sphinx sites. But, as I’ve just put together the theme this week, it is probably still rough around the edges. Any issue reports, pull requests, and feedback are most welcome.

Sunny.js Presentation and Demo Server

2011-10-20T16:00:00+00:00

Sunny.js Presentation

I gave a lightning talk at last night’s Node.js Meetup introducing Sunny.js, a multi-cloud library for Node.js. For more background on Sunny, see my previous blog post to get started with the library.

Instead of the usual PowerPoint deck, I used Prezi, which I have been wanting to try out for some time. Here is the Sunny.js prezi I gave at the meetup:

Although the Prezi took some time to put together (especially being my first time using the service), I was very happy with the appearance and flow of the presentation.

Simple Cloud/Web Proxy Server

As part of my presentation, I demo’ed a simple proxy service using Sunny and Node. I took the v0.0.1 HTML documentation for Sunny, and uploaded all of it to an Amazon S3 bucket, being careful to preserve paths from the original documentation files when naming S3 keys. (Actually, this wasn’t hard at all – I just used the reliable CyberDuck S3 client).

I then wrote a really simple 40-line web server. It basically takes credentials from the process environment, checks if a specified container exists, and then translates web request GET paths to blob names in the container I set up. The result is that I could serve my entire documentation site straight from by Amazon S3 bucket!

The source for the project, sunny-proxy, is available on GitHub, and is fairly straightforward to setup for yourself using the Readme instructions.

For the impatient, here’s the entire relevant code for the web server:

var http = require('http'),
  mime = require('mime'),
  sunny = require('sunny'),
  conn = sunny.Configuration.fromEnv().connection,
  ADDR = process.env.ADDRESS || "0.0.0.0",
  PORT = process.env.PORT || 2000,
  CONTAINER = process.env.SUNNY_PROXY_CONTAINER;

// Get our container and create server inside to get blobs.
conn.getContainer(CONTAINER, { validate: true })
  .on('error', function (err) {
    console.log(err);
    throw err;
  })
  .on('end', function (results) {
    // We have a valid container, so let's create the server now.
    http.createServer(function (req, res) {
      var path = req.url.replace(/\/$/, "/index.html").replace(/^\/*/, ""),
        status = 200,
        logResult = function () {
          console.log("[%s] %s", status, path); },
        stream;

      // Header based on MIME type (re-write on error).
      res.writeHead(status, { 'content-type': mime.lookup(path) });

      // Get blob and pass through error or pipe to response.
      stream = results.container.getBlob(path);
      stream.on('error', function (err) {
        status = err.statusCode || 500;
        res.writeHead(status, { 'content-type': "text/html" });
        res.end("<h1>" + status + ": " + err.message + "</h1>");
        logResult();
      });
      stream.on('end', logResult);
      stream.pipe(res);
      stream.end();
    }).listen(PORT, ADDR);
    console.log("Server running at http://%s:%s/", ADDR, PORT);
  })
  .end();

We use the magic of stream.pipe(res) to handle all of the transfer details of passing through our blob data chunks to the HTTP response.

I even took it a step further and set up a simple Heroku app to run the application. All it took was a quick review of the Node app instructions and a “Procfile” file (mine is available in the GitHub repo) and I was able to get my node proxy server up and running on the web in under a half hour! Moreover, I was pleasantly surprised at how fast the web application was running – especially considering the web server doesn’t use any caching logic or any intelligent optimizations that a real web server would.

The Sunny-based proxy server certainly isn’t anything amazing or revolutionary – it just serves blobs as if they were static files for a web server – but it does show just how easily Node.js and libraries like Sunny can glue cloud / service-based components together for fast and scalable applications.

Sunny.js, a Cloud Library for Node.js

2011-10-16T16:00:00+00:00

Sunny.js

Node.js provides a great environment for cloud applications and development. Node’s asynchronous design and great HTTP/REST support provide the building blocks for architecting non-blocking and scalable applications in the cloud.

There are a lot of great Node cloud clients out there for cloud datastores. Amazon Web Services S3 has: knox, node-sissy, node-s3, Node.js-Amazon-S3, and Rackspace Cloud Files has: node-cloudfiles.

After reviewing these libraries, I found a few features variously lacking that I would like in an ideal cloud datastore client:

Event-based interface.
“One-shot” requests wherever possible.
Able to create / delete arbitrary containers (buckets).
Configurable request headers, cloud options and metadata.
SSL support.

Additionally, it would be nice to have a library that worked across multiple clouds like jclouds for Java and Libcloud for Python.

With these goals in mind, I put together a basic multi-cloud datastore client called Sunny.js. Sunny initially supports Amazon Web Service’s Simple Storage Service (S3) and Google Storage for Developers (version 1). Sunny hits all the points above, and also has:

A really well-documented API and user guide. (At least I think so!)
Full unit tests for all cloud operations and providers.

Here are some basic resources to get started with Sunny, which we’ll dive into a little deeper in the subsequent sections of this post:

Sunny.js Documentation: Guides and API documentation.
- User Guide: Guide for programmers using Sunny in your applications and library.
- Developer’s Guide: Guide for developing, extending, and hacking on the Sunny.js source code.
- API: Full API.
Sunny.js GitHub Page: Source repository and issue tracker.
Sunny.js NPM Page: NPM page and history.

Getting Started

First, install sunny from npm:

$ npm install sunny

Once Sunny is installed, we need to set up some configuration information for our Node program. We can provide cloud account and option information to Sunny either via the process environment or a JavaScript object / file. For simplicity, we’ll just export some environment variables (see the user guide for more on configuration and using files).

$ export SUNNY_PROVIDER=("aws"|"google")
$ export SUNNY_ACCOUNT="ACCOUNT_NAME"
$ export SUNNY_SECRET_KEY="ACCOUNT_SECRET_KEY"
$ export SUNNY_SSL=("true"|"false")

From here, we can create our Node program, and get a live cloud datastore connection as follows:

var sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection;

Cloud Operations

Now that we have our cloud connection object (conn), we can perform all of our cloud operations on containers and blobs. For those used to Amazon Web Services parlance, a Sunny container is equivalent to an S3 “bucket”, and a Sunny blob is an S3 “key”.

Sunny cloud operations are asynchronous and event-based. The cloud methods return either a request or a stream object, that is then used to set listeners and callbacks on, before calling end() and starting the underlying real cloud network operation.

Requests

Sunny request objects are not true Node request objects, but approximate a subset of a Node HTTP request object events. The most common request events are:

“error”: The underlying real cloud operation failed.
“end”: The operation finished and we have results.
“data”: A GET request returned a data chunk.

Let’s perform a basic asynchronous operation to get a container named “sunny” from our cloud store.

var request = conn.getContainer("sunnyjs", { validate: true });
request.on('error', function (err) {
  console.log("We received error: %s", err);
});
request.on('end', function (results, meta) {
  var container = results.container;
  console.log("Retrieved container \"%s\".", container.name);
});
request.end();

which gives us the output:

Retrieved container "sunnyjs".

Breaking the code down, we call getContainer with the container name and the option validate: true. The validate option means we actually perform a cloud request to check the container exists before continuing. This is a good check to start with before launching in to other code. But, Sunny allows the programmer to choose to not validate, and wait for the first blob request to actually perform any network operation, which is faster.

The getContainer method returns a request object, which we then set our listeners on for “error” and “end” (when we have results). Finally, the call to request.end() initiates the actual network operation.

Our “end” callback takes a results parameter which contains a container method for further use with blob methods, and a meta object with information from the actual cloud call (metadata, HTTP headers, etc.) See the getContainer documentation for more options, results and uses.

The other request-based Sunny methods work in a similar fashion, and include the following:

Connection: Get a list of containers. Get/create/delete a single container.
Container: Get/create/delete a single container. Get a list of blobs. Head/delete a single blob. Get/put blob to/from a file.
Blob: Head/delete a single blob. Get/put blob to/from a file.

See the API for full method details.

Streams

Sunny returns stream objects for data-based cloud operations (PUT or GET), which are real implementations of Node streams. A GET blob method returns a Readable Stream object, and a PUT blob method returns a Writable Stream object.

Let’s take the container object we received from our successful getContainer request above, and perform a PUT blob operation with a simple string. (Note: programmatically, this would be within the “end” callback of the get container request). Data can be written with any number of write() calls and a single end() call (which starts the data transfer and ignores all subsequent writes).

var writeStream = container.putBlob("my-blob.txt");
writeStream.on('error', function (err) {
  console.log("We received error: %s", err);
});
writeStream.on('end', function (results, meta) {
  var blob = results.blob;
  console.log("Put blob \"%s\".", blob.name);
});
writeStream.write("My data string.");
writeStream.end();

which produces:

Put blob "my-blob.txt".

Now, in a later operation (say, within the putBlob “end” callback, after we know the blob was written), we can retrieve the data:

var readStream = container.getBlob("my-blob.txt");
readStream.on('error', function (err) {
  console.log("We received error: %s", err);
});
readStream.on('data', function (chunk) {
  console.log("Data chunk: \"%s\"", chunk);
});
readStream.on('end', function (results, meta) {
  var blob = results.blob;
  console.log("Get blob \"%s\".", blob.name);
});
readStream.end();

which gives us:

Data chunk: "My data string."
Get blob "my-blob.txt".

Note that we’re getting the data in raw chunks from the “data” event, which is somewhat tedious to cobble together (as strings if encoded or Buffer objects otherwise). Given that we have full read / write stream implementations, Sunny really shines with the availability of pipe()‘ing data. Essentially, you can simply connect a Sunny read / write stream to another read / write stream, call pipe() and let Node and Sunny take care of the rest.

For example, Sunny provides helper methods for getting / putting blobs to and from files. Looking at the source code, the real gist of, for example, getting a blob to a local file is the following:

var getStream = container.getBlob("name", /* ... */);
var writeStream = fs.createWriteStream(filePath, /* ... */);

// Start the pipe and call 'end()'.
getStream.pipe(writeStream);
getStream.end()

There’s a little more to it, but it is mostly this easy. Take a look at the source code for the blob convenience methods putFromFile and getToFile to see a full implementation that binds together file and cloud streams.

Node streams available for use with Sunny get / put pipe() calls include:

HTTP/HTTPS requests and responses. See my sunny-proxy project for a simple web server that proxies web requests to cloud blobs using Sunny blob get streams piped to HTTP responses.
File reads and writes – fs.ReadStream, fs.WriteStream

Cloud Headers and Metadata

Sunny abstracts common cloud provider header and metadata. For example, AWS S3 has a metadata header prefix of “x-amz-meta-“, while Google Storage has an analogous one of “x-goog-meta-“.

Sunny cloud operations return a meta object that looks like:

{
  headers: {
    /* HTTP headers. */
  },
  cloudHeaders: {
    /* Cloud-specific HTTP headers (e.g., "x-amz-"). */
  },
  metadata: {
    /* Metadata headers (e.g., "x-amz-meta-"). */
  }
}

with cloud-specific prefixes stripped out. Moreover, you can set cloud headers in the same manner by passing any of the fields above (“headers”, “cloudHeaders”, “metadata”) in the options object of a cloud request. In this manner, Sunny makes it easy to utilize metadata operations as part of your application, while remaining agnostic to your actual cloud provider. See the Sunny user guide for more information.

Error Handling

For programmatic ease, and to better abstract across cloud providers, Sunny wraps cloud-based errors with a custom class, that adds in some additional useful attributes. The CloudError class has the following members:

message: The error message (usually in XML). (from Error).
statusCode: HTTP status code, if any.

and error type methods:

isNotFound: Container / blob does not exist.
isInvalidName: Invalid name requested for container / blob.
isNotEmpty: Container cannot be deleted if blobs still exist within it.
isAlreadyOwnedByYou: Creating a container that is already owned by you.
isNotOwner: AWS and Google Storage have a global namespace, so if someone else owns the container name requested, your operations will fail.

Ultimately, Sunny “error” event listeners will get CloudError objects instead of raw Error objects on cloud method errors, which can be used to more easily programmatically react (especially since Sunny correctly handles different error codes and XML from cloud providers that correspond to the same type of error).

Putting it all Together, with a Little Help from Async.js

Now that we’ve seen how to create a Sunny connection, get a container and perform operations on blobs within the container, let’s put all of our code together in a single script. (Note: I’ve collapsed the error handling to a single utility function).

var sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection,
  errFn = function (err) { console.log(err); };

// Op 1: GET container.
var request = conn.getContainer("sunnyjs", { validate: true });
request.on('error', errFn);
request.on('end', function (results, meta) {
  var container = results.container;
  console.log("Retrieved container \"%s\".", container.name);

  // Op 2: PUT blob.
  var writeStream = container.putBlob("my-blob.txt");
  writeStream.on('error', errFn);
  writeStream.on('end', function (results, meta) {
    var blob = results.blob;
    console.log("Put blob \"%s\".", blob.name);

    // Op 3: GET blob.
    var readStream = container.getBlob("my-blob.txt");
    readStream.on('error', errFn);
    readStream.on('data', function (chunk) {
      console.log("Data chunk: \"%s\"", chunk);
    });
    readStream.on('end', function (results, meta) {
      var blob = results.blob;
      console.log("Get blob \"%s\".", blob.name);
    });
    readStream.end();
  });
  writeStream.write("My data string.");
  writeStream.end();
});
request.end();

Not too shabby! However, you’ll note that we have three serial cloud operations, each of which have to complete before the next starts, so we end up repeatedly nesting callbacks in “end” event listeners. Even with a mere three operations, the nesting makes the code hard to understand at first blush, and an indentation nightmare. Wouldn’t it be better if we could code our operations to match the general outline of what we’re trying to do? That is:

Get a container.
Once we have a container, put data to a blob.
Once we have put data to the blob, get the data back.

Fortunately, there are many asynchronous helper libraries for Node to help in just this situation. I like Async.js, which from the project page “provides around 20 functions that include the usual ‘functional’ suspects (map, reduce, filter, forEach…) as well as some common patterns for asynchronous control flow (parallel, series, waterfall…).”

In our simple three-cloud operation script, we just need async.series to serialize our operations for easier control flow. I am skipping over a lot of details about using Async.js, such as callbacks at the end of functions or errors, and passing state across serialized functions, but you can probably intuit most of how things are working here:

var async = require("async"),
  sunny = require("sunny"),
  conn = sunny.Configuration.fromEnv().connection,
  container,
  blob;

async.series([
  // Op 1: GET container.
  function (callback) {
    var request = conn.getContainer("sunnyjs", { validate: true });
    request.on('error', callback);
    request.on('end', function (results, meta) {
      container = results.container;
      console.log("Retrieved container \"%s\".", container.name);
      callback(null);  // Signal function "done" to async.series.
    });
    request.end();
  },

  // Op 2: PUT blob.
  function (callback) {
    var writeStream = container.putBlob("my-blob.txt");
    writeStream.on('error', callback);
    writeStream.on('end', function (results, meta) {
      blob = results.blob;
      console.log("Put blob \"%s\".", blob.name);
      callback(null);  // Signal function "done" to async.series.
    });
    writeStream.write("My data string.");
    writeStream.end();
  },

  // Op 3: GET blob.
  function (callback) {
    var readStream = container.getBlob("my-blob.txt");
    readStream.on('error', callback);
    readStream.on('data', function (chunk) {
      console.log("Data chunk: \"%s\"", chunk);
    });
    readStream.on('end', function (results, meta) {
      blob = results.blob;
      console.log("Get blob \"%s\".", blob.name);
      callback(null);  // Signal function "done" to async.series.
    });
    readStream.end();
  }
], function (err, result) {
  if (err) {
    console.log(err);
  } else {
    console.log("async.series is done!");
  }
});

which gives us our expected output:

Retrieved container "sunnyjs".
Put blob "my-blob.txt".
Data chunk: "My data string."
Get blob "my-blob.txt".
async.series is done!

Our code now appears in a serialized, contained order that matches the conceptual execution of the actual cloud operations, instead of a lot of callback nesting.

For more on developing with Async.js, please see the excellent Async.js Readme / tutorial. It is an investment well worth your time. Sunny internally uses Async.js extensively for unit tests. Explore the source tests directory for a lot of good use cases (parallel and serial).

Have Fun in the Sun

That wraps up our introduction to Sunny.js. Hopefully the library provides a useful interface for your cloud programming needs.

The project is still very much in an early development state. See the Sunny “to do” list to get a better idea of what features and fixes are coming down the pipeline. The big ticket items I would really like to see added are:

More cloud providers, notable Rackspace Cloud Files and OpenStack Storage.
A basic retry function wrapper to handle expected, occasional cloud operation failures and throttling.
Advanced cloud operation support for ACL’s, policies, etc.

Help, bug reports, and pull requests for the project are most welcome. Please review the developer’s guide, and contact me with any questions or comments.

Pivot Faceting (Decision Trees) in Solr 1.4.

2011-09-20T16:00:00+00:00

Solr faceting breaks down searches for terms, phrases, and fields in the Solr into aggregated counts by matched fields or queries. Facets are a great way to “preview” further searches, as well as a powerful aggregation tool in their own right.

Before Solr 4.0, facets were only available at one level, meaning something like “counts for field ‘foo’” for a given query. Solr 4.0 introduced pivot facets (also called decision trees) which enable facet queries to return “counts for field ‘foo’ for each different field ‘bar’” – a multi-level facet across separate Solr fields.

Decision trees come up a lot, and at work, we need results along multiple axes – typically in our case “field/query by year” for a time series. However, we use Solr 1.4.1 and are unlikely to migrate to Solr 4.0 in the meantime. Our existing approach was to simply query for the top “n” fields for a first query, then perform a second-level facet query by year for each field result. So, for the top 20 results, we would perform 1 + 20 queries – clearly not optimal, when we’re trying to get this done in the context of a blocking HTTP request in our underlying web application.

Hoping to get something better than our 1 + n separate queries approach, I began researching the somewhat more obscure facet features present in Solr 1.4.1. And after some investigation, experimentation and a good amount of hackery, I was able to come up with a “faux” pivot facet scheme that mostly approximates true pivot faceting using Solr 1.4.1.

We’ll start by examining some real pivot facets in Solr 4.0, then look at the components and full technique for simulated pivot facets in Solr 1.4.1.

Pivot Faceting in Solr 4.0

Pivot facets were added to Solr in SOLR-792. A good introductory article is available on the Solr.pl site. To see the basic operation in action, let’s just use the “example” setup that comes with the Solr 4.0 distribution (located at “solr_4.0_path/solr/example”).

Let’s start the Solr process:

# Start Solr as non-daemon.
$ cd solr_4.0_path/solr/example
$ java -jar start.jar

Next, we want to upload a series of documents. We’ll take the provided “exampledocs/books.csv” file, tweak it slightly and update via a CSV handler. The CSV format is: first line is field names, other lines are data. Note that I have changed some field names in the original sample “exampledocs/books.csv” file. The following should be written out to a new file, which I am calling “sample_books.csv”.

Note that we use _s fields for simplicity, forcing string fields for what would ordinarily be text fields – Solr facets only return results on indexed, not stored terms, and string fields are identical for both. In a real deployment, you would use copyField directives to copy text fields to string fields for faceting.

# Create CSV file.
$ vim sample_books.csv
id,cat_s,name_s,price_f,inStock_b,author_s,series_s,sequence_i,genre_s
0553573403,book,A Game of Thrones,7.99,true,George R.R. Martin,A Song of Ice and Fire,1,fantasy
0553579908,book,A Clash of Kings,7.99,true,George R.R. Martin,A Song of Ice and Fire,2,fantasy
055357342X,book,A Storm of Swords,7.99,true,George R.R. Martin,A Song of Ice and Fire,3,fantasy
0553293354,book,Foundation,7.99,true,Isaac Asimov,Foundation Novels,1,scifi
0812521390,book,The Black Company,6.99,false,Glen Cook,The Chronicles of The Black Company,1,fantasy
0812550706,book,Enders Game,6.99,true,Orson Scott Card,Ender,1,scifi
0441385532,book,Jhereg,7.95,false,Steven Brust,Vlad Taltos,1,fantasy
0380014300,book,Nine Princes In Amber,6.99,true,Roger Zelazny,the Chronicles of Amber,1,fantasy
0805080481,book,The Book of Three,5.99,true,Lloyd Alexander,The Chronicles of Prydain,1,fantasy
080508049X,book,The Black Cauldron,5.99,true,Lloyd Alexander,The Chronicles of Prydain,2,fantasy

We’ll upload this file to Solr using curl:

# Upload to Solr.
$ curl http://localhost:8983/solr/update/csv \
  --data-binary @sample_books.csv \
  -H 'Content-type:text/plain; charset=utf-8'

# Commit the upload.
$ curl http://localhost:8983/solr/update \
  --data-binary '<commit/>' \
  -H 'Content-type:application/xml'

You should now be able to query the 10 sample documents at: http://localhost:8983/solr/admin/form.jsp.

Now that we have some documents to work with, let’s do a simple pivot query on price by genre:

# Pivot query.
$ curl http://localhost:8983/solr/select --data indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.pivot=price_f,genre_s

(Note that I’ve added line breaks and escapes to show the parameters more clearly).

This gives us decision tree results for the facet_pivot field:

{
  /* ... */,
  "facet_counts":{
    /* ... */,
    "facet_pivot":{
      "price_f,genre_s":[{
          "field":"price_f",
          "value":5.99,
          "count":2,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":2}]},
        {
          "field":"price_f",
          "value":6.99,
          "count":3,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":2},
            {
              "field":"genre_s",
              "value":"scifi",
              "count":1}]},
        {
          "field":"price_f",
          "value":7.95,
          "count":1,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":1}]},
        {
          "field":"price_f",
          "value":7.99,
          "count":4,
          "pivot":[{
              "field":"genre_s",
              "value":"fantasy",
              "count":3},
            {
              "field":"genre_s",
              "value":"scifi",
              "count":1}]}]}}}

Nice intuitive results, for a fairly straightforward facet query. However, now to the bigger question – can we approximate this in Solr 1.4.1, which doesn’t have the facet.pivot query option?

Pivot Faceting in Solr 1.4.1

Solr 1.4.1 has much more limited facet support than 4.0. The building blocks that we will use to cobble together a “faux” pivot query are:

facet.field: The normal facet field option.
facet.query: The normal facet query option.
fq: Basic field queries (for restrictions).
Local Params: We use a couple of Solr local parameters.
- tag: Tags a fq with an arbitrary name.
- key: Tags a facet field an arbitrary name (instead of field name).
- ex: Excludes tagged fq’s from being operative on a given facet field/query.

Note that either facet fields or facet queries can be used with this technique – I’ll only show fields, but everything applies equally to queries.

Setup

At this point, you should take a Solr 1.4.1 distribution and set it up exactly as we did above for Solr 4.0 and upload our simple 10-document CSV file to the running server. For simplicity here (and to keep my head on straight), I ended up running my Solr 1.4.1 server on port 8984, so that I could also keep the Solr 4.0 server running on port 8983. Here’s what I did:

# Start Solr as non-daemon.
$ cd solr_1.4.1_path/solr/example
$ java -Djetty.port=8984 -jar start.jar

# (Copy sample_books.csv)
# Upload to Solr.
$ curl http://localhost:8984/solr/update/csv \
  --data-binary @sample_books.csv \
  -H 'Content-type:text/plain; charset=utf-8'

# Commit the upload.
$ curl http://localhost:8984/solr/update \
  --data-binary '<commit/>' \
  -H 'Content-type:application/xml'

From here on, it is assumed you now have a populated Solr 1.4.1 server running on port 8984 (switch addresses / ports as appropriate for your actual setup).

The starting point for our pivot facets is excluding certain query restrictions for facets. A basic example is provided for tagging and excluding facets on the Solr wiki.

Let’s do a simple facet query on prices with a restriction of genre_s:scifi:

# Restricted facet query.
$ curl http://localhost:8984/solr/select --data \
\indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&fq=genre_s:scifi\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field=price_f

Looking to our results in facet_fields, we see that we only have 2 hits (numFound), and the facet counts also add up to 2 (which represent our 2 SciFi books).

{
  /* ... */,
  "response":{"numFound":2, /* ... */},
  "facet_counts":{
    /* ... */,
    "facet_fields":{
      "price_f":[
        "6.99",1,
        "7.99",1]},
    /* ... */,
}

For situations like a drill down, Solr developers often want to run a basic query with full restrictions for the returned records, but get more information for facets. In this case, Solr allows tagging of fq’s, and keys / excludes on facets to conditionally remove fq’s from a given facet only.

So, let’s tag our fq as “SCIFI_FQ” and exclude it from our facet counts with ex, and then rename the facet results to “PRICE_KEY” using the key option:

# Tag fq and exclude only from the field facet.
$ curl http://localhost:8984/solr/select --data \
\indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&fq={\!tag=SCIFI_FQ}genre_s:scifi\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field={\!key=PRICE_KEY\ ex=SCIFI_FQ}price_f

Note that I have to escape the exclamation points and other characters for a command line example here. Now, let’s look at the results:

{
  /* ... */,
  "response":{"numFound":2, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "PRICE_KEY":[
        "5.99",2,
        "6.99",3,
        "7.95",1,
        "7.99",4]},
    /* ... */,
}

We can first see that the exclusion of the tagged “SCIFI_FQ” field query did not affect the overall numFound, which is still 2. However, for the facet field we applied the exclusion to, we now have facet results for records in the whole set (which is the effective query after the exclusion). Finally, our facet field has been renamed “PRICE_KEY” instead of the field name (“price_f”).

Constructing a Pivot Query

With the basic tag/key/exclude technique in mind, let’s now return to our original goal – create a pivot query on price by genre using Solr 1.4.1. We will do this by performing two queries:

Perform a facet query for the top price results ordered by index.
Create fq tagged exclusions for each facet result, then create multiple keyed facets on genre to give us each of our decision tree “leaf” results.

The first query is a very basic facet field query:

# First level facet field query.
$ curl http://localhost:8984/solr/select --data indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&facet.field=price_f

Which gives us four individual facet results: “5.99”, “6.99”, “7.95”, “7.99”.

{
  /* ... */,
  "response":{"numFound":10, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "price_f":[
        "5.99",2,
        "6.99",3,
        "7.95",1,
        "7.99",4]},
    /* ... */,
}

We take each of those results and create specific fq tagged restrictions:

fq={!tag=FQ5.99}price_f:5.99
fq={!tag=FQ6.99}price_f:6.99
fq={!tag=FQ7.95}price_f:7.95
fq={!tag=FQ7.99}price_f:7.99

Each excludes one of the components we’ll want facet results for our next level field (genre) on. To then get the pivot facet result for each of our four facets, we will exclude all the fq’s above except the matching one for the facet. Translating into facet parameters, this is:

facet.field={!key=5.99_GENRE ex=FQ6.99,FQ7.95,FQ7.99}genre_s
facet.field={!key=6.99_GENRE ex=FQ5.99,FQ7.95,FQ7.99}genre_s
facet.field={!key=7.95_GENRE ex=FQ5.99,FQ6.99,FQ7.99}genre_s
facet.field={!key=7.99_GENRE ex=FQ5.99,FQ6.99,FQ7.95}genre_s

The key is that we can specify multiple exclusions using a comma. Thus, looking to the facet key “5.99_GENRE”, we exclude all the fq restrictions except “FQ5.99”, which means that the facet results for that facet field key will be the facet counts for “fq=price_f:5.99” only. It’s kind of a twisted-double-negative logic, but it all works out.

Let’s put everything into our second-level query now:

# Second level pivot query.
$ curl http://localhost:8984/solr/select --data \
indent=on\
\&wt=json\
\&q=*%3A*\
\&rows=0\
\&facet=true\
\&facet.mincount=1\
\&facet.sort=index\
\&fq={\!tag=FQ5.99}price_f:5.99\
\&fq={\!tag=FQ6.99}price_f:6.99\
\&fq={\!tag=FQ7.95}price_f:7.95\
\&fq={\!tag=FQ7.99}price_f:7.99\
\&facet.field={\!key=5.99_GENRE\ ex=FQ6.99\,FQ7.95\,FQ7.99}genre_s\
\&facet.field={\!key=6.99_GENRE\ ex=FQ5.99\,FQ7.95\,FQ7.99}genre_s\
\&facet.field={\!key=7.95_GENRE\ ex=FQ5.99\,FQ6.99\,FQ7.99}genre_s\
\&facet.field={\!key=7.99_GENRE\ ex=FQ5.99\,FQ6.99\,FQ7.95}genre_s

Which gives us the “leaves” of the decision tree with our result keys: “5.99_GENRE”, “6.99_GENRE”, “7.95_GENRE”, and “7.99_GENRE”.

{
  /* ... */,
  "response":{"numFound":10, /* ... */},
 "facet_counts":{
    /* ... */,
    "facet_fields":{
      "5.99_GENRE":[
        "fantasy",2],
      "6.99_GENRE":[
        "fantasy",2,
        "scifi",1],
      "7.95_GENRE":[
        "fantasy",1],
      "7.99_GENRE":[
        "fantasy",3,
        "scifi",1]},
    /* ... */,
}

Looking at our original Solr 4.0 pivot query, we can cobble together our two Solr 1.4.1 queries to get an equivalent result. In the end, both produce the following decision tree for price by genre:

5.99: 2
- fantasy: 2
6.99: 3
- fantasy: 2
- scifi: 1
7.95: 1
- fantasy: 1
7.99: 4
- fantasy: 3
- scifi: 1

Victory!

Discussion and Practical Implications

Our “price by genre” example is a bit simplistic in that we can mostly get the same results with two standard Solr 1.4.1 facet field queries. But, the faux pivot facet technique really shines for a “foo by bar”-type query where there are large numbers of first (“foo”) level facet results. Say, the first level has 10 results, this would mean 11 queries (one for the top 10 “foo”’s, then one each for the 10 second-level “bar”’s for each “foo”). The faux pivot facet technique cuts this down to 2 queries total.

The method is generally applicable too. Although our examples here only use facet fields, the technique equally works for facet queries. And distributed search supports the approach as well.

Looking further, the technique can be applied to additional decision tree levels. In the Solr 4.0 world, this simply means adding another field like facet.pivot=price_f,genre_s,inStock_b to get further breakdowns for the “in stock” boolean field. For Solr 1.4.1, we would do a third query, with permutations of our previous tagged fq’s, as well as second-level fq’s. Then, we would have third-level keyed facet fields like: “6.99_fantasy_INSTOCK”, “6.99_scifi_INSTOCK”, etc. At this level, it certainly wouldn’t be pretty and would result in a beastly query, but shows that the technique only adds 1 more actual Solr query for each additional level in the faux decision tree.

However, on the topic of query complexity, it is fair to point out that this type of query hackery really should be done programmatically to ensure correctness, and definitely not via the manual queries I provided above using curl. It’s tough keeping track of just the 4 first-level pivots in our example above, let alone a larger first level group, or more than 2 levels deep of pivots. Another benefit is that you can collapse your tag / key names to integer or other simple keys, and then have your program match things up later for the final assembled decision tree result.

As a final performance note, the faux pivot facet approach doesn’t really lighten the Solr server load, it just collapses what would otherwise be multiple queries into one query.

Conclusion

Reflecting on the above method, pivot facets are possible in Solr 1.4.1 at a cost of n separate queries, where n is the number of levels in the decision tree. So, if reducing the number of round trips between a web application and Solr is the goal, and you need pivot facets in a pre-4.0 Solr, this may well be the ticket.

How I Learned to Stop Worrying and Love the Static Blog

2011-09-16T16:00:00+00:00

Farewell Blogger…

Blogger provides an easy, integrated blogging service, but I have not been posting with anywhere near the frequency I had hoped for, and have recently started thinking of the reasons why this is. The list I cam up with includes:

Page load times: To get syntax highlighting, a nice theme, etc., my posts have an incredible amount of download cruft.
Editing Interface: While blogger’s WYSIWYG editor is fairly intuitive and the HTML editor is the appropriate power tool, I don’t like the fact I have to (1) be online, and (2) find a lot of situations where I want something in between a WYSIWYG and raw HTML.
Offline Editing: Expanding on the above point, I travel a lot, and it would be great to be able to compose and view posts offline (like on a plane).
Versioning/Backups: Blogger doesn’t allow for easily versioning of posts, which would be nice.

I essentially run a programming blog, and as such, I don’t need a lot of frills, bells or whistles. What I really want is text-based, powerful and configurable blog engine. Enter Jekyll.

Hello Jekyll and GitHub!

After a decent amount of research, I settled on Jekyll. Jekyll is a static website generator, written in Ruby, supporting Markdown, Liquid templates, and custom Liquid extensions.

There are many stories of folks moving successfully to Jekyll, and the feature set really hit my pain points:

All Text: All files are text, and either configuration, markdown, template language, or whatnot.
Versioned: It’s all source, so place your source under Git, and you’re ready to go.
Offline: The whole site can be generated or locally served without and internet connection.
Markdown: There are other pre-processing options, but I just went with Markdown, and it’s really nice and easy to write posts now without jumping back and forth from a WYSIWYG editor to straight HTML. I stay in Markdown and everything (mostly) ends up looking correct.
Syntax Highlighting: Jekyll uses Pygments for source code highlighting, which is mostly pre-processed, and not an after-the-fact JavaScript processing step (which had previously been slowing down my Blogger site).
GitHub Support: GitHub’s default document generator is Jekyll, and (separately) has full website support with CNAMEs. This provides an easy means of both site storage / versioning as well as the actual serving.

However, this is not to say that a Jekyll site is everyone’s cup of tea. There are some drawbacks to this approach:

Static Site: The generated site is static, which means no database or dynamic pages for things like comments, etc. All dynamic content has to come in via JavaScript / JavaScript-based services (like Disqus for comments). And, you have to implement a manual RSS feed, etc.
Manual CSS, HTML, etc.: There are no built-in themes, etc., because you are crafting a site from scratch. Really, this isn’t too bad because other Jekyll-site authors often publicly version their source with permissive / public domain licenses at GitHub, so you can pull source for examples. Also, there are a lot of starter kits for static CSS / HTML sites. For me, it’s really a nice opportunity to get re-acquainted with site design.

Packing Up and Moving Out

Posts

The first step to migrating my Blogger site was to simply download all of the posts in a Jekyll-friendly format. I used a very simple script that a good job of downloading basic HTML from the Blogger feed URL.

I then took the HTML pages and further converted them to Markdown using the handy html2text script. From there, I had to manually edit the gremlins left in the conversion process, including redoing anchor tags, removing Blogger-specific extras, and getting local images into my source repository static media directory.

Jekyll has its own default (but customizable) format for blog post names. I like it, but it is slightly different than Blogger, which all my existing posts already use. Fortunately, each Jekyll post has a permalink attribute allowing the post to escape the normal URL processing rules and use a one-time path instead.

Comments

After getting my posts ported over, the next step was to get my comments out of Blogger and in to something that would work with the static Jekyll web site. Enter Disqus, which is inserted via JavaScript, and this is usable for my new static website.

Making things easier, Disqus has Blogger support. So, I first switched my Blogger site comments to Disqus, then used the Disqus blogger import tool, available at:

http://<DISQUS SHORT NAME>.disqus.com/admin/tools/import/platform/blogger/

I followed the instructions and started the Blogger comment import. I expected the import to finish quickly, as I only have a handful of comments at the time of the migration, but it ended up taking 6 or so hours. Searching the web, there are reports of the Disqus comment import taking up to 2-3 days, so the lesson is: be patient. Oh, and don’t get freaked out when your comments disappear entirely from Blogger in the meantime.

Once the import finished, I looked at my Blogger site and saw that all of the comments had successfully been transitioned over to Disqus. I then set up Disqus support on my post template using the Disqus “universal” installation guidelines.

Before fully transitioning over to my new Jekyll/GitHub site, I wanted to make sure that everything was good to go with Disqus. Fortunately, Disqus allows you to specify a “developer mode” to have comments pulled down, even when running a localhost site. With these configuration variables in the Disqus JavaScript block, I can view the comments for this post on my laptop at localhost:

// Live page URL.
var disqus_url = 'http://loose-bits.com/2011/09/16/moving-to-jekyll.html';

// Developer mode for localhost.
var disqus_developer = 1;

RSS Feed

You have to specify your own “atom.xml” file for RSS feeds. Fortunately, there are many examples from other Jekyll sites, including Tom Preston-Werner’s site, as well as a good Google Groups thread.

Here is my finished atom.xml for this site for reference.

Post Excerpts

My Blogger site displays a few recents post excerpts on the index page, which I liked. However, there is no native “excerpt” functionality in Jekyll at the moment. Fortunately, a very clever hack enabled me to add post excerpts of my most recent couple of articles to my new index page.

And, We’re Up!

And that finishes my journey from Blogger to Jekyll and GitHub. As a final commentary Blogger does a great many things right and is a solid platform, but my use cases favor power and extensibility over many of the conveniences that Blogger provides.

Although it took some time to put my new site together, everything feels “right” to me. Even writing this first post for the new site has been considerably easier – I’ve been both online and offline composing the post in TextMate (which has Markdown syntax highlighting as well as spell check), and it has never been as easy or fast to write up posts, so here’s to hoping for a more consistent stream of posts in the future!

ConstantDict, Yet Another Python Enumeration Pattern

2011-06-13T13:30:00+00:00

Introduction

The Python language does not natively provide enumerations. Rejected PEP 354 proposed a string-constructor-based enumeration, later provided as the separate enum package. The web is rife with various cookbook recipes and ad-hoc solutions – see, e.g., ActiveState and Stack Overflow.

Here is ConstantDict, one of my preferred enumeration patterns.

`ConstantDict`

This pattern (like most Python enumeration recipes) is quite simple and straightforward, so I’ll actually start with the enumeration class, and then discuss the merits / drawbacks after.

The enumeration class is as follows:

class ConstantDict(object):
    """An enumeration class."""
    _dict = None

    @classmethod
    def dict(cls):
        """Dictionary of all upper-case constants."""
        if cls._dict is None:
            val = lambda x: getattr(cls, x)
            cls._dict = dict(((c, val(c)) for c in dir(cls)
                             if c == c.upper()))
        return cls._dict

    def __contains__(self, value):
        return value in self.dict().values()

    def __iter__(self):
        for value in self.dict().values():
            yield value

Essentially, we create an internal dictionary of all class variables that are upper-cased (well, and methods too), and wrap up a couple of instance built-in methods. Implementing an enumeration with the pattern is pretty straightforward. Here is a “days of the week” enumeration contrived by multiply inheriting two base enumeration classes:

class Weekdays(ConstantDict):
    """Weekday days."""
    MONDAY = 'Monday'
    TUESDAY = 'Tuesday'
    WEDNESDAY = 'Wednesday'
    THURSDAY = 'Thursday'
    FRIDAY = 'Friday'

class Weekend(ConstantDict):
    """Weekend days."""
    SATURDAY = 'Saturday'
    SUNDAY = 'Sunday'

class DaysOfTheWeek(Weekdays, Weekend):
    """Days of the week."""
    pass

And a simple use case example:

# Get enumeration instance.
DAYS = DaysOfTheWeek()

print(DAYS.MONDAY)        # "Monday"
print('Saturday' in DAYS) # True
print('Notaday' in DAYS)  # False
print([x for x in DAYS])  # ["Monday", "Tuesday", ...]

Discussion

The ConstantDict pattern shines in a couple of key areas:

It has a terse, clean, and extensible structure. The enumerations are inheritable, and even support multiple inheritance (as the above examples shows). Also, the enumerations can be enhanced with helper methods, etc., because they are real classes.
The structure allows arbitrary enumeration values. In my example I used strings, but anything (even including functions) are valid values.
The enumeration members are real class variables and not strings. For me this is a key point over a pure string-based solution, as I want to be able to run pylint and have it throw errors on invalid enumeration names.

However, ConstantDict probably isn’t the best choice for many situations. Some of the drawbacks of the pattern include:

Enumeration members are unordered. Because everything is shoved into a dictionary, there are no order guarantees.
Despite being called ConstantDict, the enumeration values aren’t actually enforced constants.
Enumeration values must be manually specified and are not guaranteed to be unique. As noted above arbitrary (including identical) enumeration values are allowed. However, the pattern could be easily extended to raise an exception on detection of duplicate values.
An enumeration class cannot have other upper-cased class variables or members. This should not be too big of a deal as the enumeration class is ideally used just as a small constant holder and not a multi-purpose class.
The caller has to instantiate an instance of the ConstantDict class. This is necessary for the __contains__ and __iter__ convenience method. In the past, I have used a different version of the ConstantDict class that is used only at the class variable / method level.

There are certainly other tweaks and enhancements that can make the ConstantDict pattern better for a given situation. But, in its most basic form as described above, it provides an easy, extensible thin-wrapper enumeration.

Extending Django Settings with Derived Attributes and Methods

2011-04-20T05:07:00+00:00

Django settings construction and configuration is fairly straightforward. Configure Django and installed application settings, maybe add your own project-specific settings, and let Django take care of the rest. As long as you import settings from django.conf all the options and defaults are properly resolved into the ultimate settings object.

However, every so often I run across a situation where I wish the Django settings could do just a little bit more. Rather than try and wade into the depths of Django runtime configuration and risk mucking things up, the approach I take is to use a thin wrapper class around the already configured settings and pass through all calls directly.

What we essentially want is “pass through” wrapper class like:

from django.conf import settings as _settings

class Settings(object):
    """Thin settings wrapper."""

    def __init__(self, default_settings):
        """Initializer."""
        self.default_settings = default_settings

    def __getattr__(self, name):
        """Get setting."""
        return getattr(self.default_settings, name)

settings = Settings(_settings)

Fortunately, Django already has a slightly more complex version of this in the django.conf.UserSettingsHolder class, so we’ll use that instead. I usually have a common application in my Django projects and insert the settings wrapper as the common.settings or common.conf module.

From there, we can add any number of “extra” settings variables or derived methods. For instance, if we wanted to add a settings method to tell if our database was SQLite or PostgreSQL using knowledge of the driver names, we could implement some fairly simple methods:

def is_sqlite(self):
    """Return ``True`` if default database is SQLite."""
    # Driver: 'sqlite3'
    return 'sqlite3' in self.DATABASES['default']['ENGINE']

def is_postgres(self):
    """Return ``True`` if default database is PostgreSQL."""
    # Drivers: 'postgresql_psycopg2', 'postgresql'
    return 'postgresql' in self.DATABASES['default']['ENGINE']

While perhaps too simplistic and brittle for production use, the above example illustrates how easy it is to add extra dynamic checks. To use the above methods if very easy – just import settings from our module (here common.settings instead of django.conf).

from common.settings import settings

print("Using SQLite?: %s" % settings.is_sqlite())
print("Using Postgres?: %s" % settings.is_postgres())

which outputs:

Using SQLite?: True
Using Postgres?: False

for a project configured with 'ENGINE': 'django.db.backends.sqlite3'.

In a typical Django project there are several file paths that are usually derived and hacked into the settings file – the Django project root directory, the Django admin media directory, and sometimes the path to the settings file itself. Here are methods for our wrapped settings class to intuit these paths:

def settings_path(self):
    """Return path to project settings."""
    from os import path
    from django.utils import importlib

    # Have to use real settings directly because UserSettingsHolder
    # sets SETTINGS_MODULE to ``None`` as a class variable.
    mod = importlib.import_module(
      self.default_settings.SETTINGS_MODULE)

    # Split off the extension to replace ".pyc" with ".py"
    return path.splitext(path.abspath(mod.__file__))[0] + ".py"

def project_root_path(self):
    """Return path to project root."""
    from os import path

    # We "cheat" here to find project root because we have know
    # this file is ``PROJECT_ROOT/common/settings.py``.
    return path.abspath(path.join(path.dirname(__file__), ".."))

def admin_media_path(self):
    """Return path to Django admin media directory."""
    from os import path
    from django.contrib import admin

    return path.abspath(
      path.join(path.dirname(admin.__file__), "media"))

And we can view our results with:

from common.settings import settings

print("Settings file path: %s" % settings.settings_path())
print("Project root directory path: %s" %
      settings.project_root_path())
print("Admin media directory path: %s" %
      settings.admin_media_path())

which outputs:

Settings file path: /Users/ryan/demo/settings.py
Project root directory path: /Users/ryan/demo
Admin media directory path: /Users/ryan/env/demo/lib/python2.6/site-packages/django/contrib/admin/media

Putting a full file together with our simple extension methods. In our case, the finished “common/settings.py” file will be:

from os import path
from django.contrib import admin
from django.conf import settings as _settings, UserSettingsHolder
from django.utils import importlib

class CustomSettings(UserSettingsHolder):
    """Thin settings wrapper.

    .. note:: "Real" settings object is available as
      ``.default_settings`` member.
    """

    def is_sqlite(self):
        """Return ``True`` if default database is SQLite."""
        # Driver: 'sqlite3'
        return 'sqlite3' in self.DATABASES['default']['ENGINE']

    def is_postgres(self):
        """Return ``True`` if default database is PostgreSQL."""
        # Drivers: 'postgresql_psycopg2', 'postgresql'
        return 'postgresql' in self.DATABASES['default']['ENGINE']

    def settings_path(self):
        """Return path to project settings."""
        mod = importlib.import_module(
          self.default_settings.SETTINGS_MODULE)

        # Split off the extension to replace ".pyc" with ".py"
        return path.splitext(path.abspath(mod.__file__))[0] + ".py"

    def project_root_path(self):
        """Return path to project root."""
        return path.abspath(
          path.join(path.dirname(__file__), ".."))

    def admin_media_path(self):
        """Return path to Django admin media directory."""
        return path.abspath(
          path.join(path.dirname(admin.__file__), "media"))

settings = CustomSettings(_settings)

While these contrived example method extensions are solving problems perhaps best solved elsewhere (like in the actual project settings file), they serve to illustrate a simple means of how to extend the Django settings module easily and without interfering with the runtime settings configuration process.

Django Cloud Browser

2011-03-10T05:44:00+00:00

Introduction

I recently pushed the v.0.1.2 release of my new Django application, django- cloud-browser, to PyPi. Cloud Browser is a simple web-based file browser for cloud-based datastores, which so far includes Amazon S3, Rackspace Cloud Files (including OpenStack) and local file system. The application can expose read-only cloud files to users and/or administrators with configurations for both normal and administrative deployments.

The basic reference points to get started include:

Background

At work, we have some containers with very large numbers of objects – over 5 million and counting. While objects ultimately are named in a flat namespace, we divide up the namespace with slashes (e.g., “path/to/cloud/object/file.txt”). Nearly all cloud providers (Amazon S3, Microsoft Azure, and Rackspace) include functionality in their object APIs to return results grouped around hierarchical “implied” or “pseudo-“ directory objects based on a separator (like the slash that we use).

Despite the fact that Rackspace’s underlying REST and Python APIs both support implying directories from a delimiter like a slash, Rackspace’s own management console will only display objects using a completely flat namespace. As an example, I have an old side project where I stored US patent data in XML form in both Rackspace Cloud Files and Amazon S3, and broke up each object with a slash-delimited path. (I have obscured my bucket / container names in all of the screen shots in this post). Let’s view the objects in the Rackspace container using their web interface:

[![Rackspace control panel][img_rs_th]][img_rs] [img_rs_th]: http://loose-bits.com/media/img/2011/03/10/rackspace_thumb.png [img_rs]: http://loose-bits.com/media/img/2011/03/10/rackspace.png

Thus, you have to wade through multiple pages of flat objects to find one you are looking for, and all of the work in sectioning out the object namespace has basically gone to waste.

By contrast, Amazon’s web dashboard for S3, will imply pseudo-directories, as we can see from the analogous view in S3:

[![AWS web dashboard][img_aws_th]][img_aws] [img_aws_th]: http://loose-bits.com/media/img/2011/03/10/aws_thumb.png [img_aws]: http://loose-bits.com/media/img/2011/03/10/aws.png

Given the hassles with trying to browse our millions of objects in the Rackspace dashboard, I finally decided to bite the bullet and just code up a Django application to do the job (such that I could easily integrate it into our system at work). A few months later, I finally hacked together a first version of the django-cloud-browser application.

Django Cloud Browser

At a high level, Cloud Browser application exposes containers and objects in a Cloud datastore to users or administrators. Let’s look at a basic example:

Basic Deployment

On your system, first install the application:

$ pip install django-cloud-browser

Then, edit your Django “settings.py”. Here, I’m setting up a Rackspace Cloud Files account for browsing:

INSTALLED_APPS = (
    # ...
    'cloud_browser',
)

CLOUD_BROWSER_DATASTORE = "Rackspace"
CLOUD_BROWSER_RACKSPACE_ACCOUNT = "<my_account>"
CLOUD_BROWSER_RACKSPACE_SECRET_KEY = "<my_secret_key>"

Next, add the URLs to your “urls.py”:

urlpatterns = patterns('',
    # ...
    url(r'^cb/', include('cloud_browser.urls')),
)

And start up your Django project! You should be able to click through container and object navigation links, as well as page results when they are over the maximum number of objects per page (configurable via CLOUD_BROWSER_DEFAULT_LIST_LIMIT, but defaults to 20 objects). Here is my navigation into the same Rackspace object as before, but now the objects have implied pseudo-directories.

[![Basic Cloud Browser][img_basic_th]][img_basic] [img_basic_th]: http://loose-bits.com/media/img/2011/03/10/cb_basic_thumb.png [img_basic]: http://loose-bits.com/media/img/2011/03/10/cb_basic.png

Objects that lead to ultimate files will be downloaded / displayed when clicked. Breadcrumb links are displayed along the top of the page as well.

The application even figures out and sets appropriate headers for content type and content encoding, which means, e.g., that compressed files will be uncompressed on the fly by browsers that support the compression scheme. At this point, it is worth noting that all access through Cloud Browser is read-only, although in the future we will look to enhancements like uploads, metadata retrieval / modification, etc.

Configuration Options

Other Datastores

In addition to Rackspace Cloud Files, the application supports Amazon S3 as well as the local file system (with a mock container / objects scheme imposed on real directories and files). Examples:

# AWS S3
CLOUD_BROWSER_DATASTORE = "AWS"
CLOUD_BROWSER_AWS_ACCOUNT = "<my_account>"
CLOUD_BROWSER_AWS_SECRET_KEY = "<my_secret_key>"

and

# Local filesystem
CLOUD_BROWSER_DATASTORE = "Filesystem"
CLOUD_BROWSER_FILESYSTEM_ROOT = "/usr/share/doc"

White / Black Lists

By default, all containers and all objects are browsable. However, the application supports basic white and black lists at the container level. If a whitelist is specified, only those containers will be browsable.

CLOUD_BROWSER_CONTAINER_WHITELIST = (
    'my_container',
    'more_stuff',
)

If a blacklist is specified, only those containers are excluded from browsing.

CLOUD_BROWSER_CONTAINER_BLACKLIST = (
    'secret_stuff',
    'archive',
)

If both are specified, access is effectively only what is allowed by the whitelist.

View Decorators

The settings variable CLOUD_BROWSER_VIEW_DECORATOR can be set to a function (e.g., a decorator) that wraps all browsing views. Setting this to Django’s login_required decorator permits only logged-in users to use the Cloud Browser.

Static Media

Cloud Browser contains a minimal amount of CSS and JavaScript. If you set up Django to statically serve these files, then set the CLOUD_BROWSER_STATIC_MEDIA_DIR variable to the relative path from settings.MEDIA_ROOT. The application has a fallback where if this is not set, the CSS / JavaScript will be included (inefficiently) in the HTML of each application view. See the documentation for further details.

Admin Deployment

Cloud Browser also allows for integration with the Django admin. Here, we just make a simple change to our “urls.py”:

urlpatterns = patterns('',
    # ...
    # Admin URLs. Note: Include ``urls_admin`` **before** admin.
    url(r'^admin/cb/', include('cloud_browser.urls_admin')),
    url(r'^admin/', include(admin.site.urls)),
)

It is also a really good idea to manually set the “staff required” decorator in “settings.py” to mirror the other admin view restrictions:

from django.contrib.admin.views.decorators import staff_member_required
CLOUD_BROWSER_VIEW_DECORATOR = staff_member_required

And here’s the same view as above in the admin:

[![Cloud Browser admin (open)][img_admin_open_th]][img_admin_open] [img_admin_open_th]: http://loose-bits.com/media/img/2011/03/10/cb_admin_open_thumb.png [img_admin_open]: http://loose-bits.com/media/img/2011/03/10/cb_admin_open.png

The administrative views also have a closable container element (click the +/- box in the upper right hand corner to toggle):

[![Cloud Browser admin (closed)][img_admin_closed_th]][img_admin_closed] [img_admin_closed_th]: http://loose-bits.com/media/img/2011/03/10/cb_admin_closed_thumb.png [img_admin_closed]: http://loose-bits.com/media/img/2011/03/10/cb_admin_closed.png

This work is at a very early stage and will likely change a lot, as there are many unresolved issues: links don’t appear on the dashboard because there are no models, there are no configurable admin permissions to the Cloud Browser views (either all or none), etc. But, it’s close enough for now.

Conclusion

Cloud Browser provides a very basic cloud file viewer for your Django projects, but it gets the job done, particularly for Rackspace Cloud Files. Any and all feedback and bugs are strongly encouraged – just open a ticket on the GitHub issues page. Happy browsing.

Browserless AJAX Testing with Rhino and Envjs, Part 2.

2011-02-16T15:17:00+00:00

Introduction

This is the second of two posts looking at browserless AJAX / JavaScript testing with Rhino and Envjs. Last time, we set up Rhino and Envjs to run a basic set of QUnit tests from the command line, exercising pure JavaScript code with no AJAX or DOM manipulation. Today, we will add tests for code that hacks on the DOM and makes AJAX calls.

Let’s start with the files we had last time (with links to previous blog post):

env.rhino.1.2.js: Envjs for Rhino.
qunit.css: QUnit styles. (Link to known working version).
qunit.js: QUnit code. (Link to known working version).
setup.js: Our custom hook Envjs to QUnit.
run-tests.js: Include our setup script and actually invoke the tests at on our test HTML page.
my-lib.js: An example JavaScript library (that we want to test).
my-tests.js: An example custom QUnit tests for my-lib.js.
test.html: Basic QUnit HTML page (see QUnit documentation) with script links to “my-lib.js” and “my-tests.js”.

New Files and Tests

Let’s add some new libraries and tests to our working set, to include DOM actions and AJAX:

jquery.js: The venerable jQuery library, which we’ll use for DOM and AJAX calls.
my-tests2.js: Our new tests, described in full detail below.
my-lib2.js: We’ll add two functions – one which does simply DOM manipulation with events, the other which does an AJAX call and inserts the text from another URL into a div element.

// my-lib2.js
/**
 * Add text and clickable elements to div.
 */
function addTextAndClicker($div, text) {
  $div.
    text(text).
    css('color', "rgb(0, 0, 0)").
    css('cursor', "pointer");
  $div.click(function () {
    $div.
      css('color', "rgb(255, 255, 255)").
      css('cursor', "default");
  });
}

/**
 * Dump text from URL into 'code' div.
 */
function dumpText($div, url, callback) {
  $.ajax({
    type: 'GET',
    url: url,
    dataType: 'text',
    success: function (data) {
      // Update the div.
      // Note: The "<pre/>" tag has some issues with Envjs.
      $("<code />").appendTo($div).text(data);

      // User callback.
      if (callback) {
        callback();
      }
    }
  });
}

Interestingly, I originally had the dumpText() function create a new “<pre/>” tag, but something between Rhino and Envjs did not like it, and Rhino would error out on running this code. So, I switched to “<code/>” and everything worked fine. I guess the point is that nothing absolutely substitutes for a real browser, but Envjs does a really great job notwithstanding!

Serving Our Files for AJAX

One important thing to note is that because we are doing AJAX calls now, we quickly will run into the same origin policy. To easily get around this for purposes of this demonstration, open a new terminal window, point it to the test directory we’re working in and serve all files in the directory directly from localhost with the python built-in web server (here I chose port 8001):

$ cd /path/to/test/code
$ python -m SimpleHTTPServer 8001 .

Check things out by navigating a real browser to: http://127.0.0.1:8001/test.html. Our QUnit tests from last time should run in the window.

Testing our New Code

Now that we have created our new library and are running a slim web server (so we can actually make AJAX requests), let’s write some test code in “my- tests2.js”. At the module level, we make use of the QUnit setup and teardown options to attach test div elements to the QUnit fixture element, so we can actually do some (temporary) DOM manipulation.

We use an asynchronous test pattern for the dumpText AJAX tests to handle latency between the call and response. The call to stop() stops QUnit execution to allow the code and tests to run. If the asynchronous callback doesn’t finish (and invoke start()) within the designated time period (here 1 second), then QUnit will raise an error. Assuming we have given ourselves enough time, once start() is invoked, then QUnit starts up again for the remaining tests and a final summary. For a good overview of QUnit testing with asynchronous examples, see “How to Test your JavaScript Code with QUnit”.

Here’s everything together for “my-tests2.js”:

module("my-lib2", {
  /**
   * Add new elements before each test.
   */
  setup: function () {
    var self = this;

    // Store test page fixture.
    self.$fixture = $('#qunit-fixture');

    // Create new test div's.
    self.$addText = $('<div/>').
      attr('id', "add-text").
      appendTo(self.$fixture);

    self.$dumpText = $('<div/>').
      attr('id', "dump-text").
      appendTo(self.$fixture);
  },

  /**
   * Remove tests elements after.
   */
  teardown: function () {
    var self = this;

    // Remove test div's
    self.$addText.remove();
    self.$dumpText.remove();
  }
});

test("addTextAndClicker", function () {
  var self = this;
  equals(self.$addText.text(), "", "Div starts empty");

  // Add the text and clicker.
  addTextAndClicker(self.$addText, "Hi There!");
  equals(self.$addText.text(), "Hi There!", "Div has text");
  equals(self.$addText.css('color'),
         "rgb(0, 0, 0)", "Starting color");
  equals(self.$addText.css('cursor'),
         "pointer", "Changed cursor");

  // Click the clicker and check.
  self.$addText.click();
  equals(self.$addText.text(), "Hi There!","Div still has text");
  equals(self.$addText.css('color'),
         "rgb(255, 255, 255)", "Changed color");
  equals(self.$addText.css('cursor'),
        "default", "Changed cursor");
});


test("dumpText", function () {
  var self = this;
  equals(self.$dumpText.text(), "", "Div starts empty");

  // Stop QUnit execution to wait for AJAX call and tests.
  stop(1000);  // Wait up to 1 second.

  // Perform AJAX call.
  // Just dump our setup script as straight text.
  dumpText(self.$dumpText, "http://127.0.0.1:8001/setup.js",
    function () {
    // Now, run tests.
    var $code = self.$dumpText.find('code');
    equals($code.length, 1, "Verify code element.");

    // Find some known text strings.
    var find = function (text) {
      ok($code.html().indexOf(text) != -1,
         "Find string \"" + text + "\"");
    };
    find("beforeScriptLoad");
    find("afterScriptLoad");
    find("QUnit test runner loaded");
    find("QUnit.log");

    // Restart QUnit execution.
    start();
  });
});

Now, lets update our “test.html” file to run both our previous and new unit tests (and add the jQuery dependency):

<html>
  <head>
    <title>QUnit</title>
    <link rel="stylesheet" href="qunit.css" type="text/css" />
    <script type="text/javascript" src="qunit.js"></script>
    <script type="text/javascript" src="jquery.js"></script>
    <script type="text/javascript" src="my-lib.js"></script>
    <script type="text/javascript" src="my-tests.js"></script>
    <script type="text/javascript" src="my-lib2.js"></script>
    <script type="text/javascript" src="my-tests2.js"></script>
  </head>
  <body>
    <h1 id="qunit-header">QUnit Test Suite</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture"></div>
  </body>
</html>

Finally, we need to tweak “run-tests.js” to point to the locally served test page instead of simply the local filesystem.

load('setup.js');

console.log("Starting QUnit tests...");
window.location = "http://127.0.0.1:8001/test.html";

And, let’s run the tests!

$ java -cp ~/rhino1_7R2/js.jar org.mozilla.javascript.tools.shell.Main -opt -1 run-tests.js
[  Envjs/1.6 (Rhino; U; Mac OS X x86_64 10.6.6; en-US; rv:1.7.0.rc2) Resig/20070309 PilotFish/1.2.13  ]
Starting QUnit tests...
* QUnit test runner loaded.
  * { addTwo }( 0 )[ PASS ]  Add nothing., expected: 0
  * { addTwo }( 1 )[ PASS ]  Add numbers., expected: 3
  * { addTwo }( 2 )[ PASS ]  Add negatives., expected: -3
  * { addTextAndClicker }( 3 )[ PASS ]  Div starts empty, expected: ""
  * { addTextAndClicker }( 4 )[ PASS ]  Div has text, expected: "Hi There!"
  * { addTextAndClicker }( 5 )[ PASS ]  Starting color, expected: "rgb(0, 0, 0)"
  * { addTextAndClicker }( 6 )[ PASS ]  Changed cursor, expected: "pointer"
  * { addTextAndClicker }( 7 )[ PASS ]  Div still has text, expected: "Hi There!"
  * { addTextAndClicker }( 8 )[ PASS ]  Changed color, expected: "rgb(255, 255, 255)"
  * { addTextAndClicker }( 9 )[ PASS ]  Changed cursor, expected: "default"
  * { dumpText }( 10 )[ PASS ]  Div starts empty, expected: ""
  * { dumpText }( 11 )[ PASS ]  Verify code element., expected: 1
  * { dumpText }( 12 )[ PASS ]  Find string "beforeScriptLoad"
  * { dumpText }( 13 )[ PASS ]  Find string "afterScriptLoad"
  * { dumpText }( 14 )[ PASS ]  Find string "QUnit test runner loaded"
  * { dumpText }( 15 )[ PASS ]  Find string "QUnit.log"

*****************
* QUnit Results *
*****************
* PASSED:  16
* FAILED:  0
* Completed  16  tests total in  2.981  seconds.

From the results, we can see that we now have all of our unit tests running from pure JavaScript code to our new DOM manipulation and AJAX calls.

Other Enhancements

Beyond the setup we discussed today, there are several really cool additions and improvements you can add to your browserless testing solution.

Use Mocks for AJAX Calls

Although running a simple python web server gets us through this demonstration pretty easily for our AJAX calls, in practice AJAX calls might rely on more than a simple static server can provide. At that point, you either have to deal with running your development web application server at the same time, or switch to mocking out the AJAX calls. A great solution is to use fantastic MockJax jQuery plugin, which overrides $.ajax() calls to substitute with a configurable mock response.

Add Code Coverage Reports

Amazingly, the JSCoverage code coverage library does work with this Rhino and Envjs setup. I hooked things up by running jscoverage-server in a similar fashion to how we did the command line python server above (which completely replaces the need for it), and then added the following to “setup.js” in the QUnit.done option:

QUnit.done = function (fail, total){
  // ... previous code ...

  // Code coverage report.
  if (window.jscoverage_report) {
    console.log("Writing coverage report.");
    jscoverage_report();
  }
};

It is important that the coverage report come at the very end of the QUnit.done handler to ensure that all of our tests truly are finished running. See the JSCoverage manual for more details about running the coverage server and writing reports.

Conclusion

Revisiting the original motivation for this post series, the original pain point was that frontend tests are complicated to write and a hassle to run. With the fully browserless testing environment we have explored here, it is possible to create a system out of Rhino, Envjs and QUnit that runs all tests with a single command line argument, making the former hassle into something so easy you could add it to your pre-/post- commit hooks in your version control software. And, by being able to write our tests in mostly straight QUnit (with a little extra setup help), test writing for developers remains relatively straightforward, even dealing with AJAX, etc.

Browserless AJAX Testing with Rhino and Envjs, Part 1.

2011-02-08T06:15:00+00:00

Updates - 5/27/2011

Since this original post, there has been a lot of good feedback surrounding getting an Envjs + QUnit + Rhino setup going, so please read the comments at the bottom of this post. Here are the summarized points:

QUnit: Newer versions of QUnit have issues with Envjs, adding a <pre/> to output that Envjs can’t handle. A comment from “Ryan” (not me – I’m “Ryan Roemer” below) also points out the QUnit has changed the function signature of the logging callbacks and links to an improved “setup.js” file. I have inserted links to a known working version of QUnit.
jQuery 1.5: jQuery 1.5+ also has issues with Envjs due to calls to a couple of methods not currently available in either Envjs 1.2 or 1.3. There is a patch on GitHub for the issue for Envjs 1.3. I have backported the patch (and some other fixes) to a forked version of Envjs 1.2 on my GitHub account.
Envjs 1.3: Envjs 1.2 was a single file, with primarily Rhino support. Envjs 1.3 is massive enhancement project that supports a large number of JavaScript engines including: Node.js, SpiderMonkey, etc. It’s still in early development, but definitely worth taking a look at.

Introduction

This is the first of two blog posts focusing on browserless testing of AJAX / JavaScript using Rhino and Envjs. Today, we discuss getting browserless QUnit tests up and running from the command line. A follow-on post will discuss running QUnit tests which involve DOM manipulation and AJAX interaction.

Testing web applications can be an enormous pain. Tests take time to write correctly and usefully, the interaction between the front and back ends is complicated, and it is often a hassle just to get people to run the tests on a regular basis. At work as of late, we have been trying to kick start our JavaScript unit testing – our system has a Django/Python backend, and a JavaScript (mostly jQuery-based) frontend.

On the backend, we are in good shape, test-wise – the Django test case extensions to the Python unit test framework make backend unit testing easy and straightforward. Everything runs fairly quickly, and from the command line. In fact, we have actually made a (soft) requirement that all developers run a Fabric target called “precommit” that runs all of our Django unit tests (in addition to Python and JavaScript style checkers).

On the frontend, it’s a different story. Our previous test suite relied entirely on a browser and a running development server loaded with data in a known state. Our unit tests were written against QUnit, and for the most part worked fine for the code they did exercise. But, even where our frontend tests would catch newly introduced bugs, all too often the tests just weren’t run, and we would actually discover the bug several commit cycles later.

While we clearly need to integrate a fully automated infrastructure around browser-based tests (using great tools like Selenium or Windmill), my thoughts turned to the idea that the heavyweight nature of full end-to-end AJAX testing was getting in the way of us writing and running frontend tests. More specifically, the difficulties come down to two fundamental problems – our tests on the frontend are: (1) not easy to write and run, and (2) are not integrated into an automated, “one-touch” running scheme.

JavaScript Engines and the Pesky Browser Environment

Last week I decided to finally bite the bullet and push hard for the easiest and simplest JavaScript testing solution – execution of a full suite of QUnit tests that would execute without a browser from a single command line command.

The first stop is getting a JavaScript engine to actually run JavaScript (test and code) in lieu of a browser. Fortunately, there are lots of great open source options including:

Rhino: Mozilla’s Java JavaScript implementation.
SpiderMonkey: Mozilla’s C implementation of JavaScript.
V8: Google’s JavaScript virtual machine for Chrome.

A JavaScript engine only gets you pure JavaScript language support. The hard part is that any nontrivial web application with a good amount of JavaScript has a large percentage of code that manipulates the DOM and essentially needs a browser environment for a lot of functionality. So much so, that as general assertion, it is probably not worth trying to separate out pure and browser- dependent JavaScript code in development solely for this purpose.

Thus, the second step necessary for running QUnit tests is a simulation of the browser environment (e.g., window, console, etc.), just without the browser. This is a harder problem.

A few years ago, while working on some prototype code for a friend’s startup, I started researching browserless JavaScript testing, specifically looking for browser emulation options. At the time, the only serious emulation solution I found was in a blog post by John Resig (of jQuery fame), where he had successfully hacked together a pure JavaScript browser environment (called “env.js”) that was capable of AJAX, DOM manipulation, and even running the jQuery set of QUnit tests.

Unfortunately, my buddy’s side project eventually went on hiatus, and I never got around to rigging up a completely browserless test project. Working on purely backend server code for the past couple of years (C++, C#, python), it has taken me some time to get both the opportunity and desire to research a browserless JavaScript environment.

So years later, looking back into the issue, I was pleased to revisit Resig’s proof-of-concept browser simulation environment and find that it had evolved into a full open source project – Envjs – which provides an amazingly powerful browser-like environment, capable of DOM manipulation, AJAX interaction, cookie setting, etc.

Making Rhino, Envjs, QUnit, and JavaScript Tests Play Nicely Together

Rediscovering Envjs and finding it in good shape meant that I simply had to plug in a JavaScript engine, make sure it works with QUnit, and then fire up my test suites without a browser. I ended up choosing Rhino as my JavaScript engine as Envjs has specific support for it, but presumably other engines can be made to work with Envjs and get equivalent results as I will describe below.

So, let’s get started.

Rhino and Envjs Installation

In terms of extra tools / libraries, all we need is Rhino and Envjs:

Rhino is available from the Mozilla FTP site. I ended up going with rhino1_7R2 – the Envjs web site notes that it is best to go with a recent Rhino release. The package can be installed anywhere, but we are going to need to know the path to the file “js.jar” in the package to pass as a class path argument to Java.

After installing, let’s check that Rhino actually works:

$ cd /path/to/rhino
$ java -cp js.jar org.mozilla.javascript.tools.shell.Main -opt -1
Rhino 1.7 release 2 2009 03 22
js> print("hello world");
hello world
js> quit();

Next, install Envjs. I went with Envjs 1.2. (I did end up having to patch the library to get cookies working properly, but that’s a post for another day – and presumably fixed in trunk). Envjs can be installed anywhere the rest of your code can access it – I went with the same JavaScript directory as all our other third party library code.

Example Libraries and Test Code

Ignoring how I actually installed and placed files, for the rest of this discussion, let’s assume we have a simplified world view in a single working directory, where we’ll place all our third party, test and library code. None of these files really need (or should) be in the same directory, but it is easier to describe things without multiple directory traversal.

Here is the third party code:

env.rhino.1.2.js: The Envjs file we downloaded. If you are using jQuery 1.5+, you may want this patched Envjs instead.
qunit.css: QUnit styles. (Link to known working version).
qunit.js: QUnit code. (Link to known working version).

We are going to create two more files in a subsequent section.

setup.js: Hook Envjs to QUnit.
run-tests.js: Include our setup script and actually invoke the tests at on our test HTML page.

And, here is some library and test code files:

my-lib.js: An example JavaScript library (that we want to test).

function addTwo(x, y) {
    return x + y;
}

my-tests.js: An example custom QUnit tests for my-lib.js.

module("My Module");

test("addTwo", function () {
    equals(addTwo(0, 0), 0, "Add nothing.");
    equals(addTwo(1, 2), 3, "Add numbers.");
    equals(addTwo(-1, -2), -3, "Add negatives.");
});

test.html: Basic QUnit HTML page (see QUnit documentation) with script links to “my-lib.js” and “my-tests.js”.

<html>
  <head>
    <title>QUnit</title>
    <link rel="stylesheet" href="qunit.css" type="text/css" />
    <script type="text/javascript" src="qunit.js"></script>
    <script type="text/javascript" src="my-lib.js"></script>
    <script type="text/javascript" src="my-tests.js"></script>
  </head>
  <body>
    <h1 id="qunit-header">QUnit Test Suite</h1>
    <h2 id="qunit-banner"></h2>
    <div id="qunit-testrunner-toolbar"></div>
    <h2 id="qunit-userAgent"></h2>
    <ol id="qunit-tests"></ol>
    <div id="qunit-fixture"></div>
  </body>
</html>

Once we’ve got all of these custom and third party files, we can just open test.html in a browser, and the QUnit tests should all run, pass and look something like:

[![QUnit results][img_qunit_th]][img_qunit] [img_qunit_th]: http://loose-bits.com/media/img/2011/02/08/qunit_thumb.png [img_qunit]: http://loose-bits.com/media/img/2011/02/08/qunit.png

Configuring Envjs and QUnit

We need to hook QUnit into Envjs, so we can display output to stdout (using console.log()). Fortunately, QUnit explicitly provides a lot of hooks to do everything we want, which I’ve implemented in a file named “setup.js”, placed alongside our other JavaScript libraries.

I mostly followed the excellent guide from the Envjs folks, and here’s the gist of my configuration for “setup.js”:

load('env.rhino.1.2.js');
load('qunit.js');

var starttime = new Date().getTime();

// Envjs/QUnit Bridge.
Envjs({
    // Straight from the Envjs guide.
    scriptTypes: {
        "": true,
        "text/javascript": true
    },
    // Straight from the Envjs guide.
    beforeScriptLoad: {
        'sharethis': function (script) {
            script.src = '';
            return false;
        }
    },

    // Hook QUnit logging to console.
    afterScriptLoad: {
        'qunit': function () {
            var count = 0, testName;

            console.log("* QUnit test runner loaded.");

            // Grab current test name.
            QUnit.testStart = function(name, testEnvironment) {
                testName = name;
            };
            // Override log to display to stdout.
            QUnit.log = function (result, message) {
                // Strip out HTML in results messages.
                message = message.replace(/<\/?.*?>/g, '');
                console.log("  * {%s}(%s)[%s] %s",
                    testName, count++,
                    result ? 'PASS' : 'FAIL', message);
            };
            QUnit.done = function (fail, total){
                var endtime = new Date().getTime();
                var pass = total - fail;
                console.log("\n" +
                    "*****************\n" +
                    "* QUnit Results *\n" +
                    "*****************\n" +
                    "* PASSED: %s\n" +
                    "* FAILED: %s\n" +
                    "* Completed %s tests total in %s seconds.\n",
                    pass, fail, total,
                    parseFloat(endtime-starttime) / 1000.0);
           };
        },

        // Straight from the Envjs guide.
        '.': function (script) {
            script.type = 'text/envjs';
        }
    }
});

Note that the Envjs guide actually went further than this example, as the guide’s setup code writes the QUnit HTML report out to a new file. The interesting thing about running JavaScript in Rhino is that code runs with full system permissions outside of the JavaScript sandbox, so you can also do normal file system manipulation. (And, as the Envjs documentation notes with big scary warnings, there are huge security issues with running outside code within an Envjs / Rhino environment).

Test Runner

The last file we have to compose is our actual test runner. Fortunately, this is quite simple, as shown in “run-tests.js”– just load the setup and point to the test page:

load('setup.js');

console.log("Starting QUnit tests...");
window.location = "test.html";

Test it Out!

Now that we have Rhino, Envjs and QUnit all hooked together, it’s time to actually run some QUnit tests. Assuming your file / directory setup matches the above contrived example, we should be able to run the tests by invoking Rhino with our test runner script:

$ java -cp ~/rhino1_7R2/js.jar org.mozilla.javascript.tools.shell.Main -opt -1 run-tests.js
[  Envjs/1.6 (Rhino; U; Mac OS X x86_64 1 ... ]
Starting QUnit tests...
* QUnit test runner loaded.
  * { addTwo }( 0 )[ PASS ]  Add nothing., expected: 0
  * { addTwo }( 1 )[ PASS ]  Add numbers., expected: 3
  * { addTwo }( 2 )[ PASS ]  Add negatives., expected: -3

*****************
* QUnit Results *
*****************
* PASSED:  3
* FAILED:  0
* Completed  3  tests total in  1.989  seconds.

Awesome. We have QUnit testing goodness for our library, running within Rhino, and log / results output to the console, all through a single bash command. We can add and include more libraries and test suites to test.html, and they automatically get picked up by the test suite.

Next Time – Browserless AJAX, Playing with the DOM and More!

Today we examined and set up basic browserless JavaScript unit tests with Rhino and Envjs. Despite the promises of the title of this post, we’ll have to wait until next time to extend out QUnit test suite to include actual / mocked AJAX testing and DOM manipulation with this set up. But, just removing the browser from the frontend test path is a great step towards more lightweight JavaScript unit tests – and situating us for the type of tests that developers will want to write and run.

Rackspace Cloud Files and Pseudo-Directories

2010-12-28T19:57:00+00:00

Cloud Providers and Listing Storage Objects

Rackspace’s Cloud Files provide a basic distributed blob / file abstraction. Like Amazon Web Services Simple Storage Service (AWS S3) and Microsoft Azure Blobs, Cloud Files are organized into a two level hierarchy consisting of:

Containers: A collection of file objects. In AWS S3 parlance this is called a “bucket”. For all three cloud providers, a container is a single collection unit that cannot be further subdivided.
Storage Objects: A single file object. In AWS S3, this is an “object”, while in Azure this is a “blob”. A storage object for all the providers similarly cannot be further nested / subdivided.

Thus, there are no further levels of hierarchy than container, then object, and for all three providers, the storage object namespace is completely flat.

As most of the world is used to nested file hierarchies much deeper than this, cloud providers allow wide leeway in naming storage objects, most notably allowing characters like a slash (“/”). To provide the illusion of a nested hierarchy, AWS S3 and Azure Blobs provide listing operations that take a delimiter character to treat as a nested directory delimiter. In this manner, calls to the list objects API will return results as if there are intermediate directories in the (really) flat storage object namespace.

The Old Way - Rackspace Cloud Files and Dummy Directory Objects

To much frustration, Rackspace Cloud Files did not originally provide a delimiter query API for treating a delimiter as a nested directory organizer. Instead of delimiter queries, Rackspace required that clients upload a storage object of type “application/directory” at each level in which a nested pseudo- directory was desired. Only then would the results start to resemble those from AWS / Azure with delimiter queries.

Here’s a quick example showing how the dummy directory objects work. First let’s create a container:

>>> import cloudfiles
>>> conn = cloudfiles.get_connection(ACCOUNT, SECRET_KEY)
>>> cont = conn.create_container("my_test_container")
>>> cont.list_objects()
[]

Now, put in some objects with internal slashes, and list all.

>>> cont.create_object("foo/bar/file1.txt").write("some text.")
>>> cont.create_object("foo/bar/file2.txt").write("some text.")
>>> cont.create_object("foo/diff/file3.txt").write("some text.")
>>> cont.create_object("baz/file4.txt").write("some text.")
>>> cont.list_objects()
['baz/file4.txt',
 'foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

We can use the “prefix” parameter to limit files to only “foo”, but the internal pseudo-directories are not inferred.

>>> cont.list_objects(prefix="foo")
['foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

The “path” parameter will infer directories, but not without internal dummy directory objects.

>>> cont.list_objects(path="foo")
[]

So, let’s create the dummy objects for “foo”.

>>> dummy = cont.create_object("foo")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

>>> dummy = cont.create_object("foo/bar")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

>>> dummy = cont.create_object("foo/diff")
>>> dummy.content_type = "application/directory"
>>> dummy.sync_metadata()
>>> dummy.write("")

And, now we get the pseudo-directory results we expect for “foo” and children paths.

>>> cont.list_objects(path="foo")
['foo/bar',
 'foo/diff']

>>> cont.list_objects(path="foo/bar")
['foo/bar/file1.txt',
 'foo/bar/file2.txt']

>>> cont.list_objects(path="foo/diff")
['foo/diff/file3.txt']

While the dummy objects technically make nested pseudo-directories possible, the onus falls to the user to create and maintain all the dummy objects. This really becomes a pain when creating new objects that might contain nested delimiters for which there is not already a delimiter (say adding a nested pseudo-directory). The user then has to either check / scrub all existing files for missing dummy objects, or create duplicate dummy directory objects on each new object creation.

The New Way - Rackspace Cloud Files and Delimiter Queries

Fortunately, Rackspace finally enabled “real” delimiter queries in their list storage objects API. I couldn’t actually find the date that this was added to REST API and added server side (as the current API PDF doesn’t even note the change, although I expect it soon will).

I first noticed the new query functionality when perusing commits on the Python API GitHub page. On Dec. 3, 2010, a change was pushed that added the “delimiter” parameter to object listing queries for the Python API. The Python API was incremented to version 1.7.4 for the change, so make sure to pull the proper version of the library to test the new feature out:

$ pip install -U https://github.com/rackspace/python-cloudfiles/tarball/1.7.4

Interestingly, while the container list_objects() and list_objects_info() methods picked up a delimiter parameter, the get_objects() method did not. At any rate, this gets the job done, and we can remove our dummy objects now to retry our examples:

>>> cont.delete_object("foo")
>>> cont.delete_object("foo/bar")
>>> cont.delete_object("foo/diff")

Let’s check we don’t have the dummy objects anymore:

>>> cont.list_objects()
['baz/file4.txt',
 'foo/bar/file1.txt',
 'foo/bar/file2.txt',
 'foo/diff/file3.txt']

And now, let’s try equivalent queries using the prefix and delimiter parameters. (Note that you must append an extra slash (e.g. “foo/”, not “foo”) for the queries to work.)

>>> cont.list_objects(prefix="foo/", delimiter="/")
['foo/bar/',
 'foo/diff/']

>>> cont.list_objects(prefix="foo/bar/", delimiter="/")
['foo/bar/file1.txt',
 'foo/bar/file2.txt']

>>> cont.list_objects(prefix="foo/diff/", delimiter="/")
['foo/diff/file3.txt']

… and we finally get the results we want without having to track or create any extraneous objects.

At the time of this post, many clients do not take advantage of more convenient / advanced object querying. The CyberDuck client currently will only display pseudo-directories via the dummy directory objects method. In fact, Rackspace’s own control panel doesn’t display any pseudo-directories using either the old dummy directory or new delimiter methods. So, hopefully with the new delimiter query API support, some folks out there will make some New Year’s resolutions to improve pseudo-directory support in Cloud Files clients.

Celery Logging with Python Logging Handlers

2010-12-13T15:54:00+00:00

Logging in Celery

We use Celery as our backend messaging abstraction at work, and have lots of disparate nodes (and across different development, test, and production deployments). As each system deployment now contains a large (and growing) number of nodes, we have been making a heavy push towards consolidated logging through a central logging server sink, using syslog (specifically rsyslog).

So, ideally, we would just use Celery’s configuration to specify a syslog handler (or maybe use a pipe). Unfortunately, it seems there is just not a simple way of doing this straight from Celery, as the only logging configuration parameters (with possible values) are:

CELERYD_LOG_FILE = "/path/to/file.log"  # File logging.
                                        # (OR)
CELERYD_LOG_FILE = None                 # stderr.

which means you either get a file logger (filehandler) or a stderr logger (streamhandler).

Hints and guidance on the web in terms of getting arbitrary logging handlers (or syslog specifically) is sparse, as well as noisy. A lot of the code and discussion I found used Celery as the backend for a generic logging framework or library. But, I found very little in terms of getting Celery tasks and processes to actually log to syslog.

Bringing Arbitrary Logging to Celery

As nothing magically appeared as the “right” solution, I considered a couple of different ways to hook things up:

Watched Files: Have Celery log to a file per usual, then add extra configuration and scripts to watch the file for changes and submit the changes to syslog directly. I didn’t go with this approach, as I really prefer to have configuration for our project within generic Python settings, and not need extra, system-specific scripts and setup.
Patch Celery: The Celery logging hooks and code are fairly straightforward. I could have added the patch and submitted upstream. Unfortunately, for the project at work, we are moving to fast to wait for changes, and as we are looking forward to some upstream Celery releases, I’d rather not maintain a private custom patch set through all of that.
Monkey Patch Celery: In the same vein as the former option, the same hooks and changes could be applied as a monkey patch instead. This is what I eventually went with.

Patching Logging Handlers into Celery

Monkey patching is oft-controversial and generally discouraged practice, as getting things wrong is easy, and interactions within the patched library / code can get really messy. In our case, I reluctantly chose monkey patching, as it was a short patch, easy to disable, and I had no good other solutions.

Now that I’ve given the standard disclaimer, let’s get to the patch! The relevant Celery function we want to patch in celery.log._setup_logger():

def _setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):

    if logger.handlers:                 # already configured
        return logger

    handler = _detect_handler(logfile)
    handler.setFormatter(formatter(format, use_color=colorize))
    logger.addHandler(handler)
    return logger

As we can see, there is only one shot at a handler, and it comes from celery.log._detect_handler(), which either returns a stream or file handler. What I would ideally like is to define be able to define a Django settings variable (we use Django and not native Celery settings at work):

PATCH_CELERYD_LOG_HANDLERS = (
    lambda: logging.FileHandler("path/to/another_log.txt"),
    lambda: logging.handlers.SysLogHandler(),
)

and define a new _setup_logger():

import celery.log

from celery.log import ColorFormatter
from django.conf import settings

old_setup_logger = celery.log._setup_logger

def _setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):

    patched_logger = old_setup_logger(
        logger, logfile, format, colorize, formatter, **kwargs)

    for handler_fn in getattr(settings, "PATCH_CELERYD_LOG_HANDLERS", []):
        handler = handler_fn()
        handler.setFormatter(formatter(format, use_color=colorize))
        patched_logger.addHandler(handler)

    return patched_logger

I would have preferred a pure “*args, **kwargs” version of the patched function, but unfortunately, we need to dig out the formatter parameter if set, so I copied the method signature of the original. Also, I used lambda’s in PATCH_CELERYD_LOG_HANDLERS and called the function in the new setup logger so as to create a new handler object each time for loggers. It is important to have different handler objects because formats attach to handler objects, and Celery uses different formats for general/default/process logging vs. task-specific logging. If a handler has last been set to a task logger format, that includes a format for among other things “task_name”, then you will get a KeyError on trying to issue a logging statement.

Other than that, the patched method is very easy to read – it iterates the custom handler function list and creates new handlers for each different logging object. Putting this in to a patched version, I also added a marker attribute to the logger object (to tell the patching is done), as well as the patching code to lock, switch and mark the methods:

import logging
import celery.log

from celery.log import ColorFormatter
from django.conf import settings

old_setup_logger = celery.log._setup_logger  # Store the real method.
def my_setup_logger(logger, logfile, format, colorize,
        formatter=ColorFormatter, **kwargs):
    """Replacement for :method:`celery.log._setup_logger`."""

    patched_logger = old_setup_logger(
        logger, logfile, format, colorize, formatter, **kwargs)

    # Check if not patched.
    if not getattr(patched_logger, "_LOG_PATCH_DONE", False):
        # Lock and patch.
        logging._acquireLock()
        try:
            handler_fns = getattr(settings, "PATCH_CELERYD_LOG_HANDLERS", [])
            for handler_fn in handler_fns:
                handler = handler_fn()
                handler.setFormatter(formatter(format, use_color=colorize))
                patched_logger.addHandler(handler)

            # Mark logger object patched.
            setattr(patched_logger, "_LOG_PATCH_DONE", True)
        finally:
            logging._releaseLock()

    return patched_logger

# Apply patches.
CELERY_MOD_PATCHED = False
if not CELERY_MOD_PATCHED:
    logging._acquireLock()  # Lock logging during patch.
    try:
        celery.log._setup_logger = my_setup_logger  # Patch old w/ new.
        CELERY_MOD_PATCHED = True
    finally:
        logging._releaseLock()

With the patch applied, any number of Python standard logging handlers can now be added to the celery task / process loggers.

Applying the Patch at Runtime

So we’ve defined the patch, but how do we apply it? Part of the problem is celery configuration. If you apply the patch too early, you get the following exception:

.../python2.6/site-packages/celery/loaders/default.py:53: NotConfigured:
No celeryconfig.py module found! Please make sure it exists and is available to Python.

So, we have to find a proper place in the initialization process, after Celery settings. The best solution I found for this is to create a faux Django “application” , that contains only the patch code. Let’s call the application “patch_celery”:

$ cd path/to/project
$ django-admin.py startapp patch_celery

Copy and paste all of the patch code above into the init.py. And then, we simply add the patch application to the Django settings.py installed apps:

INSTALLED_APPS = (
    # ...
    'djcelery',
    'patch_celery',
    # ...
)

The best place to put the patch is right after the djcelery application. Now, add your patch variable, PATCH_CELERYD_LOG_HANDLERS, to settings.py with any designated handlers (per previous example) and you’re off with any logging handlers for celery tasks that you want! You can disable the patch by simply removing the “patch_celery” application from installed applications in settings, which enables easily toggling per appropriate context without major code swaps.

Afterthoughts

The monkey patch to Celery certainly gets the job done for now. However, the fact that the full method signature for celery.log._setup_logger() is used makes the patch very brittle – if a parameter changes in the underlying method, the patched method may cease to work, or worse, give a wrong parameter to the underlying method.

Rackspace Cloud Files and Servicenet

2010-10-18T00:01:00+00:00

Cloud Files, Cloud Servers

At work, we use Rackspace Cloud Files for bulk blob storage of large sets of documents (in our case, patents). We use a farm of Rackspace Cloud Servers to process and serve blobs from Cloud Files. Rackspace’s offering has been very solid to date (modulo some idiosyncrasies of their particular APIs) and quite interestingly, the underlying software is being released as an open source project, OpenStack.

One of the nice things building a solution on Rackspace is that bandwidth is free between Cloud Servers and Cloud Files, if properly configured. However, as our recent experience bears out, there are some pitfalls and caveats towards getting everything set up correctly.

Public, Private Interfaces and Servicenet

Rackspace Cloud Servers are essentially Xen VMs that come with two network interfaces: (1) a public interface on eth0 with a world-routable IP address, and (2) a “private” interface on eth1 with an RFC 1918 address. (The private interface isn’t really “private” in the sense that any other VMs on the same network can route to your VM, so security-wise, it should be treated as a public, internet-facing interface). For more details, Rackspace has published a good article on the public and private interfaces.

For billing purposes, bandwidth between Cloud Files and a Cloud Server VM on the public interface is charged per normal, whereas bandwidth on the private interface is free. To enable Cloud Files communication on the private interface, you can either set an environment variable in your application like follows:

export RACKSPACE_SERVICENET=True

or programmatically specify the private network, as shown here using the Python Cloud Files client library:

import cloudfiles
conn = cloudfiles.get_connection("acct", "api_key", servicenet=True)

In both cases, “Servicenet” refers to the private interface network internal to a Rackspace data center.

Servicenet Gotchas

All seems pretty simple! However, when we finally enabled “servicenet=True” in our Cloud Servers, we immediately hit errors talking to Cloud Files and everything refused to run. After a lot of back and forth with the Rackspace folks, we finally found out the main issue: Servicenet only works between Cloud Files and Cloud Servers in the same data center.

Before we ran into this snag, we hadn’t even checked that our servers and blob storage were in the same data center. As it turned out, our Cloud Files location was in Texas (DFW) and our Cloud Servers were in Chicago (ORD). And, after much hand-wringing, we learned that: (1) new Cloud Servers will most likely be provisioned in Chicago, and (2) you can’t easily migrate Cloud Files accounts from Texas to Chicago.

So, how can one easily tell where your Cloud Servers and Cloud Files are, and whether or not Servicenet will be available? The answer for Cloud Servers is straightforward: in the Rackspace management dashboard, look at your server list and the “Datacenter” column. In our case, all servers list as “ORD1”, which means Chicago. Moving on to Cloud Files, one way to figure things out is to fire up a Python shell and issue some basic commands to see what we have. In all cases, looking at a connection object’s “connection_args[0]” value will yield a string for public interfaces like:

storage<NUMBER>.<DATACENTER>.clouddrive.com

and on the private interface:

snet-storage<NUMBER>.<DATACENTER>.clouddrive.com

So, on to some example code using the Python client. Here is a Chicago Cloud Server talking to Chicago Cloud Files on the public interface:

>>> import cloudfiles

>>> chicago_pub_conn = cloudfiles.get_connection("chicago_acct", "1234567890", servicenet=False)
>>> chicago_pub_conn.connection_args[0]
'storage101.ord1.clouddrive.com'
>>> chicago_pub_conn.list_containers()
['foo1', 'bar1', 'baz1']

… and on the private interface:

>>> chicago_priv_conn = cloudfiles.get_connection("chicago_acct", "1234567890", servicenet=True)
>>> chicago_priv_conn.connection_args[0]
'snet-storage101.ord1.clouddrive.com'
>>> chicago_priv_conn.list_containers()
['foo1', 'bar1', 'baz1']

Now, switching to a Chicago Cloud Server talking to Texas Cloud Files on the public interface, we get:

texas_pub_conn = cloudfiles.get_connection("texas_acct", "1234567890", servicenet=False)
texas_pub_conn.connection_args[0]
'storage101.dfw1.clouddrive.com'
>>> texas_pub_conn.list_containers()
['foo2', 'bar2', 'baz2']

but, trying to talk on the private interface yields the following exception:

>>> texas_priv_conn = cloudfiles.get_connection("texas_acct", "1234567890", servicenet=True)
>>> texas_priv_conn.connection_args[0]
'snet-storage101.dfw1.clouddrive.com'
>>> texas_priv_conn.list_containers()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/cloudfiles/connection.py", line 410, in list_containers
    raise ResponseError(response.status, response.reason)
cloudfiles.errors.ResponseError: 404: Not Found

Confusingly, the exception raised is a 404 “Not Found”, when it really should be something more context-specific.

Final Thoughts

The last time I chatted with the Rackspace folks, they noted most new accounts will have both Cloud Servers and Cloud Files allocated in Chicago, so using Servicenet should be as simple as setting the appropriate variable. In our specific case, we got around our Texas-Chicago split by allocating an entirely new account that has both servers and files in Chicago, and ported over our Texas files. Somewhat tedious, but a worthwhile long-term investment.

But, if you are trying to use Servicenet and experience 404’s or other strange failures, you probably want to go through the above exercise and identify the data center where your servers and blobs reside. And, if you use Cloud Servers and Cloud Files and aren’t already using Servicenet, you definitely should because you are paying for bandwidth that should otherwise be free!

Distributed Task Locking in Celery

2010-10-10T20:20:00+00:00

Background

We use the Celery distributed task queue library at work, which is great for running asynchronous tasks across multiple processes and servers. Celery has both user-initiated and periodic (think cron replacement) tasks, and we have found in practice that the system distributes tasks quite nicely across our farm of celery servers.

One issue we have is that for several of our periodic tasks, we need to ensure that only one task is running at a time, and that later instances of the same periodic task are skipped if a previous incarnation is still running.

The Celery documentation has a cookbook recipe for this scenario: “Ensuring a task is only executed one at a time”. The crux of the solution is to make a distributed lock using the Django cache (memcached in the example) with the following lambda’s:

lock_id = "something unique"
lock_expire = 60 * 5  # five minutes

acquire_lock = lambda: cache.add(lock_id, "true", lock_expire)
release_lock = lambda: cache.delete(lock_id)

Non-persistent Locks?

This approach works fine if the cache is shared across all celery worker nodes and the cache is persistent. However, if memcached (or some other non- persistent cache) is used and (1) the cache daemon crashes or (2) the cache key is culled before the appropriate expiration time / lock release, then you have a race condition where two or more tasks could simultaneously acquire the task lock. This “distributed cache lock” approach has been discussed in various posts, which all acknowledge the danger of relying on memcached for persistent data.

Distributed Locks with Redis

As noted by one of the links above, the simplest solution to this problem if you like using the cache for distributed locks is to switch to memcachedb which is not a caching solution per se, but rather a persistent key-value store that implements the memcached interface.

However, for our system, using memcached as our Django cache is great in its current non-persistent form for what it is – a cache. So, I investigated more generally to find a high performance, persistent key-value store (ideally with a decent Python interface). After reviewing a lot of neat and interesting systems, I finally settled on Redis. Redis provides a very high performance key-value store (data is maintained in memory) with persistence (using the append only file feature) and distribution / replication. As an added bonus, I found the server setup, installation and CLI interaction to be very straightforward.

So, back to locking… The Redis python client already has a lock class with “with” operator support:

import redis

with redis.Redis().lock("my_key"):
    print("Got lock.")

However, the above example is a blocking lock, and for the “single task” issue, we want a non-blocking lock, that simply exits if the lock is not acquired:

import redis

have_lock = False
my_lock = redis.Redis().lock("my_key")
try:
    have_lock = my_lock.acquire(blocking=False)
    if have_lock:
        print("Got lock.")
    else:
        print("Did not acquire lock.")

finally:
    if have_lock:
        my_lock.release()

Beyond this simple example, the Lock class implements key expiration (via the Redis setnx command) to enable timeouts in the python client.

Enforced Single Celery Task

So, bringing this back to our celery tasks, we can use this distributed lock to have our tasks try to acquire a non-blocking lock, and exit if the lock isn’t acquired. Also, we want to set a lock timeout (lasting for a generous overestimate of task duration time) so that tasks will eventually be able to re-acquire the lock if some task / celery node hard crashes or goes in to an unresponsive state before releasing the lock.

All of this can be put together as a decorator around a Task.run() method:

import redis

REDIS_CLIENT = redis.Redis()

def only_one(function=None, key="", timeout=None):
    """Enforce only one celery task at a time."""

    def _dec(run_func):
        """Decorator."""

        def _caller(*args, **kwargs):
            """Caller."""
            ret_value = None
            have_lock = False
            lock = REDIS_CLIENT.lock(key, timeout=timeout)
            try:
                have_lock = lock.acquire(blocking=False)
                if have_lock:
                    ret_value = run_func(*args, **kwargs)
            finally:
                if have_lock:
                    lock.release()

            return ret_value

        return _caller

    return _dec(function) if function is not None else _dec

Note that this decorator preserves task return values. If your tasks don’t have return values, you can get rid of the ret_value code.

Using the decorator is easy – just annotate a task run() method:

from celery.task import Task

class SingleTask(Task):
    """A task."""

    @only_one(key="SingleTask", timeout=60 * 5)
    def run(self, **kwargs):
        """Run task."""
        print("Acquired lock for up to 5 minutes and ran task!")

… and your task will only ever have one running instance at any given time.