Zen Templates Development Journal, Part 1

Once my concept was well documented (see Part 0), I was ready to start developing my prototype. I had many questions I needed to answer:

  • Is the concept feasible, useful, and superior in some meaningful way to existing solutions?
  • What kind of performance could I expect?
  • How would derivation work in real-world scenarios? What shortcomings are there in the simple system described in my concept?
  • Ditto for content injection and substitution.
  • How would I handle model data scoping?
  • Would it be better to parse the template file into a DOM Document, or to parse it as a stream?
I started with an extremely simple use case: two templates, one deriving from the other; a couple of model data fields, one of them a list; use of basic derivation, injection, and substitution, with no scope concerns. I built the template files and dummy model data, such that I could quickly tell what was working and what wasn’t (“this text is from the parent template”, “this text is from the child template”, “this text shouldn’t appear in the final output”, etc.) I also build a dead-simple servlet that did nothing but build the model, run the template renderer, and dump the output to the HttpServletResponse.
With this most basic use case in place, I started to work on the template renderer. I started with the state, initialization, and entry point. For the state, I knew I needed a reference to the template file, and I needed a Map for the model data. For initialization, I needed to take in a template file, and initialize the empty model. For convenience, I allowed initialization with a relative file path and a ServletContext, to allow referencing template files located under WEB-INF, so that they couldn’t be accessed directly (a good practice borrowed from JSP.) I created accessors for adding data to the model.
The entry point was a function simply named “render”. It was 5 lines long, each line calling an unimplemented protected method: loadTemplate, handleDerivation, handleInjection, handleSubstitution, and writeOut. These were the five steps needed for my basic use case.
I then went to work on building out each of the steps. The easiest was loading the template file from disk into a DOM Document using Jsoup (since XML handlers don’t deal well with HTML content). At this point I added two Documents to the renderer’s state, inDoc and outDoc. inDoc was the Document parsed from the template file, outDoc was the Document in memory being prepared for output. I followed a basic Applications Hungarian Notation, prefixing references to the input document with “in” and references to the output document with “out”.
Since I needed to be able to execute derivation recursively, I decided to do it by creating a new renderer, passing it the parent template, and running only the loadTemplate and handleDerivation methods; then the parent renderer’s outDoc became the child’s starting outDoc. In this way, if the parent derived from another template, the nested derivation would happen automagically. I would then scan the parent document for ID’s that matched elements in the child document, and replace them accordingly. Derivation was done.
Next up was injection: I started out by iterating over the keys in my model Map, scanning the document for matching class names. Where I found them, I simply replaced the innerHtml of the found element with the toString() value of the model data; if the model data was an array or collection, I would instead iterate the data, duplicating the matched element for each value, and replacing the cloned element’s innerHtml with the list item’s toString() value. This was enough functionality for my simple test case.
Reaching the home stretch, I did substitution ridiculously simply, using a very basic regex to find substitution placeholders (${somevariable}) and replacing each with the appropriate value from the model. I knew this solution wouldn’t last, but it was enough for this early prototype.
Last up was writing the rendered output, and in this case, I allowed passing in an HttpServletResponse to write to. I would set the content type of the response, and dump the HTML of my final Document to the response stream.
I ran it, and somehow, it actually worked. I was shocked, but excited: in the course of a little over an hour, I had a fully working prototype of the most basic functions of my template engine. Not a complete or usable product by any means, but an excellent sign. I made a few tweaks here and there, correcting some minor issues (collection items were being inserted in reverse order, for example), but it was pretty much solid. I also replaced my RegEx-based substitution mechanism with the StrSubstitutor from commons-lang; this was pretty much a direct swap that worked perfectly.

Time for the next test, my moderate complexity test case.

The Development Stream

I was reading today about GitHub’s use of chat bots to handle releases and continuous integration, and I think this is absolutely brilliant. In fact, it occurs to me that using a chat bot, or a set of chat bots, can provide an extremely effective workflow for any continuous-deployment project. Of course, it doesn’t necessarily have to be a chat room with chat bots; it can be any sort of stream that can be updated in real-time – it could be a Twitter feed, or a web page, or anything. The sort of setup I envision would work something like this:

Everyone on the engineering team – developers, testers, managers, the whole lot – stay signed in to the stream as long as they’re “on duty”. Every time code is committed to a global branch – that is, a general-use preproduction or production branch – it shows up in the stream. Then the automated integration tests run, and the results are output to the stream. The commit is deployed to the appropriate environment, and the deployment status is output to the stream. Any issues that occur after deployment are output to the stream as well, for immediate investigation; this includes logged errors, crashes, alerts, assertion failures, and so on. Any time a QA opens a defect against a branch, the ticket summary is output to the stream. The stream history (if it’s not already compiled from some set of persistent-storage sources) should be logged and archived for a period of time, maybe 7 to 30 days.

It’s very important that the stream be as sparse as possible: no full stack traces with error messages, no full commit messages, just enough information to keep developers informed of what they will need to look into further elsewhere. This sort of live, real-time information stream is crucial in the success of any continuous-deployment environment, in order to keep the whole team abreast of any issues that might be introduced into production, along with when and how they were introduced.

Now, what I’ve described is a read-only stream: you can’t do anything with it. GitHub’s system of using an IRC bot allows them to issue commands to the bot to trigger deployments and the like. That could be part of the stream, or it could be part of another tool; as long as the deployment, and its results, are output to the shared stream for all to see. This is part of having the operational awareness necessary to quickly identify and fix issues, and to maintain maximum uptime.

There are a lot of possible solutions for this sort of thing; Campfire looks particularly promising because of its integration with other tools for aggregating instrumentation data. If you have experience with this sort of setup, please post in the comments, I’d love to hear about it!