Behold, My Newest Creation!

I’m entirely too proud too announce my latest creation, Rogue Prognosticator. This is a web-based application for doing project estimation and schedule for software development. I’ve written about these topics before, and rest assured I will again; you can count on the concepts you see discussed here being taken into account in the software.

Right now the site is in open beta, free for public use. As features are added, some may be subscriber-only, or may start out being subscriber-only.

This project breaks a lot of new ground for me, and I’ve learned a lot already.

  • It’s my first project from scratch using ASP.NET MVC or Entity Framework.
  • It’s my first personal project in production using C# or .NET.
  • It’s the first time I’ve used Windows Azure.
  • It’s the first time I’ve used UserVoice.
  • It’s the first time I’ve used continuous deployment from Git on a personal project in production.
  • It’s the first time I’ve used SQL Server on a personal project in production.
  • It’s the first time I’ve used WordPress in production.
This project was, as you may have guessed, the source for my post on entity framework model queries, as well as my post on value-based prioritization.
I’ve been using the project as I’ve been building it, and it’s already been an excellent tool for me. Prioritizing features by estimated return was a particularly enlightening experience; it really helped me to get an objective look at the near-term plan and organize development more effectively.
I’ll still talk here about the nitty-gritty of development but official product announcements will be coming through the product blog, Rogue Prognostications. I hope that others will find this project as useful as I have. Please feel free to drop any comments, questions, suggestions, or other feedback on the Rogue UserVoice.
More to come – watch this space!

Zen Templates Development Journal, Part 2

Having my concept complete (see Part 0) and my simple test case working (see Part 1), I was ready to start on my moderate-complexity test case. This would use more features than the simple test case, and more pages. I really didn’t want to have to build a complete site just for the proof of concept, so I decided to use an existing site, and I happened to have one handy:

The site is currently built in HTML5, using server-side includes for all of the content that remains the same between pages. It seemed like a pretty straightforward process to convert this to my template engine, so I got to work: I started with one page (the home page), and turned it into the master template. I took all of the include directives and replaced them with the content they were including. I replaced all of the variable references with model references using injection or substitution. I ID’d all the elements in the master template that would need to be replaced by child templates. I then made another copy of the homepage, and set it up to derive from the master template.

I didn’t want to convert the site to use servlets, since it wasn’t really a dynamic site; I just wanted to be able to generate usable HTML from my template files. So I created a new class that would walk a directory, parse the templates, and write the output to files in an output directory. Initially, it set up the model data statically by hand at the start of execution.

All was well, but I needed a way for the child template to add elements to the page, rather than just replacing elements from the parent template. I came up with the idea of appending elements, using a data attribute data-z-append=”before:” or “after:”, to cause an element to be appended to the page either before or after the element from the parent with the specified ID. This worked perfectly, allowing me to add the Google Webmaster Tools meta tag to the homepage.

With this done, I set to work converting the remaining pages. Most of the pages were pretty straightforward, being handled just like the homepage; I dumped the SSI directives, added some appropriate IDs and classes, and all was well. However, the software pages presented some challenges. For one thing, they used a different footer than the rest of the site. It was time to put nested derivation to the test.

I created a software page template, which derived from the master template, that appended the additional footer content. I then had the software pages derive from this template, instead of deriving from the master template and – by some stroke of luck – it worked perfectly on the first try. I wasn’t out of the woods yet, though.

The software pages also used SSI directives to dynamically insert the file size for downloadable files next to the links to download them. I wasn’t going to reimplement this functionality, however, I was prepared to replace these directives with file size data stored in the model. But I wanted to keep the model data organized, so I needed to support nesting. The software pages also used include directives to include a Google+ widget on the pages; this couldn’t be added to the template, as it was embedded in the body content, so it seemed like a perfect case for snippets – which meant I needed to implement snippet support.

Snippet support was pretty easy – find the data attribute, look up the snippet file, parse it as an HTML fragment, and replace the placeholder element with the snippet. Easy to implement, worked pretty well.

Nested properties I thought would be a breeze, as I had assumed it was natively supported by StrSubstitutor. Unfortunately it wasn’t, so I had to write my own StrLookup. I decided that, since I was already doing some complex property lookups for injection, I’d build a unified model lookup class that could act as my StrLookup and could be used elsewhere. I wanted nested scope support as well, for my project list: each project had an entry in the model, that consisted of a name, latest version, etc. I wanted the engine to iterate this list, and for each entry, rather than replacing the entire content of the element with the text value of the model entry, I wanted it to find sub-elements and replace each with an appropriate property of the model entry. This meant I needed nested scoping support.

I implemented this using a scope stack and a recursive lookup. Basically, every time a nested scope was entered (e.g., content injection using an object or map, or iteration over a list), I would push the current scope onto the stack. When the nested scope was exited (i.e., the end of the element that added the scope), I popped the scope off. When iterating a loop, at the start of the iteration, I’d push the current index, and at the end of the iteration, I’d pop it back off.

This turned out to be very complex to implement, but after some trial and error, I got it working correctly. I then re-tested against my simple test case, having to fix a couple of minor defects introduced there with the new changes. But, at last, both my simple and moderate test cases were working.

I didn’t like the static creation of model data – not very flexible at all – so I decided to swap it out with JSON processing. This introduced a couple of minor bugs, but it wasn’t all that difficult to get it all working. The main downside was that it added several additional dependencies, and dependency management was getting more difficult. I wasn’t too concerned on that front though, since I was already planning for the real product to use Maven for dependency tracking; I was just beginning to wish I had used Maven for the prototype as well. Oh well, a lesson for next time. For now, I was ready for my complex test case – I just had to decide what to use.

Zen Templates Development Journal, Part 0

Zen Templates is based on an idea I’ve been tossing around for about six months. It started with a frustration that there was no way to validate a page written in PHP or JSP as valid HTML without executing it to get the output. It seemed like there had to be a way to accomplish that.

I started out looking into what I knew were global attributes, class and id. I did some research and found that the standard allows any character in a class or id; this includes parens and such, meaning a functional syntax could be used in these attributes, which a parser could then process to render the template.

This seemed practically ideal; I could inject content directly into the document, identifying the injection targets using these custom classes. I toyed with the idea of using this exclusively, but saw a couple of serious shortcomings. For one, sometimes you want to insert dynamic data into element attributes, and I didn’t see a good way to handle that without allowing a substitution syntax like that of JSP or ASP. I decided this would be a requirement to do any real work with it.

I also saw the problem of templates. Often each page in a dynamic site is called a template, but I’m referring to the global templates that all pages on a site share, so there is only one place to update the global footer, for example. I had no good solution for this. I started thinking about the idea of each page being a template and sharing a global template – much akin to subclasses in object oriented programming, one template derives from another.

I started batting around different possibilities for deriving one template from another, and decided on having a function (in a class attribute) to identify the template being derived from, with hooks in the parent template to indicate which parts the “subtemplate” would be expected/allowed to “override”.

I let the idea percolate for a while – a few weeks – as other things in life kept me too busy to work on it. Eventually it occurred to me that all these special functions in class attributes were pretty messy, and a poor abstraction for designers. It could potentially interfere with styling. It would produce ugly code. And I was on a semantic markup kick, and it seemed like a perfect opportunity to do something useful with semantic markup.

So I started rebuilding the concept and the current Zen Templates was born (and the name changed from its original, Tabula Obscura.) As I committed to maximizing the use of semantic markup and keeping template files as valid, usable HTML, I reworked old ideas and everything started falling into place. I remembered that the new HTML5 data attributes are global attributes as well, and would give me an additional option for adding data to markup without interfering with classes or ruining the semantics of the document.

I ironed out all the details of how derivation should work; it made semantic sense that a page that derived from another page could be identified by class; and, taking a page from OOP’s book, it made sense that an element in the subpage with the same ID as an element in the parent page would override that element, making any element with an ID automatically a hook; somewhat like a subclass overriding methods in the superclass by defining a method with the same signature.

I sorted out the details of content Injection as well, thinking that, semantically, it made sense that an element of a certain class would accept data from the model with an identifier matching the class name. Even better, I didn’t need a looping syntax; if you try to inject a list of items into an element, it would simply repeat the element for each item in the list. This simplified a lot of syntax I’ve had to use in the past using JSP or Smarty.

I also wrote out how substitution should work, using a syntax derived from JSP. Leaning on JSP allowed me to answer a lot of little questions easily. I would try to avoid the use of functions in the substitution syntax, because it does make the document messier, and forces more programming tasks on the designer. I conceded that some functions would likely be unavoidable.

When I felt like I had most of the details ironed out, a guiding principal in mind, and a set of rules of thumb to help guide me through questions down the road, I was ready for a prototype. Stay tuned for Part 1!

Zen Templates Development Journal, Part 1

Once my concept was well documented (see Part 0), I was ready to start developing my prototype. I had many questions I needed to answer:

  • Is the concept feasible, useful, and superior in some meaningful way to existing solutions?
  • What kind of performance could I expect?
  • How would derivation work in real-world scenarios? What shortcomings are there in the simple system described in my concept?
  • Ditto for content injection and substitution.
  • How would I handle model data scoping?
  • Would it be better to parse the template file into a DOM Document, or to parse it as a stream?
I started with an extremely simple use case: two templates, one deriving from the other; a couple of model data fields, one of them a list; use of basic derivation, injection, and substitution, with no scope concerns. I built the template files and dummy model data, such that I could quickly tell what was working and what wasn’t (“this text is from the parent template”, “this text is from the child template”, “this text shouldn’t appear in the final output”, etc.) I also build a dead-simple servlet that did nothing but build the model, run the template renderer, and dump the output to the HttpServletResponse.
With this most basic use case in place, I started to work on the template renderer. I started with the state, initialization, and entry point. For the state, I knew I needed a reference to the template file, and I needed a Map for the model data. For initialization, I needed to take in a template file, and initialize the empty model. For convenience, I allowed initialization with a relative file path and a ServletContext, to allow referencing template files located under WEB-INF, so that they couldn’t be accessed directly (a good practice borrowed from JSP.) I created accessors for adding data to the model.
The entry point was a function simply named “render”. It was 5 lines long, each line calling an unimplemented protected method: loadTemplate, handleDerivation, handleInjection, handleSubstitution, and writeOut. These were the five steps needed for my basic use case.
I then went to work on building out each of the steps. The easiest was loading the template file from disk into a DOM Document using Jsoup (since XML handlers don’t deal well with HTML content). At this point I added two Documents to the renderer’s state, inDoc and outDoc. inDoc was the Document parsed from the template file, outDoc was the Document in memory being prepared for output. I followed a basic Applications Hungarian Notation, prefixing references to the input document with “in” and references to the output document with “out”.
Since I needed to be able to execute derivation recursively, I decided to do it by creating a new renderer, passing it the parent template, and running only the loadTemplate and handleDerivation methods; then the parent renderer’s outDoc became the child’s starting outDoc. In this way, if the parent derived from another template, the nested derivation would happen automagically. I would then scan the parent document for ID’s that matched elements in the child document, and replace them accordingly. Derivation was done.
Next up was injection: I started out by iterating over the keys in my model Map, scanning the document for matching class names. Where I found them, I simply replaced the innerHtml of the found element with the toString() value of the model data; if the model data was an array or collection, I would instead iterate the data, duplicating the matched element for each value, and replacing the cloned element’s innerHtml with the list item’s toString() value. This was enough functionality for my simple test case.
Reaching the home stretch, I did substitution ridiculously simply, using a very basic regex to find substitution placeholders (${somevariable}) and replacing each with the appropriate value from the model. I knew this solution wouldn’t last, but it was enough for this early prototype.
Last up was writing the rendered output, and in this case, I allowed passing in an HttpServletResponse to write to. I would set the content type of the response, and dump the HTML of my final Document to the response stream.
I ran it, and somehow, it actually worked. I was shocked, but excited: in the course of a little over an hour, I had a fully working prototype of the most basic functions of my template engine. Not a complete or usable product by any means, but an excellent sign. I made a few tweaks here and there, correcting some minor issues (collection items were being inserted in reverse order, for example), but it was pretty much solid. I also replaced my RegEx-based substitution mechanism with the StrSubstitutor from commons-lang; this was pretty much a direct swap that worked perfectly.

Time for the next test, my moderate complexity test case.