Simplicity, Flexibility, and Agility

Agile programming is supposed to be about flexibility in the face of changing requirements. It’s supposed to be about rapid development and iteration. But all too often it ends up being like classical methodologies in many ways. Many agile methodologies drown developers in process, taking time away from development. Test-driven development is a brilliant concept, but it puts more time between planning and iteration, making it more difficult to deal with changing requirements, not easier, and increasing the burden of change and the cost of refactoring.


Every developer wants carefully, precisely defined requirements. Developers often try to handle changing requirements by developing for flexibility, but flexibility often comes at the cost of added complexity. Trying to write-in endless flexibility to allow for changing requirements is very much akin to premature optimization. You end up doing a whole lot of work to make some code “better” – more flexible in this case, versus more performant in the case of optimization – when you don’t yet know which code really needs it and which code doesn’t.

“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.” –C. A. R. Hoare 

Often the best way to maintain flexibility is through simplicity. A program that meets its requirements with the simplest possible implementation is one that will be naturally flexible, maintainable, stable, and manageable.Of course, intelligent development plays a major role; making appropriate use of design patterns can do a lot to improve both flexibility and simplicity. Key design tenets like DRY, YAGNI, separation of concerns, and avoiding premature optimization (including overarchitecting for flexibility) help keep complexity down and productivity up.

What if requirements change? What if the simplest possible solutions isn’t as extensible? Good news: having invested in the simplest possible solution, you’ve lost little in development. You haven’t built anything that wasn’t strictly necessary. The new solution will still aim for simplicity, still reap the same rewards. By keeping things simple, you’ve kept down the cost of change.

Software development is a learning process by nature; you’re building something that’s never been done before, or at least building something in a way that’s never been done before. Innovation is at its heart, and learning is the personal experience of innovation. That being the case, every iteration has value in what’s learned, even if the code is later removed or replaced. The experience of writing it adds value to the team, and the team defines the value of the end product.


Many readers may think of targeting simplicity as a given, but it truly isn’t; while simplicity is often a goal, all too frequently it takes a back seat to other concerns, or is abandoned altogether because simplicity becomes far more difficult to achieve with many of the popular frameworks and libraries available today. Frameworks have to aim for maximum flexibility in order to be successful; in order to be general-purpose, they can’t make many assumptions, and they have to account for a whole host of different usage scenarios and edge cases. This increases the complexity of the framework, and accordingly, the complexity of any implementation using the framework. The fewer assumptions the framework can make, the more effort a developer has to put in just telling the framework what she’s trying to accomplish.


I can’t count how many implementations I’ve seen that are drastically more complex than necessary, simply because they have been forced to apply the conventions required by their chosen framework; and even following those conventions, they’re still left managing arcane configuration files, and tracking down bugs becomes an epic undertaking requiring delving deep into the inner workings of the framework and libraries. All too often, the quick-start “hello world” app is far more complex with a framework than without it, and adding functionality only makes the situation more bleak.


So, what does all this add up to? Here’s the bullet-point version:

  • If you’re aiming for agility – the ability to adapt quickly to changing requirements – don’t invest too much time on nailing down requirements. Get as much detail as you can, and start iterating. If you’re planning for requirements to change, go all the way – assume that your initial requirements are wrong, and think of each iteration as an opportunity to refine the requirements. All code is wrong until proven right by user acceptance.
  • Use interface mockups (for software with a UI) or API documentation (for libraries and services) as a tool to give stakeholders a chance to revise requirements while looking at a proposed solution and thinking about using it in real-world scenarios.
  • Don’t choose flexibility or modularity over simplicity. Choose a framework that won’t get in your way; if there isn’t one, then don’t use a framework at all. Don’t write what you don’t need. Don’t turn a piece of code into a general-purpose API just because you might need to use it again. If you need it in multiple places now, separate it out; otherwise, you can refactor it when it’s appropriate.
  • Think about separation of concerns early in the game, but don’t sacrifice simplicity for the sake of compartmentalization. Simple code is easier to refactor later if refactoring turns out to be necessary. Overarchitecting is the same sin as premature optimization, it’s just wearing a nicer suit.
  • The simplest solution isn’t always the easiest. The simplest solution often requires a lot of thought and little code. Don’t be a code mason, laying layer after layer of brick after brick; be a code poet, making every line count. If a change could be implemented by increasing the complexity of existing code, or by refactoring the existing code to maintain simplicity, always take the latter route; you’ll end up spending the same amount of time either way, but will reap far more benefits by maintaining simplicity as a priority.
  • Simplicity carries a great many implicit benefits. Simpler code is very often faster (and easier to optimize), more stable (and easier to debug), cleaner (and easier to refactor), and clearer to read and comprehend. This reduces development, operational, and support costs across the board.
  • Simplicity doesn’t just mean “less code”, it means better code. SLOC counts don’t correlate directly to complexity.
  • Don’t reinvent the wheel – unless you need to. All wheels aren’t created equal; there’s a reason cars don’t use the same wheels as bicycles.
What are your experiences? What ways have you found to keep hold of simplicity in the face of other pressures? Feedback is welcome in the comments!

On New Tricks, Old Hacks, and Web Browsers

I must say, I’m a little curious why I haven’t seen mention of this before; a quick Google search didn’t turn anything up either. For the last, oh, ten years or so, web designers have been wrestling with all the different browsers, and different versions of each browser, to get their web pages to behave the same – or at the very least, behave relatively well – on all the browsers their users are likely to employ.

The whole time, the W3C has been releasing new standards and new versions of old standards to give web designers new tricks… and every time, the browsers all catch up to the new standards at different speeds, and implement different parts of the standards, or implement them slightly differently.
My question, then, is this: why is there no W3C specification for browser detection? Why can’t I use CSS selectors to target certain styles at certain browsers, without resorting to lousy hacks? Even CSS3’s new media queries allow me to check the screen size before applying styles, but not whether or not the browser supports, say, CSS3 Of course, it’ll take forever for designers to be able to count on all the browsers supporting a new feature like that, but I haven’t even seen a proposal.
Today, putting together a design involves pulling up your design in all the browsers, figuring out what works and what doesn’t, and then applying hacks specific to each browser. Life would be so much easier in the web design world if instead you could say something like, “if the browser doesn’t support CSS3 background properties, apply this style instead.”
Suddenly, I don’t need to use the hack that hides CSS from IE, and the other hack that hides CSS from everything but IE, and test it, and then find another set of hacks for the Android browser, and another for FireFox, and so on. I can apply styles logically by selecting for specific features, rather than selecting for specific browsers, then having to keep up with the features of each browser – because the features are all I really care about as a designer.
I would much rather “hack” for specific features than specific browsers because it’s more intuitive, and it’s less work to support multiple browsers and different versions of each browser. The browser makers know what features they support. If I can select for the features I want to use, I don’t have to worry about keeping up-to-date with what features are supported by what versions of what browsers.
I’m bringing it up on the W3C mailing list, but I thought I would bring it up here… I’d love to hear your thoughts in the comments!

NetBeans + Tomcat + Java for MacOS X 10.5 Update 4 = Fail

Just struggled with this for a couple of days. After updating MacOS X with the latest Java update (Java for MacOS X 10.5 Update 4, patching security holes mentioned in the news the last few weeks), I could no longer launch Tomcat from NetBeans. And that’s a Bad Thing. I use NetBeans all day for work, on a web application that gets tested through the local copy of NetBeans.

The error I was getting was that it couldn’t find a file/directory under “/var/folders/ZC/ZCcmX61vGaqOjbHwwgwW-k+++TI/-Tmp-/”. The filename changes every time, in the format “context1234567890123456789.xml”. The “+++” throws it off because the path is passed as a URL parameter to the Tomcat Manager, which converts the +’s to spaces.
The solution is to go to your NetBeans application folder, right-click the application, and choose Show Package Contents. Under Contents/Resources/NetBeans/etc, open netbeans.conf, and find the line for “netbeans_default_options”. Go to the end of this line, and add, inside the end quote, “-J-Djava.io.tmpdir=/tmp”. Restart NetBeans, and you should be good to go.

Stackoverflow.com

Today marks the public launch of Stackoverflow.com, the new programming Q&A site started by the famous Joel Spolsky. So far it looks like it’s shaping up to be a very good resource for developers; and I agree with everything they said about Googling for solutions in this post. You can look me up on the site.

Beyond the site itself, which I think will become an oft-used resource for me, I also really like their badge system. I think a similar system could be a huge boon to any forum, social networking site, or other community site. Game developers learned years ago that people will be more active in an area they already enjoy if they can be recognized for their activity. The same idea applies just as well to community sites. I think it’s brilliant, and I think we’re going to be seeing similar implementations a lot more often.

XML: No, it isn’t.

XML, well, just isn’t. It’s a raging misnomer. XML is, in theory, the eXtensible Markup Language. I have a couple of problems with that idea.

First, it’s not extensible. It just isn’t. You can’t extend it. I can’t extend it. No one can extend it. You know how I know? Because there aren’t any extensions. Not a single one. Go ahead. Go find an extension to XML. I’d love to hear about it.

But it’s just as well – there’s no reason to extend it. XML defines very little; it’s a syntax definition, nothing more. DTDs and Schemas are what make XML useful. They aren’t extensions to XML, they’re applications of XML. What’s more, the DTDs and Schemas can be combined in a single document, but even they can’t be extended.

Second, while it can be used as a markup language, it almost never is. XHTML is a markup language based on XML. There are a few others that are (debatably) markup languages, like DocBook, but even the likes of DocBook are more on the side of data structure definition than markup. A database file isn’t markup. A Java properties file isn’t markup. It’s a data structure. Per Wikipedia:

A markup language is an artificial language using a set of annotations to text that describe how text is to be structured, laid out, or formatted.

Does that sound like most of the XML formats you’ve encountered? How many XML config files have you had to deal with? Do they fit that description, even a little bit? Of course not. You don’t care about the structure, layout, or formatting of the text in a config file. All you care about is getting at the particular block of text you want. So, what is XML then? Something of a generic hierarchical data file format – though I suppose GHDFF just isn’t as catchy as XML.

Now, besides being aggregiously misnamed, it’s also a wretched tool for nearly every purpose to which it is applied. It’s a language that aims for the middle ground between human-readable and machine-readable, and while it achieves both, it does so very poorly. XML is annoying to read, tedious to write, and resource-intensive to process.

I’m not suggesting dumping XML entirely, not at all. The angle-bracket tax is a fee worth paying for actual markup – you need syntax to seperate the markup from the text. XML is a flexible and effective format for marking up text. What it isn’t is an effective format for storing arbitrary data. It’s usable, but nowhere near optimal. What’s the solution? Something else.

Programmers have a tendency to cling to standards, to try to apply them as much as possible. “Don’t reinvent the wheel,” we say. And that’s a perfectly reasonable mantra – but that doesn’t mean all wheels are created equal. When’s the last time you saw a bicycle wheel on a car? Would the world be a better place if every wheel were the same? Sure, they’d be interchangeable – but they wouldn’t be anywhere near as effective.

We need to step back sometimes, and think about whether there is, or could be, a better wheel for any given situation.

More on this to come.

The Joys of Hibernate

I just spent the last few days troubleshooting a HibernateException with the error message Illegal attempt to associate a collection with two open sessions. After much Googling, I had tried every solution I could find, and none of them worked – so hopefully this solution will make it into the next person’s search.

In my case, the issue was that I was taking a persistent object, storing it in the HTTP session, and trying to reconstitute it later. By storing the ID in the browser session and loading by ID each time, the error was eliminated.

I hope this helps someone – if so, or if you’re having a similar issue, post in the comments!

LAMPP and then some

I recently built Apache, MySQL, PHP, Python, SQLite, OpenSSL, Subversion , and Trac on a Mac, an Ubuntu box, and a RHEL 4 box. Don’t ask why, just see these tips:

  • Try building your own APR. Also, check what APR is being used; if you already have an APR version 0.9.x, the new APR will be named apr-1-config instead of apr-config, and likewise apu-config will be apu-1-config to get the proper version.
  • Try building Apache –with-included-apr.
  • Try using a different version of OpenSSL, even if you have to go back a version. Security holes are typically backported as a letter release to the previous one or two point releases.
  • Under linux, remember to run sudo ldconfig, make clean, make if you’re having trouble.
  • Under MacOS, if you’re building under a prefix, make sure to add the prefix to the environment variable DYLD_LIBRARY_PATH.

More tips, and maybe even a step-by-step, will be forthcoming.

Urgent Apps – Mac Development Kit

After yesterday’s post, I got to thinking. I had pulled some items out of that list because they were highly developer-centric applications. However, that does mean that some really top-notch programs didn’t make the list, and I think that’s unfair. There are some apps that I really can’t live without when it comes to development work.

Before we begin, I should point out that my particular development tasks typically include the administration of a MySQL database, editing PHP, Java, HTML, CSS, and JavaScript files, and operating revision control; the tools laid out here are centered around those tasks.

Now, without further ado, the list:

jEdit

jEdit calls itself “the programmer’s text editor”, but that’s selling it short. jEdit is, to put it lightly, a god among executables. I’ve never seen another program come close to its level of flexibility, modularity, and customizability. The sacrifice for all this goodness is that it’s a bit of a RAM-hog, particularly running under the MacOS JRE (I highly recommend updating to the 1.6 JRE available on apple’s website, and completely switching over to 1.6; it provides some vast performance and footprint improvements.)

NetBeans
NetBeans is, of course, the Java IDE, unless you’re one of those people that thinks that Eclipse is the Java IDE, but I’m not.

SmartSVN (payware)
The best SVN client I’ve seen for the Mac. Unfortunately, there aren’t many good free options.

MySQL GUI Tools
The real deal, straight from the source.

Platypus
This nifty little app lets you take any shell or other script file and turn it into a Mac application package.

ArgoUML
Java-based UML designer.

Subversion
Revision control extraordinaire. Any box I do development on has a local Subversion server for anything I happen to want to keep a history for.

Trac
I’ve recently fallen in love with Trac, which is why it made the list. It’s not exactly an application – it’s a web application. However, it can be installed on a Mac, so it made the cut, and I do love it dearly. Go check out their page; the Trac site runs on Trac.

Anything I’m missing? Post in the comments!!

Development Tactics

I recently set up an account with hosted-projects.com, because I wanted a Subversion repository more accessible & stable than the one running on my home desktop. I shopped around for a while, and decided on this place – it’s a small project, and a starter account is only $7/month, so I figure, what the heck.

My account was set up within a few minutes, even though I ordered after business hours – I’m guessing they’ve got a pretty good automation system going. I get fast, secure access for unlimited users to unlimited projects in 100M of space, plus a free Trac – not a bad deal. As far as reliability and support, well – only time will tell.

The host is all well and good, but what I really wanted to talk about is Trac. I had looked Trac up some time ago, and decided to take a pass on it – it just wasn’t mature enough at the time, and didn’t have most of the features I was looking for.

Now, however – after some time, and a few bug tracking schemes – I find myself with a free Trac page sitting around, and I figure, what the hey, I’ll give it a shot. And you know what? It still doesn’t have some of the features I was looking for. But it works so well, it doesn’t matter.

The whole thing runs on a Wiki engine. This Wiki engine identifies all CamelCase as wiki links, which I find a bit annoying, but I got used to it pretty quickly. It lets you easily link to pretty much anything, and inline, too: #123 is ticket 123, r456 is revision 456, etc. It hooks up to your Subversion repo and lets you keep an eye on changelogs and browse the repo; plus, this means if you put properly formatted notes in your commit messages (which isn’t hard), you get links in the changelog, for free.

While not quite as versatile as MediaWiki, for example, in terms of page layout and design, it’s probably easier to use – and programmers tend to go for form over function anyway. It’s a developer’s tool. Developers probably won’t spend all day perfecting page templates and macros.

The system provides for a roadmap of milestones, a list of issue tickets, the wiki, and the repository. That’s it. What’s the big deal? How insanely easy it is to wire them all together. With some really basic formatting, you can turn a simple list of milestones into this.

It’s got some rough edges, and there are definitely some huge opportunities yet to be taken advantage of – particularly, I have yet to discover decent, proper JavaDoc support, with full wiki integration. I may just have to learn enough Python to write a plugin for it. I’d also really like to see automatic backlinks added to all the internal links.

I know it’s still version “0.10.3”, but it’s pretty stable so far, and everything works pretty well. I have yet to run into any bugs or bad behavior – however, you should keep in mind that this is bleeding-edge software if you’re considering deploying it. Don’t let that scare you off though: if you don’t mind the under-heavy-development label, you really should give this little application a try and see what you think. At the very least, check out Trac’s own website to see what it can do.

Sun Opens Java

Ars Technica is reporting that Sun has finally made good on their promise to open up Java for the masses. This is good news for everyone – in fact, the only one whose benefit stands in question would be Sun. While I’m confidant that they will turn this to their advantage, it certainly assuages fears that trouble for Sun might mean trouble for Java – realistic fears when Sun’s financials look shakier and shakier with each passing quarter. Now, if Sun goes down, they won’t take Java with them.

What’s prevented Java being open-sourced before (according to Sun’s PR department) wasn’t an issue of profitability for Java products – after all, the JVM/JRE and JDK are free, and all of their paid products are not being open sourced – it was an issue of branching. Sun was afraid that if they opened up the Java source, we’d see forks which would eventually diverge, bringing about compatibility issues; it’s bad enough having to make sure that a user has the JVM/JRE installed and that they have an adequately recent version, without having to worry about which JVM they’re using out of an array of options which may not all support the same features.

Personally, I think that’s a rather silly fear – they need to revamp Java’s dependency handling anyway, so why not take the opportunity to do so now? Instead of an application saying “I need this version of the JDK or newer”, say “I need to use this library, and this one, and this one” and make sure those are available. This resolves both the issue of determining what your minimum JRE requirement need be, and the issue of diverging forks supporting different featuresets.

All in all, as a Java developer, I can’t see this as anything but a Good Thing(tm).