Real Sprints

Agile methodologies talk about “sprints” – workloads organized into one to four week blocks. You schedule tasks for each sprint, you endeavour to complete all of it by the end of the sprint, then you look back and see how close your expectations (schedule) were to reality (what actually got done).

Wait, wait, back up. When I think of a sprint, I think short and fast. That’s what sprinting means. You can’t sprint for a month straight; you’ll die. That’s a marathon, not a sprint.

There are numerous coding competitions out there. Generally, you get around 48 hours, give or take, to build an entire, working, functional game or application. Think about that. You get two days to build a complete piece of software from scratch. Now that’s what I call sprinting.

Of course, a 48 hour push is a lot to ask for on a regular basis; sure, your application isn’t in a competition, this is the real world, and you need to get real work done on an ongoing basis. You can’t expect your developers to camp out in sleeping bags under their desks. But that doesn’t mean turning a sprint into a marathon.

The key is instilling urgency, while moderating burnout. This is entirely achievable, and can even make development more fun and engaging for the whole team.Since the term sprint has already been thoroughly corrupted, I’ll use the term “dash”. Consider this weekly schedule:

  • Monday: Demo last week’s accomplishments for stakeholders, and plan this week’s dash. This is a good week to schedule any unavoidable meetings.
  • Tuesday and Wednesday: your 48 hours to get it done and working. These are crunch days, and they will probably be pretty exhausting. These don’t need to be 18-hour days, but 10 hours wouldn’t be unreasonable. Let people get in the zone and stay there as long as they can.
  • Thursday: Refactoring and peer reviews. After a run, athletes don’t just take a seat and rest; they slow to a jog, then a walk. They stretch. The cool off slowly. Developers, as mental athletes, should do the same.
  • Friday: Testing. QA goes through the application with a fine-toothed comb. The developers are browsing the web, playing games, reading books, propping their feet up, and generally being lazy bums, with one exception: they’re available at a moment’s notice if a QA has any questions or finds any issues. Friday is a good day for your development book club to meet.
  • By the end of the week, your application should be ready again for Monday’s demo, and by Tuesday, everyone should be well-rested and ready for the next dash.
Ouch. That’s a tough sell. The developers are only going to spend two days a week implementing features? And one basically slacking off? Balderdash! Poppycock!

Think about it, though. Developers aren’t factory workers; they can’t churn out X lines of code per hour, 40 hours per week. That’s not how it works. A really talented developer might achieve 5 or 6 truly productive hours per day, but at that rate, they’ll rapidly burn out. 4 hours a day might be sustainable for longer. Now, mind you, in those four hours a day, they’ll get more done, better, with fewer defects, than an army of incompetent developers could do in a whole week. But the point stands: you can’t run your brain at maximum capacity eight hours straight, five days a week. You just can’t – not for long, anyway.

The solution is to plan to push yourself, and to plan to relax, and to keep the cycle going to maximize the effectiveness of those productive hours. It’s also crucial not to discount refactoring as not being productive; it sets up the following weeks’ work, and reduces the effort required to get the rest of the development done for the rest of the life of the application. It’s a critical investment in the future.

Spending a third of your development time on refactoring may seem excessive, and if it were that simple, I’d agree. But if you really push yourself for two days, you can get a lot done – and write a lot of code to be reviewed and refactored. In one day of refactoring, you can learn a lot, get important work done, and still start to cool off from the big dash.

That lazy Friday really lets you relax, improve your craft, and get your product ready for next week, when you get to do it all over again.

The Development Stream

I was reading today about GitHub’s use of chat bots to handle releases and continuous integration, and I think this is absolutely brilliant. In fact, it occurs to me that using a chat bot, or a set of chat bots, can provide an extremely effective workflow for any continuous-deployment project. Of course, it doesn’t necessarily have to be a chat room with chat bots; it can be any sort of stream that can be updated in real-time – it could be a Twitter feed, or a web page, or anything. The sort of setup I envision would work something like this:

Everyone on the engineering team – developers, testers, managers, the whole lot – stay signed in to the stream as long as they’re “on duty”. Every time code is committed to a global branch – that is, a general-use preproduction or production branch – it shows up in the stream. Then the automated integration tests run, and the results are output to the stream. The commit is deployed to the appropriate environment, and the deployment status is output to the stream. Any issues that occur after deployment are output to the stream as well, for immediate investigation; this includes logged errors, crashes, alerts, assertion failures, and so on. Any time a QA opens a defect against a branch, the ticket summary is output to the stream. The stream history (if it’s not already compiled from some set of persistent-storage sources) should be logged and archived for a period of time, maybe 7 to 30 days.

It’s very important that the stream be as sparse as possible: no full stack traces with error messages, no full commit messages, just enough information to keep developers informed of what they will need to look into further elsewhere. This sort of live, real-time information stream is crucial in the success of any continuous-deployment environment, in order to keep the whole team abreast of any issues that might be introduced into production, along with when and how they were introduced.

Now, what I’ve described is a read-only stream: you can’t do anything with it. GitHub’s system of using an IRC bot allows them to issue commands to the bot to trigger deployments and the like. That could be part of the stream, or it could be part of another tool; as long as the deployment, and its results, are output to the shared stream for all to see. This is part of having the operational awareness necessary to quickly identify and fix issues, and to maintain maximum uptime.

There are a lot of possible solutions for this sort of thing; Campfire looks particularly promising because of its integration with other tools for aggregating instrumentation data. If you have experience with this sort of setup, please post in the comments, I’d love to hear about it!

Truly Agile Software Development

Truly agile software development has to, by nature, allow for experimentation. In order to quickly assess the best option among a number of choices, the most effective method is empirical evidence: build a proof of concept for each option and use the experience of creating the proof, as well as the results, to determine which option is the best for the given situation.

While unit tests are valuable for regression testing, a test harness that supports progression testing is at least as useful. Agile development methodologies tend to focus on the idea of iterating continuously toward a goal along a known path; but what happens when there’s a fork in the road? Is it up to the architect to choose a path? There’s no reason to do so when you can take both roads and decide afterward which you prefer.

Any large development project should always start with a proof of concept: a bare-bones, quick-and-dirty working implementation of the key functionality using the proposed backing technologies. It doesn’t need to be pretty, or scaleable, or extensible, or even maintainable. It just has to work.

Write it, demo it, document what you’ve learned, and then throw the code away. Then you can write the real thing.

It may seem like a waste of time and effort at first.  You’ll be tempted to over-engineer, you’ll be tempted to refactor, you’ll be tempted to keep some or all of the code. Resist the urge.

Why would you do such a thing? If you’re practicing agile development, you might think your regular development is fast enough that you don’t need a proof. But that’s not the point; the point is to learn as much as you can about what you’re proposing to do before you go all-in and build an architecture that doesn’t fit and that will be a pain to refactor later.

Even if it takes you longer to build the proof,it’s still worth it – for one thing, it probably took longer because of the learning curve and mistakes made along the way that can be avoided in the final version, and second because again, you’ve learned what you really need and how the architecture should work so that when you make the production version you can do it right the first time, with greater awareness of the situation.

This approach allows much greater confidence in the solutions chosen, requiring less abstraction to be built in to the application, which allows for leaner, cleaner code, and in less time. Add to that the value of building a framework that is flexible enough to allow for progression testing, and you’ve got the kind of flexibility that Agile is really all about.

Note: Yes, I understand that Scrum calls prototypes “spikes”. I think this is rather silly – there are already terms for prototypes, namely, “prototype” or “proof of concept”. I’m all for new terms for things that don’t have names, but giving new names to things that already have well-known names just seems unnecessary.

Simplicity, Flexibility, and Agility

Agile programming is supposed to be about flexibility in the face of changing requirements. It’s supposed to be about rapid development and iteration. But all too often it ends up being like classical methodologies in many ways. Many agile methodologies drown developers in process, taking time away from development. Test-driven development is a brilliant concept, but it puts more time between planning and iteration, making it more difficult to deal with changing requirements, not easier, and increasing the burden of change and the cost of refactoring.


Every developer wants carefully, precisely defined requirements. Developers often try to handle changing requirements by developing for flexibility, but flexibility often comes at the cost of added complexity. Trying to write-in endless flexibility to allow for changing requirements is very much akin to premature optimization. You end up doing a whole lot of work to make some code “better” – more flexible in this case, versus more performant in the case of optimization – when you don’t yet know which code really needs it and which code doesn’t.

“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.” –C. A. R. Hoare 

Often the best way to maintain flexibility is through simplicity. A program that meets its requirements with the simplest possible implementation is one that will be naturally flexible, maintainable, stable, and manageable.Of course, intelligent development plays a major role; making appropriate use of design patterns can do a lot to improve both flexibility and simplicity. Key design tenets like DRY, YAGNI, separation of concerns, and avoiding premature optimization (including overarchitecting for flexibility) help keep complexity down and productivity up.

What if requirements change? What if the simplest possible solutions isn’t as extensible? Good news: having invested in the simplest possible solution, you’ve lost little in development. You haven’t built anything that wasn’t strictly necessary. The new solution will still aim for simplicity, still reap the same rewards. By keeping things simple, you’ve kept down the cost of change.

Software development is a learning process by nature; you’re building something that’s never been done before, or at least building something in a way that’s never been done before. Innovation is at its heart, and learning is the personal experience of innovation. That being the case, every iteration has value in what’s learned, even if the code is later removed or replaced. The experience of writing it adds value to the team, and the team defines the value of the end product.


Many readers may think of targeting simplicity as a given, but it truly isn’t; while simplicity is often a goal, all too frequently it takes a back seat to other concerns, or is abandoned altogether because simplicity becomes far more difficult to achieve with many of the popular frameworks and libraries available today. Frameworks have to aim for maximum flexibility in order to be successful; in order to be general-purpose, they can’t make many assumptions, and they have to account for a whole host of different usage scenarios and edge cases. This increases the complexity of the framework, and accordingly, the complexity of any implementation using the framework. The fewer assumptions the framework can make, the more effort a developer has to put in just telling the framework what she’s trying to accomplish.


I can’t count how many implementations I’ve seen that are drastically more complex than necessary, simply because they have been forced to apply the conventions required by their chosen framework; and even following those conventions, they’re still left managing arcane configuration files, and tracking down bugs becomes an epic undertaking requiring delving deep into the inner workings of the framework and libraries. All too often, the quick-start “hello world” app is far more complex with a framework than without it, and adding functionality only makes the situation more bleak.


So, what does all this add up to? Here’s the bullet-point version:

  • If you’re aiming for agility – the ability to adapt quickly to changing requirements – don’t invest too much time on nailing down requirements. Get as much detail as you can, and start iterating. If you’re planning for requirements to change, go all the way – assume that your initial requirements are wrong, and think of each iteration as an opportunity to refine the requirements. All code is wrong until proven right by user acceptance.
  • Use interface mockups (for software with a UI) or API documentation (for libraries and services) as a tool to give stakeholders a chance to revise requirements while looking at a proposed solution and thinking about using it in real-world scenarios.
  • Don’t choose flexibility or modularity over simplicity. Choose a framework that won’t get in your way; if there isn’t one, then don’t use a framework at all. Don’t write what you don’t need. Don’t turn a piece of code into a general-purpose API just because you might need to use it again. If you need it in multiple places now, separate it out; otherwise, you can refactor it when it’s appropriate.
  • Think about separation of concerns early in the game, but don’t sacrifice simplicity for the sake of compartmentalization. Simple code is easier to refactor later if refactoring turns out to be necessary. Overarchitecting is the same sin as premature optimization, it’s just wearing a nicer suit.
  • The simplest solution isn’t always the easiest. The simplest solution often requires a lot of thought and little code. Don’t be a code mason, laying layer after layer of brick after brick; be a code poet, making every line count. If a change could be implemented by increasing the complexity of existing code, or by refactoring the existing code to maintain simplicity, always take the latter route; you’ll end up spending the same amount of time either way, but will reap far more benefits by maintaining simplicity as a priority.
  • Simplicity carries a great many implicit benefits. Simpler code is very often faster (and easier to optimize), more stable (and easier to debug), cleaner (and easier to refactor), and clearer to read and comprehend. This reduces development, operational, and support costs across the board.
  • Simplicity doesn’t just mean “less code”, it means better code. SLOC counts don’t correlate directly to complexity.
  • Don’t reinvent the wheel – unless you need to. All wheels aren’t created equal; there’s a reason cars don’t use the same wheels as bicycles.
What are your experiences? What ways have you found to keep hold of simplicity in the face of other pressures? Feedback is welcome in the comments!

Urgent Apps – Mac Development Kit

After yesterday’s post, I got to thinking. I had pulled some items out of that list because they were highly developer-centric applications. However, that does mean that some really top-notch programs didn’t make the list, and I think that’s unfair. There are some apps that I really can’t live without when it comes to development work.

Before we begin, I should point out that my particular development tasks typically include the administration of a MySQL database, editing PHP, Java, HTML, CSS, and JavaScript files, and operating revision control; the tools laid out here are centered around those tasks.

Now, without further ado, the list:

jEdit

jEdit calls itself “the programmer’s text editor”, but that’s selling it short. jEdit is, to put it lightly, a god among executables. I’ve never seen another program come close to its level of flexibility, modularity, and customizability. The sacrifice for all this goodness is that it’s a bit of a RAM-hog, particularly running under the MacOS JRE (I highly recommend updating to the 1.6 JRE available on apple’s website, and completely switching over to 1.6; it provides some vast performance and footprint improvements.)

NetBeans
NetBeans is, of course, the Java IDE, unless you’re one of those people that thinks that Eclipse is the Java IDE, but I’m not.

SmartSVN (payware)
The best SVN client I’ve seen for the Mac. Unfortunately, there aren’t many good free options.

MySQL GUI Tools
The real deal, straight from the source.

Platypus
This nifty little app lets you take any shell or other script file and turn it into a Mac application package.

ArgoUML
Java-based UML designer.

Subversion
Revision control extraordinaire. Any box I do development on has a local Subversion server for anything I happen to want to keep a history for.

Trac
I’ve recently fallen in love with Trac, which is why it made the list. It’s not exactly an application – it’s a web application. However, it can be installed on a Mac, so it made the cut, and I do love it dearly. Go check out their page; the Trac site runs on Trac.

Anything I’m missing? Post in the comments!!

Development Tactics

I recently set up an account with hosted-projects.com, because I wanted a Subversion repository more accessible & stable than the one running on my home desktop. I shopped around for a while, and decided on this place – it’s a small project, and a starter account is only $7/month, so I figure, what the heck.

My account was set up within a few minutes, even though I ordered after business hours – I’m guessing they’ve got a pretty good automation system going. I get fast, secure access for unlimited users to unlimited projects in 100M of space, plus a free Trac – not a bad deal. As far as reliability and support, well – only time will tell.

The host is all well and good, but what I really wanted to talk about is Trac. I had looked Trac up some time ago, and decided to take a pass on it – it just wasn’t mature enough at the time, and didn’t have most of the features I was looking for.

Now, however – after some time, and a few bug tracking schemes – I find myself with a free Trac page sitting around, and I figure, what the hey, I’ll give it a shot. And you know what? It still doesn’t have some of the features I was looking for. But it works so well, it doesn’t matter.

The whole thing runs on a Wiki engine. This Wiki engine identifies all CamelCase as wiki links, which I find a bit annoying, but I got used to it pretty quickly. It lets you easily link to pretty much anything, and inline, too: #123 is ticket 123, r456 is revision 456, etc. It hooks up to your Subversion repo and lets you keep an eye on changelogs and browse the repo; plus, this means if you put properly formatted notes in your commit messages (which isn’t hard), you get links in the changelog, for free.

While not quite as versatile as MediaWiki, for example, in terms of page layout and design, it’s probably easier to use – and programmers tend to go for form over function anyway. It’s a developer’s tool. Developers probably won’t spend all day perfecting page templates and macros.

The system provides for a roadmap of milestones, a list of issue tickets, the wiki, and the repository. That’s it. What’s the big deal? How insanely easy it is to wire them all together. With some really basic formatting, you can turn a simple list of milestones into this.

It’s got some rough edges, and there are definitely some huge opportunities yet to be taken advantage of – particularly, I have yet to discover decent, proper JavaDoc support, with full wiki integration. I may just have to learn enough Python to write a plugin for it. I’d also really like to see automatic backlinks added to all the internal links.

I know it’s still version “0.10.3”, but it’s pretty stable so far, and everything works pretty well. I have yet to run into any bugs or bad behavior – however, you should keep in mind that this is bleeding-edge software if you’re considering deploying it. Don’t let that scare you off though: if you don’t mind the under-heavy-development label, you really should give this little application a try and see what you think. At the very least, check out Trac’s own website to see what it can do.