What is “Technical Debt”?

I’ve seen many different definitions of technical debt (as it applies to software), but most of them are very specific and indeed often tied to some kind of technical debt metrics dashboard that vendor X is selling.

So when a co-worker asked me to define it a few days ago, I thought I’d better come up with something a bit more useful.

For me, “technical debt” is the long-term price companies pay for adding functionality and features hackily, i.e. without really thinking about how those changes impact architecture, documentation, usability, clarity, maintainability, etc. When the things that come back to bite you actually do bite you, you’re paying interest on your technical debt (i.e. you only pay it off completely if you can get rid of it).

Hence to assess your company’s current level of technical debt, you need to assess what proportion of work time a typical engineer spends actually working productively, as compared to performing other ancillary work-time activities that relate to dancing around the accumulated mountain of hacks and increments that it (perhaps laughably) calls its codebase.

Such unproductive activities include:-
* Finding, reading and understanding informal documentation (often left abandoned on internal Wikis)
* Working out how to add new workarounds to work around the current set of abandoned workarounds
* Spending time in meetings with other engineers trying to convince them that changing old code is a good idea
* Fixing regression failures that just happen to reveal old bugs that had previously been untested by unit tests
* Following arcane coding standards that have not been updated to reflect current tools and practices
* Reinventing software wheels because of policies that prohibit code reuse outside formal APIs
* Using log files to debug full stack builds (because they have become too complicated to use gdb etc)
* And so on.

The problem is that as a company’s technical debt reaches 80% or so, the company becomes largely paralyzed: while having a technical debt of 90% is close to terminal, with each engineer having on average only 6 minutes out of each hour being applied productively. Few companies can sustain this level of burden for long without collapsing.

What is not widely understood is that this isn’t just a feature of large companies with old applications. With their explicit reliance on incrementalism, Agile companies too can exhibit all of these problems if they have no engineers with architectural flair or no time explicitly put aside for refactoring. (The “Lean Startup” is no different.)

All in all, high technical debt can be a crushingly huge problem – how high is your company’s level of technical debt? Can you honestly say that it’s less than 50%?

Advertisement

What does “Agile” mean?

To most people, “Agile” is – like “Lean”, “Cloud”, “SCRUM”, and “Virtual” – one of those modern software development buzzwords that sounds vaguely scientific and valuable, while probably meaning almost nothing.

Yet a vast consulting and training industry has sprung up on the back of Agile. People take courses in how to be Scrum Masters, how to Unblock, to build self-managing teams and to design Epics. They ingest the Seventeen (the precise number varies) Golden Rules Of Effective Stand-up Meetings, and Groom and Iterate cheerfully through their Backlogs to increase their Story Point Velocity.

(Don’t worry if the above sounds to you like an explosion in a buzzword factory, because that’s basically what it is.)

Moreover, software companies that commit their workforces to Agile (typically by getting them to stand on the edge of a cliff overlooking the Bay of Agility, and ordering them to all jump off at the same time) seem to get praised by allegedly smart investors, while non-Agile companies get slated as old-fashioned, waterfall-obsessed fossils.

Is this whole [ Agile vs Dinosaur ] dichotomy fair? Or accurate? Or helpful?

But to me, the most-ducked question of all is this: what, in any useful sense, does “Agile” actually mean? Few Agile proponents seem happy to answer this without lapsing into the whole avalanche of secondary buzzwords typical of the second paragraph.

Even though it would be easy to dismiss Agile as no more than a way of collecting buzzwords like baseball cards, it turns out that there is a core of useful truth at its heart. In fact, you may be surprised to learn that this revolves about what it means to be a professional modern software developer.

You see, during the 20th Century, programming grew up in smart-guy (yes, usually male) ghettos within large organizations: it was an anti-social, isolated, unintegrated pastime sold as a value-adding professional intellectual skill.

On top of that, a long-standing structural problem within the whole industry has been that a traditional software developer isn’t someone you would like to show to your sister, let alone to a customer. And for a very long time, many programmers have played up to this self-indulgent stereotype: the beards, the sandals, the obscure terminology and bad acronymic puns when naming things. Yecccch. (Or do I mean ‘YACC’?)

At their most programmery, these sad creatures liked their ghettos, and fought long and hard to keep their comfy development world just that way. They far preferred to remain aloof and remote from sales, marketing, finance, HR… in fact, to stay apart from just about everything else (even from hardware, truth be told). Hence software’s guilty little secret is that it has been a profession only lightly integrated within broader organizations.

But business has changed – really changed. Modern software professionals now need to be able to present their work, and to negotiate (and iterate) requirements with customers in a more parallel, self-determining way. Software design is becoming just as much a social medium as an exercise in system optimization. As a result, the old, stay-out-of-the-loop approach to building software just doesn’t cut it any more.

So here’s the central paradox (or, indeed, the future challenge): though programming is an industry still socially dominated by code ghettos, modern business demands that the people in it need “social smarts” just as much as technical smarts, even though they only typically get trained in the latter. And their day-to-day tools of choice are usually technical, too.

But… where does a modern software professional go to get trained in social smarts? The answer is: they don’t. But the nearest they currently get to this is a kind of pragmatic social Boot Camp – by working within an Agile team. Agile practices, though dogmatic and an awkward fit for many tasks, often force programmers to confront and surmount their social limitations: which is usually a good thing.

As a result, what “Agile” means (particularly on someone’s CV) is that they have been ‘blooded’ into the difficult arena of modern software development – that they have (or should have) at least some of the combination of both technical skills and social skills now needed to call themselves a ‘proper programmer’.

What, then, is going on with this whole ‘Agile Movement’? Personally, I believe that what a company is doing by embracing Agile is forcing its workforce to enter into a transition from antisocial 20th century software development into far more social 21st century software development. That is, to move from the old world of serial, top-down, micro-managed, manager-led development to the new world of parallel, bottom-up, team-managed, customer-led development.

(Though contrary to what many claim, this doesn’t mean that the code produced by Agile organizations is any better. In many ways, the incrementalism necessitated by Agile goes against proper software engineering and proper software architecture, which in practice can often yield weaknesses and fragilities as profound as those engendered by traditional software development practice.)

Really, to make a good practical contribution to the majority of the projects I see happening these days, you need to have the skills both of traditional software engineering and of contemporary Agile practices. (It’s not an either-or choice, you almost always need the two simultaneously.)

Would I hire someone with Agile on their resume? Why, yes I would. But I’d always interview them and ask myself: regardless of how well they can programme, would I be comfortable with putting them in front of customers? Because that, increasingly, is where they are going to be over the coming decades, whether you like it or not.

Finally… in time, it seems almost certain that Agile’s constellation of idiosyncratic terminology will be lambasted as a domain-specific ‘Buzzword Bingo’ of only historic value. But I would argue that this will only be true because its core values of what it means to be a modern software professional will have been thoroughly absorbed into the development mainstream. Chillingly, we’ll all be Agile then. Who’d have thought it, eh?