Why is So Much Software So Bad?

When I went back to Gartner in 2022 I joined the team creating the Custom Software Development Magic Quadrant. Over the course of the project I sat through presentations by the world’s leading 20 software development shops and examined the case studies of work they had done for their Fortune 500 clients. The official outcome of the project can be seen here, https://www.gartner.com/en/documents/4021203 (if you’re a Gartner client). My unofficial sense at the end of the process was however, that the vast majority of the work on display was completely underwhelming and literally nothing to write home about, let alone parade in front of professionally cynical Gartner analysts.

Since I’ve left Gartner, I’ve spent a lot of time pondering this observation. Was I perhaps over indexing on professional cynicism? Were my expectations too elevated? Was the work on display simply a poor harvest in a problematic part of the post Covid business cycle?

All of those thoughts probably deserve some merit, but at root, I think something else, something more fundamental, is more materially in play.

Most of the work I saw was just not very good. Not new, not innovative, not original, not beautiful, not exciting, not useful, not a solution, not creative, not startling, not thought provoking. Most projects were the development of websites that frankly looked no better than you or I could create on Wix or Squarespace. There were one or two things I saw that impressed, perhaps especially the creation of an in-car fascia for a German automobile manufacturer, but they were very much the exception rather than the rule.

Executive after executive walked through their case study, breathless and proud as a new parent, suggesting that what we were witnessing was cutting edge, market leading, ground breaking, unprecedented, envelope stretching IP that the client was delighted by and was an instrumental contributor to their client’s business over performance.

Maybe that was true but I doubt it.

Collectively, those clients spent close to $100bn on the work that we saw. Let that sink in. $100bn on routine, run of the mill, game unchanging software.

Let’s dig in further. Why was the software not very good?

Were the clients at fault? Were they asking for the wrong thing? Simple things, lacking vision and ambition. Were they manifesting corporate arrogance? Asking for custom software when a commercial off the shelf solution would have sufficed if they had swallowed their pride and dropped the notion that their needs were unique. Were they manifesting political battles and peacocking their power - to their subordinates and peers - by commissioning vanity projects from their household name suppliers?

Were the vendors at fault? Overpromising and underdelivering, sowing FUD, parachuting in their thought leaders while backing up the school bus? Were they selling custom and delivering pre-baked cookie cut templates? Were they getting bogged in down in unresolvable client side issues and just trying to deliver something that ensured they got paid?

Perhaps it was some combination of the above.

Perhaps also it was something more mundane. That the software being developed to support and enable the tasks and processes within the clients was by definition not very exciting because the tasks and processes themselves are not very exciting. Anyone who has spent more than five minutes working for a big western corporation knows that the HR systems, the finance systems, the procurement systems, even in the biggest, seemingly most impressive household name organizations, are, well, a bit rubbish. At best almost invisible, in that they work. At worst, toe curlingly painful, and to be avoided at all costs. This is true, it should be noted, even after 20 plus years of “consumerization” and business to consumer (aka Amazon) inspired software as a service. As a further aside, it should also be noted, that this dynamic is even more extreme in the custom software development shops themselves - the classic case of the cobbler’s children.

Perhaps it was something else altogether. Perhaps the fact that developing software is hard. Developing code that is functional, engaging, robust, secure, efficient, cost effective, usable, and better than what preceded it, is non-trivial, and as with all things in life, doesn’t come cheap. Perhaps my ennui overlooked this fact and my 17 years in the Gartner trenches made me hard to impress when I should have had my Carol Dweck Mindset to mind.

https://www.goodreads.com/work/quotes/40330-mindset-the-new-psychology-of-success

Good things - including software - are rare and in most things we are surrounded by mediocrity. Most cars are mediocre, most houses, most football teams, most clothes, most movies, most people. Most of the time those mediocre things (and people) are developed by people trying their hardest, putting in 110%, working their butts off, sweating the small stuff, doing the right things, but ending up with outcomes that are simply, mediocre. (More asides; of course, lots of things (and people) are made by people doing none of the above).

That, I think, is what I saw in the Magic Quadrant process. Mediocre projects made by well intentioned people. People who had been to Harvard and Yale and Cambridge and IIT Madras and had spent years at the coal face and in scrums figuring out the tricks of the trade and where the bodies were buried. In short, people who are far from mediocre.

If, like me, your inner Munch is activated every time you go onto your insurance company’s payment portal or the ticket reservation system for the fancy museum you’re visiting, or when your smart TV goes dumb, you, like me, probably think “how come this system is so, pardon my Norwegian, dritt? This is 2024! This should be easier!” The reason it’s not is because a) making things easy is hard and b) you have unrealistic expectations.

A final word. Will GenAI/LLMs make better software? I will explore this in upcoming posts - but as a preview of my thinking … probably, but maybe not.

To be continued …