I have been developing software for over 5 years now, and one of my largest pitfalls that gets me is my tendency to overengineer.
Overengineering, for those who have yet to experience, or recognise it in their own work, is creating additional complexity to a solution with no, or negative results stemming from it. Usually added in for the sake of abstraction, preferred software patterns or sometimes irrational levels of encapsulation.
I started overengineering when I finally “got” Object Oriented Programming (OOP) back in 2007. Just like many of us, I saw the power in representing solutions in terms of objects. My reaction was to convert all of my current projects into Object Oriented format within the PHP language.
Anyone who knows PHP will know why PHP is bad. At this point it is important to note that PHP has never been designed for Object Oriented Programming.
I was developing an image host that I have mentioned in the previous post called “The Rotating Signature Host”. When I completed the first edition of the site in OOP, I looked back at the code and saw it had patterns, that parts could be abstracted and reused. And so I set out to abstract all of these behaviours so that I could reuse them. Everything from page loading to session management, to language, to every little aspect was consolidated into it’s own module.
This might seem ideal at first, as this is essentially what a framework does. It might seem that all of my future products would have benefited from this awesome framework.
That was not the case
Why? I overengineered it. The cause of this overengineering needs to be explored before I can sufficiently explain how.
When I started this project I had the idea that HTML forms were incredibly tedious and messy, that dealing with the number of inputs was a sheer nightmare. I also desired to handle every single possible case of input and present a very explicit message upon receiving bad input. Considering that I was a lone programmer, this meant I had a lot to deal with. In total, the site had many hundreds of inputs, many of them unique in one way or more.
To handle the amount of exceptional cases, exceptions were thrown often, so I had to build an exception handler to ensure that I could keep track of the exceptions.
This desire to regulate, to own the code, to ensure that I could not make mistakes, and that the “system” was ordered left me with a 30000 line mess that took me 2 years. I rewrote it from the ground up several times, restructuring file organisation, code style, copyright and legal comments. Everything was meticulously crafted.
This was all for a single image host. I then decided I’d make this into a framework. So I started using this overengineered framework in other projects, and realised that the result was so complex that to simply load a single page, I had to manage about 20 different variables, files and entries over the “structure” to ensure it loaded correctly. Large configuration files marred any attempt to make the system more dynamic.
Again, this was originally just for an image host. A small one.
I ended the project
Two years ago I cancelled the project. It had become a monolithic complicated mess of anti-patterns and other mistakes. In the mean time I also over-engineered 50 line scripts into 400 line programs, and other similar bloatings.
The image host ended up collapsing due to rapidly rising costs, and no development. I had planned to implement a way to generate revenue to the site, which was the entire reason I rewrote it. It took 2 years and I scrapped the rewrite, with it I scrapped the whole site.
I have made this mistake many times, and so have fellow colleagues, students and friends. I see over-engineering in many projects, particularly enterprise solutions.
What did I do about it?
I took up a job last year at a company, and I similarly overengineered their site, and didn’t really bring it to completion because of it. These two failures within a year of each other led me to sit down with myself and ask “Why do not finish most things I start?”
I had been told I overengineer by a more experienced developer in the past, but it was hard to understand for me. Why wouldn’t I ensure that all of the features I might ever need are in the code, just in case?
Then it hit me
I realised how inefficient I had become. From writing quick scripts that “just about worked” in a matter of days to writing long programs that “barely worked” I had actually reduced my ability as a programmer despite being better at actually writing code. I knew more ways to solve problems, but this lead to me not picking the optimal one, which is often the simplest, I was ignoring Occam’s Razor.
My mathematical and technical prowess had increased, but the time it took me to develop a solution increased about 4 fold. I am sure a lot of you will recognise one time or other where this has happened.
In the past year the time it has taken me to develop programs has been decreasing. Largely from knowing I overengineer, and trying other languages.
In the end I realised something important; However well I write code, someone will always break it. Rather than accounting for the 2% of very difficult edge cases for every input, output and function and trying to intelligently handle every possible complication, abstraction and action, both in code and interface, will lead to an absurd complexity and an even more absurd timescale.
The result of all of this was for me to test myself, can I write a general solution for problems in PHP without over-engineering? The project is known as CleanPHP.
I recommend anyone who reads this who hasn’t ever experienced overengineering to take a step back and ask themselves “Do I have too many classes/ methods/ lines/ interfaces/ abstractions?” and “Is this simple website/ applet meant to be taking more than 3 months?” and “Do I really need to write another framework?”, because if you do, you might save yourself, and your clients, employers and coworkers a lot of time, stress and money. We can’t perfect, or even close to perfect most medium to large projects, and as developers we must be as good at managing ourselves as we are engineering our software.
So next time you “just want to abstract out the loading of the