The famous, American poet Emily Dickinson is credited back in the 19th century as stating that “forever is composed of nows.” Today, the world of software is all about rapid releases, innovative new features, continuous builds and deployments, maintaining high quality, and making consistent improvements. In order to stay afloat in this competitive market, companies often sprint to the finish line — and when they sprint, they tend to fall and make mistakes. Software quality suffers, money is spent on the wrong resources, and employees’ perception of the organization and their work is damaged.
It is during these rushed development processes that many basic R&D concepts like technical design and technical review are left behind. Neglecting these activities leads to technical debt. Non-scalable code, double (or more) coding, and inefficient code are the most common factors of technical debt.
However, code is not the only factor of debt. Failing to address functional problems, platform-level design holes, or security issues in time usually leads to refactoring, and can even result in software entropy.
In this article, we discuss the different aspects of technical debt, how to manage existing debt, and how to minimize debt in the future.
Charging forward without looking back
What Causes Technical Debt?
To satisfy customer needs and stay competitive in this fast-paced industry, almost all software organizations have embraced the agile methodology and implemented it in one way or another. Agile pushes R&D and DevOps to use continuous delivery (CD) processes to deploy software quickly and safely, which requires automatic processes for building, testing, and deploying the code in the cloud.
Many R&D groups try to increase the amount of deliveries and shorten delivery times by trimming some of the development steps out of the process. Developers, pressured to create new functionalities quickly, tend to write code that makes the required functionality available, but that does not take into account future business requirements or refactoring that will be needed should this code be used in the future/in other areas of the product. Such code debt eventually leads to longer development periods, draws more attention to the code from developers and management than it should, and ends up in a decrease of release content. In many cases, developers who over-architect or over-code software are blamed for product launch delays and wasted money.
When software companies create software, it usually involves several teams and many stakeholders. The higher the number of participants in the development process, the bigger the technical debt created during development will be. When different teams code and the group lacks a high level of technical overview, a similar functionality might be implemented more than once in different ways. While the software may function correctly, code maintenance becomes more difficult and productivity diminishes.
Another substantial cause of a growing technical debt is defects. Funny enough, defects are also the outcome of this debt. Many software organizations handle defects that are found in production, or beforehand with quick fixes and patches. Defects that are fixed under pressure might solve an immediate problem, but they can affect the way code and/or software behaves and introduce new problems.
Technical debt created in the development process is the root cause of many defects that affect the product as a whole. Eventually, stakeholders begin to lose confidence in the quality of the code.
Does Technical Debt Affect Other Areas of the Software?
Another effect of technical debt is the limited ability to implement new technologies. Developers and teams need to keep up with the latest technologies and updates in order to continuously create and deliver high-quality products for their customers. When older code prevents the implementation of a new technology or new libraries of the used language, it is necessary to change the way the code is written — or even rewrite the whole area of the product.
Security can easily become another critical debt. Many software organizations tend to execute security testing and verifications as the last step before the release. Unlike with other types of bugs in the product, security bugs that are found after the code has been tested and approved can lead to a deep refactoring and complete change of the code — or even the whole project’s infrastructure. If the security risk is high enough, the release will be delayed. The risk aggravates when R&D decide to use open source code or infrastructure as their platform for developing new features, which is a very common practice throughout the software industry. As open source code vulnerabilities are made public, becoming easily available to hackers, the inclusion of vulnerable open source components can create a significant threat surface for attackers to exploit. Organizations choosing to build their software on top of open source code without verifying the code’s sturdiness in the first place, end up with a huge debt to the software security which can later lead — once the security examination is conducted — to change to an alternative code and start developing from scratch.
Technical debt management
In order to manage and keep new technical debt under control, Ward Cunningham introduced the concept of Technical Debt Management (TDM). TDM is a term that describes the consequences of taking shortcuts throughout the software design and development process, and how to treat this outcome. The level of TDM implemented by R&D is strongly correlated with the maturity of the software and development teams. More experienced teams usually have high awareness and apply more advanced processes, techniques, and tools in TDM.
Types of Technical Debt
TDM is based on four types of debt, which can all exist within the same organization. Time constraints, limited resources, and redundant coding written by multiple people all play a role in determining which type of debt is being created.
The prudent and deliberate teams are usually aware when they create debt, but they do so because they feel that they need to respond to business demands or attend to urgent production problems (e.g., new security threats). The reckless and deliberate teams have poor code and team management. They often cut corners in order to meet a deadline. This behavior is a very common cause of technical debt. The reckless and inadvertent teams are probably less experienced and are unaware of the implications of changing or adding code, which often incurs a huge amount of technical debt. The prudent and inadvertent type occurs naturally. As long as debt that is created in the development process is managed, then it is acceptable.
How to Handle Technical Debt
TDM suggests a handful of strategies that can help you manage technical debt:
- Agree upon what is considered to be technical debt
- Quantify the debt
- Continuously monitor the debt
- Track debt over time and releases
- Understand the cost of the debt that is being incurred
The first strategy is used during the design phase of the TDM. It determines what code will be reengineered and what will be left as-is. Measuring the existing debt results in time and cost estimation for fixing the debt, and lets you break it down into smaller portions so that it can be assigned to the software releases.
Since debt is an inseparable and significant part of the product — as it affects the quality, the ability to adopt new technologies and the capacity for improving the product and its development and testing processes, all existing debt (handled in the current release or in future releases) must be monitored constantly and displayed for all of the relevant personnel involved in the product. This will ensure that the remaining work is defined and being tracked. Full visibility of the debt will open up a healthy conversation between the business-oriented teams and the technical ones. Since debt is not only liquidated but also accumulated, tracking the old debt, the debt that has been handled, and the new debt is important. Doing this tracking drives constant reevaluations of your thinking, estimations, and priorities.
There are many practices that can help vendors handle technical debt. For example, they can implement an orderly security code review as part of the development processes, create a verification process for code prior to merging it into master, choose the best solution rather than the easiest one, create automation processes for validating security and execute them as part of the CI process, and constantly update the code and infrastructure. Regrettably, very few R&D management teams have taken the time to seriously consider these helpful options, so very few vendors have implemented them thus far. As explained earlier, this causes delays in release, reduces the product quality, and harms the company’s reputation.
As more and more mid-sized software organizations and enterprises recognize that addressing technical debt is a major priority, they are assimilating TDM into their development processes. They are beginning to realize why these strategies are crucial for governing code and handling debt, and they are paying close attention to the influence on the product and the team.
Conclusion
Technical debt is a natural and expected cost of doing business, but your company must handle it responsibly. While accepting technical debt is sometimes necessary in order to reach your goal, you should avoid taking on additional debt when it is not an absolute must. Be aware of the fact that the time saved from taking shortcuts in the present will result in more work later on.
Since some technical debt can and should be avoided, utilizing the correct tools in the right places in the development, testing, and deployment processes can help decrease debt and improve effectiveness. R&D teams should learn how to calculate their technical debt and share this information with stakeholders and management. Doing so will hopefully convince those who are higher up that debt needs to be addressed, and might even prioritize it over developing new functionality.
Preventing future debt is also a substantial task, so code debt which was handled will consume more attention and resources. Code and security reviews must become and remain an integral part of the development process and security verification must be mechanized and embedded into the CI/CD process. Only that way, the team can be sure that new content developed doesn’t generate additional work for the future.
