With headlines highlighting tech debt disasters from some of the world’s most trusted brands, we wondered: what are the lessons for middle market companies?
We reached out to Crosslake’s practitioner community to get advice for investors and their portfolio companies. Here’s what they had to say about dealing with technology growing pains – before they create tomorrow’s news.
Here’s what they had to say.
Contributors
Jason Botts, Practitioner
Jason brings more than two decades of IT operational and strategic experience in life sciences, high tech, FinTech and construction. His background includes transforming and leading technology teams repeatedly recognized by InformationWeek magazine as Elite 100 Innovators – from startup / Fast 50 companies to the S&P 500. Jason serves by appointment of the Governor to the North Carolina Board of Science, Technology and Innovation and as a technology leader in Aspen Institute’s Technology Executive Leadership Initiative (TELI).
Vlad Papayan, Practitioner
Vlad brings more than 25 years’ experience in engineering leadership, software product delivery and professional services. He’s known for his ability to deliver within complex public and private enterprises. His expertise spans late-stage startups, midsize and Global 500 companies. His successes include solutions for enterprise-wide authorization and entitlements, customer care systems and innovative monetization of existing product and data portfolios.
Uma Palepu, Director
Uma has 25+ years of experience leading global engineering teams at companies including Mitchell, EMC and Captiva. He is passionate about operational efficiency, organizational design, scaling up and digital transformation. He enjoys building high performance teams and helping leaders achieve their highest potential.
Kim Walters, Senior Managing Director
Kim brings more than 30 years of leadership, engineering, product and program management experience in technology – including 18 years at Microsoft, where she led large, geographically diverse engineering teams. She is known for her strategic vision, her ability to optimize resourcing and her aptitude in solving business problems through a mix of process improvements and innovative technology.
What are the warning signs that a company is outgrowing its current platform?
Jason: Something we saw in the case of Southwest Airlines is that the earliest indicator is likely your people. Since your teams are holding the ship together, they’ll know first when something is no longer working. If you lack time or resources to keep your finger on the pulse of your people, you’re on borrowed time.
Uma: Often, you’ll see technology challenges reflected in the business’s ability to serve its customers. In technology diligence, we look for things like increasing ticket volumes for customer support, ticket resolution times that are increasing year-over-year or unusually long onboarding timelines for new customers.
Vlad: We’re seeing more companies include annual technology assessments in their budget, so they can report results in operational KPIs. These assessments help surface technology pain points and serve as an early-warning system for the technology team. And consistent reporting of these metrics to the C-suite ensures that required technology investments remain top-of-mind in budget and risk management discussions.
What technology risks may be missed when conducting an M&A transaction?
Vlad: It’s not uncommon to uncover a previously unknown security issue in a third-party library or service during diligence. You have to specifically look for these issues using a third-party scanning tool or service – so it’s especially important to run a scan like this when there’s no history of them in the target’s documented security assessments.
Uma: There are some nuances that might not present as obvious areas of concern:
- Inappropriate level of cloud utilization. Maturity of cloud adoption tends to vary quite a bit. There are laggards who refuse to switch over and those who switch over without a clear handle on how to leverage the cloud. You need deep and broad architecture experience to keep costs in check, maximize the benefits and ensure the level of cloud adoption is appropriate for the business.
- Outdated tech stack. Open-source and other contemporary platforms, such as NodeJS, and .NET, are the hallmarks of modern technology. If the target has a high growth trajectory, but is leveraging outdated technology, tech debt can balloon. Legacy technologies pose security risks and create challenges for finding talent as they enter the “sunsetting” path of declining support and obsolescence. This is especially true for older, open-source frameworks.
- Integration challenges. When the target has a unique strategy that differs from common practices – lacks an API-first design strategy or uses a customer-hosted model, for example – it can be difficult to quickly integrate their technology with other PortCos.
Kim: It’s natural to focus on technology and processes during diligence, but don’t overlook the current talent pool. We often see that the leadership team responsible for achieving the current level of revenue and success is not the right team to manage an integration or move the company forward. A talent gap analysis clarifies the need: an interim leader to mentor and upskill the existing team, a permanent change in leadership or general upleveling of the talent.
What are some best practices for addressing tech debt challenges while continuing to serve customers?
Jason: Addressing technology challenges is serving customers – but preventing technology challenges serves them even better. Architecting out bottlenecks, eliminating single points of failure and incorporating up-and-down scalability into systems design is key. When failures occur, it’s about communicating transparently, understanding the priorities of the moment or situation, and seeking root causes during recovery. You don’t want to “recover” into another situation of failure.
Uma: Start with a modernization or transformation plan that lists technical debt and automation priorities in alignment with your overall technology roadmap. This allows you to consider capacity constraints and set priorities. Of course, that can be easier said than done! We often recommend seeking investor alignment for surge investment to fund additional resources. With extra resources, you can dedicate a team to tackling modernization and transformation activities without impacting product development and customer support initiatives. Eventually, you’ll catch up the tech debt backlog and integrate best practices into your workflows. This will not only address the underlying issues, it will also prevent them from recurring.
What are the security impacts of tech debt?
Kim: Teams can’t just address security periodically. Like quality, it needs to be part of a team’s DNA. We say that teams should dedicate a percentage of their resources every sprint to tech debt. This should include regular security hygiene, as well. Penetration tests, open-source scans and code quality scans provide an ongoing list of security vulnerabilities to triage, place into the technical debt backlog and resolve.
Vlad: When tech debt is due to duplicative artifacts (too many systems or components doing the same thing) or significant variabilities in the tech stacks used (for example, one app that has Ruby on Rails, PHP, and Java), it creates a larger attack surface that is harder to defend.
Uma: Keeping open-source software current is always critical to improving your security profile. Older open-source software tends to have known security vulnerabilities. Hackers build script kits and share them via the dark web to exploit these vulnerabilities for ransomware attacks and other hacking activities.
Any guidelines for allocating budget and resources to infrastructure maintenance and improvements?
Vlad: For me, this depends on the risk profile, maturity level and market expectations. Budgeting discussions often revolve around two main concerns: growing new revenue and protecting existing revenue. What we’ve seen with some high-profile tech failures is that under-budgeting for tech investment compromises both revenue streams. Technology leaders must bridge the gap with finance by translating terms like “technical debt” into expected revenue impact.
Uma: IT infrastructure budgets can vary based on a number of factors: company size, lifecycle stage, number of products, cloud or self-hosted. When companies manage their own datacenters, hardware refresh cycles are typically every three to five years. Where the public cloud is used, the focus is on managing costs through effective utilization, focusing on architecture and test automation, and leveraging the elastic capabilities of the cloud. One good rule of thumb is to keep the total infrastructure cost to grow sub-linearly to revenue growth.
Your annual R&D budget should include allocations for:
- New features and functionality
- Support and maintenance
- Technical debt management
- Architecture evolution and platform upgrades
The percentage allocation will vary based on business context.
As you’re budgeting, keep in mind that human capital tends to be the most significant factor in any technology effort. To avoid delays in execution, make sure your budget and plan includes training for required upskilling. It should also account for the time required to hire and onboard any new resources.
Kim: As a general rule for SaaS companies, about 15% of resources should be dedicated to platform improvements. These may include:
- Breaking down the monolith, through further migration to microservices and self-supporting components
- Modernizing the deployment process through the introduction of infrastructure as code (IaC)
- Performing an audit and mitigation exercise to reduce infrastructure cost or improve security
When does tech debt become insurmountable?
Uma: Some significant business impacts signal when tech debt is approaching a critical level:
- Innovation slows. The technology team is unable to respond as quickly to industry trends and customer feedback. Typically, that’s because significant capacity is diverted to maintenance and bug-fixing efforts.
- Product releases are delayed or become less frequent. Legacy tech challenges can require significant testing of new products and features to ensure quality, slowing the release schedule.
- Defects increase. The team discovers more defects in production, and support ticket volume increases.
- Support issues escalate. As issues pile up, the support team can no longer resolve customer issues within defined SLAs.
- Crucial infrastructure is sunsetted. As foundational platforms and technologies are phased out and no longer supported, your team encounters difficulties maintaining your systems. In addition, you’ll struggle to find talent with the skillsets needed.
Jason: As technologists, we rarely see any technical hurdle as insurmountable. It’s technology – we can do anything! As a business leader, however, “insurmountable” translates to negative ROI. Building a plan that supports the business needs and addresses significant technical debt is a collaborative effort among a company’s sales, operations, finance and technical teams. As a going concern, “insurmountable” simply isn’t an option. As an advisor to the business, it’s the technical team’s job to bring what’s possible to the conversation. The surmountable path then becomes a collective business decision.
Read more: Learn the signs that your business is approaching the tech debt tipping point.
