With headlines frequently highlighting tech disasters from some of the world’s most trusted brands, we started to wonder: are there lessons here for middle market companies?
We reached out to Crosslake’s practitioner community to get advice for investors and their portfolio companies about dealing with technology growing pains – before they create tomorrow’s headlines.
Here’s what they had to say.
Contributors
Jason Botts, Practitioner
Jason brings more than two decades of IT operational and strategic experience in life sciences, high tech, FinTech and construction. His background includes transforming and leading technology teams that have been repeatedly recognized by InformationWeek magazine as Elite 100 Innovators – from entrepreneurial startup/Fast 50 companies to the S&P 500. Jason serves by appointment of the Governor to the North Carolina Board of Science, Technology and Innovation and was recently selected as a technology leader in Aspen Institute’s Technology Executive Leadership Initiative (TELI).
Vlad Papayan, Practitioner
Vlad brings more than 25 years’ experience in engineering leadership, software product delivery and professional services. He’s known for his ability to deliver within complex public and private enterprises, and his expertise spans late-stage startups, midsize and Global 500 enterprises. His portfolio of successes includes solutions for enterprise-wide authorization and entitlements, customer care systems and innovative monetization of existing product and data portfolios.
Uma Palepu, Director
Uma has 25+ years of experience leading global engineering teams at companies including Mitchell, EMC and Captiva. He is passionate about operational efficiency, organizational design, scaling up and digital transformation. He enjoys building high performance teams and helping leaders achieve their highest potential.
Kim Walters, Senior Managing Director
Kim brings more than 30 years of leadership, engineering, product and program management experience in technology – including 18 years at Microsoft, where she led large, geographically diverse engineering teams. She is known for her strategic vision, her ability to optimize resourcing and her aptitude in solving business problems through a mix of process improvements and innovative technology.
What are the warning signs that a company is outgrowing its current platform(s)?
Jason: Something we saw in the case of Southwest Airlines is that the earliest indicator is likely to be your people. Your teams are holding the ship together, so they’ll know first when something’s no longer working. If there’s not enough time or resources to keep your finger on the pulse of your people, you’re on borrowed time.
Uma: Often, you’ll see technology challenges reflected in the business’s ability to serve its customers. Some of the signs we look for in technology diligence are things like increasing ticket volumes for customer support, ticket resolution times that are increasing year-over-year or unusually long onboarding timelines for new customers.
Vlad: We’re seeing more companies including annual technology assessments in their budget, so they can report results as part of their operational health KPIs. Not only can these assessments help surface technology pain points and serve as an early-warning system for the technology team – but consistent reporting of these metrics to the C-suite also ensures that required technology investments remain top-of-mind in budget and risk management discussions.
What are some technology risks that may be missed when conducting an M&A transaction?
Vlad: It’s not uncommon to uncover a previously unknown security issue in a third-party library or service during diligence. You have to specifically look for these issues using a third-party scanning tool or service – so it’s especially important to run a scan like this when there’s no history of them being performed as part of the target’s documented security assessments.
Uma: There are some nuances that might not present as obvious areas of concern:
- Inappropriate level of cloud utilization. Maturity of cloud adoption tends to vary quite a bit – from laggards who refuse to switch over to those who switch over but don’t have a clear handle on how to leverage the cloud. You need deep and broad architecture experience to keep costs in check, maximize the benefits and ensure the level of cloud adoption is appropriate for the business.
- Outdated tech stack. Modern technologies are built on open-source and other contemporary platforms, such as NodeJS, and .NET. If the target has a high growth trajectory, but is leveraging outdated technology, tech debt can balloon. Legacy technologies pose security risks (especially for older open-source frameworks) and create challenges for finding talent as they enter the “sunsetting” path of declining support and eventual obsolescence.
- Integration challenges. When the target has a unique strategy that differs from common practices – lacks an API-first design strategy or utilizes a customer-hosted model, for example – it can be difficult to quickly integrate their technology with other PortCos.
Kim: It’s natural to focus on technology and processes during the diligence process, but you can’t overlook the current talent pool. We often see that the leadership team responsible for achieving the current level of revenue and success is not the right team to manage an integration or move the company forward. With a talent gap analysis, you can understand what is needed: an interim leader to mentor and upskill the existing team, a permanent change in leadership or general upleveling of the talent.
What are some best practices for addressing technology challenges while continuing to serve customers?
Jason: Addressing technology challenges is serving customers – but preventing technology challenges serves them even better. Architecting out bottlenecks, eliminating single points of failure and incorporating up-and-down scalability into systems design is key. When that hasn’t been the case and failures occur, it’s about communicating transparently, understanding the priorities of the moment/situation, and seeking root causes during recovery so you don’t “recover” into another situation of failure.
Uma: You need to start with a modernization or transformation plan that catalogs technical debt and automation priorities in alignment with your overall technology roadmap so you can consider capacity constraints and set priorities. That can be easier said than done – so we often recommend that you seek investor alignment for surge investment to fund additional resources. With extra resources, you can dedicate a team to tackling modernization and transformation activities without impacting product development and customer support initiatives. Eventually, you’ll catch up the tech backlog and will have integrated best practices into your workflows to address the underlying issues and prevent them from recurring.
What are the security impacts of technical debt?
Kim: Team’s can’t just address security periodically. Like quality, it needs to be part of a team’s DNA. We say that teams should dedicate a percentage of their resources every sprint to technical debt – and this should include regular security hygiene. Penetration tests, open-source scans and code quality scans will provide an ongoing list of security vulnerabilities that need to be triaged, placed into the technical debt backlog and mitigated.
Vlad: When technical debt is due to duplicative artifacts (too many systems or components doing the same thing) or significant variabilities in the tech stacks utilized (for example, one app that has Ruby on Rails, PHP, and Java), it creates a larger attack surface that is harder to defend.
Uma: Keeping open-source software current is always critical to improving your security profile. Older open-source software tends to have security vulnerabilities that are publicly known. Hackers build script kits and share them via the dark web to exploit these vulnerabilities for ransomware attacks and other hacking activities.
Any guidelines for allocating budget and resources to infrastructure maintenance and improvements?
Vlad: For me, this depends on the risk profile, maturity level and market expectations. Budgeting discussions often revolve around two main concerns: growing new revenue and protecting existing revenue. What we’ve seen with some of these high-profile tech failures is that inadequately budgeting for tech investment can compromise both revenue streams. Technology leaders need to be able to bridge the gap with finance by translating terms like “technical debt” into expected revenue impact.
Uma: IT infrastructure budgets can vary based on a number of factors: company size, lifecycle stage, number of products, cloud or self-hosted. When companies manage their own datacenters, hardware refresh cycles are typically every three to five years. Where the public cloud is used, the focus is on managing costs through effective utilization, focusing on architecture and test automation and leveraging the elastic capabilities of the cloud. One good rule of thumb is to keep the total infrastructure cost to grow sub-linearly to revenue growth.
Your annual R&D budget needs to include allocations for each of these four areas (although the percentage allocation will vary based on business context):
- New features and functionality
- Support and maintenance
- Technical debt management
- Architecture evolution/platform upgrades
As you’re budgeting, keep in mind that human capital costs tend to be the most significant contributing factor to any technology effort. To avoid delays in execution, make sure your budget and plan includes training for any upskilling required – and accounts for the time required to hire and onboard any new resources.
Kim: As a general rule for SaaS companies, about 15% of your resources should be dedicated to platform improvements, which may encompass:
- Breaking down the monolith, through further migration to microservices and self-supporting components
- Modernizing the deployment process through the introduction of infrastructure as code (IaC)
- Performing an audit and mitigation exercise to reduce infrastructure cost or improve security
When does technical debt become insurmountable?
Uma: There are some significant business impacts that may signal when technical debt is approaching a critical level:
- Innovation slows. When significant capacity is diverted to maintenance and bug-fixing efforts, the technology team is not able to respond as quickly to industry trends and customer feedback.
- Product releases are delayed or become less frequent. Legacy tech challenges can require significant testing of new products and features to ensure quality, slowing the release schedule.
- Defects increase. More defects are found in production and surfaced via support tickets.
- Support issues escalate. As issues pile up, the support team is unable to resolve customer issues within defined SLAs.
- Crucial infrastructure is sunsetted. When foundational platforms and technologies are phased out and no longer supported, your team will have difficulties maintaining your systems, and you’ll struggle to find talent with the skillsets needed.
Jason: As technologists, we rarely see any technical hurdle as insurmountable. It’s technology – we can do anything! As a business leader, however, “insurmountable” translates to negative ROI. Building a plan that supports the business needs and addresses significant technical debt is a collaborative effort among a company’s sales, operations, finance and technical teams. As a going concern, “insurmountable” simply isn’t an option. As an advisor to the business, it’s the technical team’s job to bring what’s possible to the conversation. The surmountable path then becomes a collective business decision.
Note: Intechnica, a Crosslake company, has even more to say about understanding and addressing the technical debt tipping point.
