At Crosslake, we often get asked by people who do not have a deep technical background, “How do I measure the effectiveness of our software development team without understanding the technical details?” Investors acquiring software companies (e.g., private equity firms) and other less-technical leaders in a software organization often don’t know where to start.
In this post, we provide a set of questions to ask a software engineering team to gauge their effectiveness.
What is “effectiveness?”
There are many characteristics of an effective R&D team. Before we jump into some questions to ask around R&D effectiveness, here are some key success factors to continuously improve that the questions take into account:
- Higher product value (delivering the right thing)
- Faster time to delivery / revenue (delivering on time)
- Increase product quality (delivering it right)
- Enhance productivity (delivering without wasted cycles)
- Reduce the cost of delivery (delivering with minimal costs)
The questions
Based on the success factors above, here are some questions to monitor to help ensure optimized delivery of software.
Product value
How much time is being spent on roadmap items vs. customer requests?
- WHY: Customer requests that inundate the team may prohibit the development of innovative functionality a customer doesn’t even know they need — and may spur a significant jump ahead of the competition. Real innovation stalls.
- LOOK FOR: All customer requests have ROI justification that maps to business value and is combined with other strategic product investments.
How much time is being spent on maintenance vs. new innovation?
- WHY: If the team is spending a disproportionate amount of time doing maintenance activities when the roadmap stresses innovation, there may be a problem with product quality.
- LOOK FOR: < 15% of total effort being spent on pure maintenance activities.
How much value is being delivered at the end of every iteration or increment?
- WHY: Measuring value can be difficult. Push the product team to measure value on each of the bodies of work being defined and delivered. Monitoring value delivered helps ensure the team is always working on the highest-impact items first.
- LOOK FOR: A basic formula that accounts for growth, retention, cost reduction, urgency and level of effort can help put a value on each work item; ensure the backlog is ranked from highest-to-lowest business value.
Time to delivery
Is the team and delivery tracking to the roadmap?
- WHY: There is a roadmap to guide R&D, right? How does the team track delivery against it? Frequently missed, high-level, roadmap deliverables can be symptomatic of a struggling team. BUT don’t expect long-term, low-level commitments from the team. We are talking roadmap-level here.
- LOOK FOR: 80-95% roadmap items delivered vs. planned.
Quality
How much technical debt exists?
- WHY: Keeping technical debt to a minimum is key to a good long-term business. Have the team quantify the technical debt in units of time (like weeks). Substantial technical debt may indicate that the team is taking too many shortcuts and not being effective at keeping quality high. Poor development choices now can cost more later. Too little technical debt can be (a) a symptom of perfectionism slowing down time to market or (b) not being critical enough of how the software is built.
- LOOK FOR: 2-10 weeks of technical debt (very rough order of magnitude with a lot of constraints dependent on specific company situations).
How frequently is the team delivering shippable-quality software?
- WHY: The team should produce “working software” regularly, even if it doesn’t get released. Effective R&D teams maintain high quality continuously so as not to build up quality debt.
- LOOK FOR: An end-to-end, fully-integrated working system every 8-12 weeks and ideally at the end of every iteration (e.g., 2 weeks); critical use cases should be demonstrable at least at the end of an iteration or ideally, every day.
What are the bug trends during some increment of time (e.g., an iteration or a month), particularly post-release?
- WHY: Excessive and upward-trending bug numbers after a release can point to effectiveness and quality issues with the development team. Be sure to analyze trends (rolling averages), and not specific points in time, to get an accurate picture.
- LOOK FOR: A decreasing rolling trend of bugs, particularly close to release.
What are the general availability trends?
- WHY: First, there must be an easy way to gather availability information through robust monitoring tools. Lower availability may indicate product issues and subsequently issues with how the product is developed and / or deployed.
- LOOK FOR: Assuming a web-based style of product, 5 9s of uptime (i.e., 99.999%) is a reasonable goal.
What are the overall user satisfaction metrics (e.g., Net Promoter Score)?
- WHY: Poor satisfaction metrics can be indicative of several problems, including quality issues inside the development team (e.g., Dev or QA practices) and inattention to the user experience (e.g., developer-driven UI design).
- LOOK FOR: A Net Promoter Score (https://www.netpromoter.com/know/) of > 0 is a positive sign and >= +50 is considered excellent.
Productivity
How much time is being spent on manual work vs. work that could be automated?
- WHY: The most effective development teams automate any error-prone and rote processes. This includes regression testing and deployment to various environments. Measuring the number of test cases that are manually executed vs. those that are automated can provide some indication of efficiency.
- LOOK FOR: 60-80% of test cases automated; deployment operations 100% automated.
What is our cumulative flow diagram looking like across the program?
- WHY: A cumulative flow diagram, or CFD, can provide an indication of completion rate, too much work in progress slowing the team down, a growing backlog that may indicate feature creep and other symptoms of issues in a development cycle. Learn how to read one at the program level to get a reasonable indication of how the team is doing.
- LOOK FOR: Downward or flat trends with work items or thin bands of in-progress work.
What is the velocity trend of the team over the past few iterations?
- WHY: Teams should be able to generate velocity numbers reasonably easily. Velocity indicates the amount of value that a team can deliver per unit time (usually an iteration). The absolute number, in this case, is less important than the relative number (trend) over time. If velocity is going down, ask why, and address the root cause.
- LOOK FOR: Mildly increasing trends in team velocity (assuming static team size over time).
Cost of delivery
What is the overall cost of R&D as a percentage of revenue (as dictated by business goals)?
- WHY: Effective R&D teams continuously work on cost efficiency while delivering value that increases revenue. The high cost could indicate problems like an overabundance of senior engineers or an expensive development location. Of course, the number varies depending on the mode of delivery. For example, a team simply maintaining a cash-flow product should be spending significantly less on R&D than a team in heavy innovation mode.
- LOOK FOR: Percentages less than 15%; greater percentages can be indicative of excessive R&D spend and may be an optimization candidate.
The questions above should get you started, even though they are not comprehensive and there are nuances within most of them. As always, don’t hesitate to reach out with any questions you may have.
