How to Evaluate a Department – Software Development
Evaluating any department in any company takes time and experience. Even more so if it’s not the department you work in. Discovering the vocabulary and cadence of activities and their way of thinking is critical to assessing their performance. Metrics those departments use to measure performance can speed up that learning but doesn’t tell the full story.
For a finance or accounting professional, gauging the performance of a software engineering department can be challenging. It requires assessing a mix of quantitative metrics, qualitative feedback, and alignment with the company’s strategic goals. The following guide outlines key indicators and methods to determine whether an engineering department is performing well:
1. Delivery Metrics
The ability to deliver high-quality software efficiently and on time is a critical measure of success.
- Velocity: Track the number of story points or tasks completed in a sprint. While velocity alone doesn’t define success, trends over time can indicate whether the team is improving or facing bottlenecks. A story point is sometimes defined as the amount of work a software engineer can code in eight uninterrupted hours of coding. However, this definition is not universal across companies or even across teams within the same company. To “empower” software engineers companies will sometimes allow the engineers to define their story points or other measurement of the number tasks required to complete a build. If multiple ways of measuring effort is discovered, push the software engineering leader to define one standard for the company. If not, it’ll be a lot harder to measure efforts of the engineers.
- Cycle Time: Measure how long it takes for a feature, bug fix, or task to move from start to completion. Shorter cycle times often indicate efficiency. In practice, this has been hard to measure and enforce. Related to on-time delivery discussed later, asking for estimated completion dates helps a lot. If they hit it great. If they don’t it the deliver deadline, inquire of the team leads or other leaders and take note of the reasons given. You’ll want to track whether those reason recur and how often. You can then start socializing those reason with other leaders in the company to determine whether there’s merit to those reasons. As time goes by, you can start to gain an understanding how specific teams perform.
- Release Frequency: Assess how often the engineering team deploys updates or new features. High-performing teams often embrace continuous delivery practices. This is often done in conjunction with marketing and other go-to-market teams to prepare for new releases.
- On-Time Delivery: Track the percentage of projects completed within the estimated timeline. As discussed above, if deadlines are missed or extended that could mean a number of issues. It could be that they software leader giving the estimate is unrealistic, or that the problem was harder than originally believed, or that the scope grew during the build, or that projects were deprioritized to advance another initiative. All these are reasons for concern as a one off situation, but if they constantly occur, then there are larger issues with the company or software engineering department. Any deadlines you get from any engineer should be checked with engineer leaders or other leaders in the company to confirm those expectations. Nothing blows the revenue forecast faster than revenue dependent on a new product that gets delayed.
2. Code Quality
Code quality affects the maintainability, scalability, and reliability of the SaaS product.
- Code Review Metrics: Evaluate the time spent on and frequency of peer code reviews. High-performing teams prioritize rigorous reviews to ensure quality.
- Defect Rates: Monitor the number of bugs identified during testing or post-release. A low defect rate often indicates strong engineering practices. As with many things, bugs vary tremendously. Some bugs are quick fixes and some are major and need to go back to the original build team. That said, the absolutely number of bugs does not necessarily tell how good the original build was. Ask software development leaders how much of the developers time should be attributable to bug fixes. I’d recommend 20% as a starting threshold. More than 20% is bad, less is good. However, with time this can be gauged to see what is normal. Customer feedback will be very helpful here. The support department typically has good insight into what parts of the product customers need help with.
- Technical Debt: Track and address technical debt regularly. High technical debt can slow down progress and reduce long-term performance. This is another way to track performance that’s harder in practice than in theory. Fixing or updating older parts of the code is often just assumed, understood and rolled into new projects or bug fixes. Unless the software development team actively tracks this metric, you probably move on and tackle lower hanging fruit.
3. System Reliability
For a SaaS company, system uptime and performance are crucial.
- Uptime/Downtime: Measure the percentage of time the system is operational. High-performing engineering teams aim for 99.9% uptime or better. For a software department this metric is a no brainer and likely any contracts signed with customers have commitments or requirements for uptime performance. Downtime should be so rare that you don’t worry about it as a finance department. If the company, especially a SaaS company, can’t keep their product running then the company has major problems. If software is only a portion of the product offering, then downtime might be more common based on the circumstances of the company and customers. For example industrial equipment with that needs regular servicing might has significant downtime and is understood by the parties involved.
- Response Time: Track system response times under various loads to ensure consistent performance for users. Response time for engineers in software functions are typically an escalated occurrence and less relevant than frontline support responses.
- Incident Response: Assess how quickly the team resolves outages or performance issues. The mean time to resolve (MTTR) incidents is a key metric.
4. Innovation and Feature Development
A strong engineering team contributes to product innovation and development.
- Feature Completion Rate: Measure how many planned features are completed within a quarter or year. As the finance or accounting leader, you need to tag any features with a planned deliverable date. Typically the quarter or month is sufficient. That allows you to then build revenue expectations with the go-to-market leaders. As discussed previously, if those deadline are constantly missed, there’s more work to do with the software development departments.
- Customer Feedback: Gather feedback from customers on newly developed features to gauge their usefulness and impact. Customer feedback is gold. They will tell you what’s wrong with the product and who is going to churn and so much more. It’s not always easy to get. If customer feedback is lacking, work with the support or success or sales functions to start gathering. Any professional in those functions should have ideas for how to collect and aggregate feedback.
- Alignment with Product Goals: Ensure that the engineering department is working closely with product management to deliver features that align with strategic objectives.
5. Employee Engagement and Retention
The health of the engineering team itself is a strong indicator of overall performance.
- Employee Turnover Rate: High turnover can signal dissatisfaction or burnout. A stable team is often a sign of good leadership and morale. Some turnover however, can be good. Involuntarily terminating team members that are causing strife within the company can be really good for the company.
- Engagement Scores: Use surveys to gauge job satisfaction, collaboration, and overall engagement within the team. A common metric is the eNPS (employee net promoter) score. Very simple and standard HRIS (human resource information system) systems will be able to provide these by departments and/or functions. Often these HRIS systems also have other types of surveys that can go out at regular or random cadences to measure employee happiness.
- Skill Development: Evaluate whether team members have opportunities for growth through training, certifications, or challenging projects. Most software engineers know they live in a world where required skills are constant changing. Be weary of any engineer that doesn’t think they need to continue growing and learning.
6. Collaboration and Communication
SaaS companies thrive on cross-functional collaboration between engineering, product, design, and other departments.
- Stakeholder Satisfaction: Collect feedback from other teams (e.g., product management, customer support) about their experience working with engineering. This can be as simple as asking the sales leadership whether they feel like product listens to their requests for product improvements.
- Transparency: Monitor the engineering team’s ability to communicate progress, challenges, and solutions during regular meetings or updates. The roadmap should be well defined and shared, if not with the whole company, certainly senior leadership. If you can’t get a roadmap or one isn’t maintained, that’s a red flag.
- Dependency Management: Evaluate how effectively the team manages dependencies with other teams to ensure smooth workflows. Hopefully the company has a “smooth rollout” plan for new features and products that’s maintained at a level higher than any one department. Launching a new product or feature to make the biggest impact possible in the marketplace requires significant effort from a lot of teams. If you’re the finance or accounting leader and a new product is launched and no one ever asked whether the new product would impact invoicing or revenue recognition, then there’s a problem with launches.
7. Cost Efficiency
Efficient use of resources is essential for assessing departmental performance.
- Engineering Costs: Track costs related to salaries, tools, infrastructure, and outsourcing. Compare these to industry benchmarks and company budgets. This is often done as a percentage of revenue. Leveraging industry averages is a great way to start, however, with time, those percentages should fall or stabilize.
- Return on Investment (ROI): Evaluate the ROI of engineering efforts by assessing the revenue or customer satisfaction impact of delivered features.
8. Customer Experience
Engineering performance directly affects customer satisfaction with the SaaS product.
- Customer Satisfaction (CSAT): Gather feedback on product reliability, speed, and new features.
- Net Promoter Score (NPS): Monitor how likely users are to recommend your SaaS product.
- Support Tickets: Track the volume of support tickets related to bugs or performance issues. A decreasing trend is a positive indicator.
9. Scalability and Infrastructure
SaaS products must scale effectively as user demand grows. These metrics, unless explicitly measured by the engineering teams, is difficult to implement by the finance or accounting professional.
- Scalability Tests: Conduct regular load tests to ensure the product can handle increasing demand.
- Infrastructure Costs: Monitor and optimize spending on cloud services or hosting infrastructure.
- Performance Under Load: Ensure system performance remains consistent even during peak usage.
10. Alignment with Strategic Goals
The engineering team should contribute directly to the company’s broader objectives.
- Goal Achievement: Evaluate whether the team is meeting OKRs (Objectives and Key Results) or KPIs set at the organizational level. A good dose of common sense helps to keep software development performance in context. Do the products and features contribute to the overall goals of the company? If they contribute to the top and bottom lines then presumably they are. However, knowing the industry and specific market can help immensely when developing an expectation of the true impact of a product or feature.
- Innovation Contributions: Assess how the team is contributing to long-term strategic initiatives, such as entering new markets or enhancing competitive differentiation.
Key Tools for Measurement
- Project Management Tools: Use tools like Jira, Trello, or Asana to track project progress and measure velocity or cycle time.
- Code Analysis Tools: Leverage tools like SonarQube or GitHub Insights for code quality metrics.
- Monitoring Tools: Use New Relic, Datadog, or AWS CloudWatch for uptime, performance, and reliability metrics.
- Feedback Tools: Use survey tools like Culture Amp or Officevibe to assess employee engagement and satisfaction.
Qualitative Indicators
Quantitative metrics are important, but qualitative feedback also matters:
- Team Morale: Engage with engineers regularly to understand their satisfaction and challenges.
- Leadership Effectiveness: Evaluate how well engineering managers and leads support their teams.
- Customer Feedback: Positive feedback from customers about product improvements and reliability reflects well on the engineering team.
Conclusion
A high-performing software engineering department in a company is one that delivers quality software efficiently, contributes to innovation, ensures reliability, and aligns with strategic goals. By tracking these metrics and gathering feedback, you can make informed assessments and support the continuous improvement of your engineering team.
