How We Reduced Our Lead Time to Change by 75%
Most companies want to deliver the best possible experience to their customers. This means releasing new features and bug fixes quickly and efficiently. BAsed on DORA, key metrics to track for engineering teams are lead time to change(LTC) and deployment frequency(DF). Lead time to change(LTC) is the time it takes to implement a code change from the moment it is requested until it is deployed into production. Deployment Frequency(DF) is the number of deployments in a given timeframe.
In the past, our lead time to change was about a month, which was too long. This was frustrating for our customers and for our team. We knew that we needed to improve our lead time to change, but we didn’t know where to start.
Benchmarking
We began by closely tracking our LTC and DF to identify bottlenecks and potential solutions. We selected middlewarehq.com as our choice of measurement tool. After several weeks of integration and observation, we obtained our initial benchmarks: LTC of approximately one month (4 weeks and 3 days) and DF of 3 features per week. On average, it took about a month for any new code change to reach production.
Our long LTC suggests that there are bottlenecks in the delivery process. This could be due to a number of factors, such as a complex or cumbersome change management process, a lack of automated testing, or a shortage of resources.
Our low DF suggests that they are not able to release new features to production very frequently. This could be limiting our ability to respond to customer needs and market changes in a timely manner.
Brainstorming
Having gained a better understanding of our delivery speed, we brainstormed potential improvements. After research and deliberation, we began focusing on the following areas:
- Isolated feature development cycle
- Project management (Jira ticketing)
- Code quality (code reviews)
- CI/CD automation (automation test suite)
On-Demand Feature Environments
On-demand feature environments are short-lived environments that are created on demand and destroyed once they are no longer needed. This type of environment is typically used to test new features or changes to existing features in isolation.
Previously, our team was working on 4–5 different features at a given time and had 3 permanent environments: Production (master branch), Pre-Production (uat branch), and Development (dev branch). Any new code change related to a feature was pushed into the Development environment directly, which caused inefficiencies among QAs when testing the changes. It would take about 2–3 weeks for feature testing, with yet low quality of code.
To address this, we introduced on-demand feature environments. This means that each feature has its own dedicated environment, which is created on demand and destroyed once the feature is complete. This allows QAs to test changes to individual features in isolation, without having to worry about other features that are being developed in parallel.
Git branching strategy for on-demand feature environments
The following Git branching strategy was used to accommodate on-demand feature environments:
- Create a new feature branch for each new feature.
- Commit changes to the feature branch frequently and push them to the remote repository.
- When ready to test all the changes, create an on-demand feature environment from the feature branch.
- Test code changes in the on-demand feature environment and make any necessary changes.
- Once satisfied with the changes, merge the feature branch into the pre-production branch.
- Deploy the changes to production from the pre-production branch.
Benefits of using on-demand feature environments
- Isolation: Keep changes separate, reducing risk and increasing testability.
- Efficiency: Allow devs and testers to work simultaneously, increasing productivity.
- Scalability: Create as many environments as needed, when needed, supporting teams of all sizes.
On-demand feature environments were a valuable addition to our development process. By isolating changes to individual features and improving testing efficiency, on-demand feature environments can lead to a higher quality product and faster development cycles. In our case, we saw a significant reduction in feature testing time from 2–3 weeks to less than a week, with reduction in production issues after the release.
JIRA Ticketing Structure
- JIRA Epic: A JIRA Epic represents a large project or body of work with complete context. It is typically broken down into smaller, more manageable pieces of work, such as JIRA Stories and Tasks.
- JIRA Story: A JIRA Story represents a change in user behavior. It is typically described in a user-friendly format, such as “As a user, I want to be able to log in to the application using my Google account.”
- JIRA Task: A JIRA Task represents a change in the system. It is typically described in a technical format, such as “Add a new login screen to the application that allows users to log in with their Google account.”
- JIRA Bug: A JIRA Bug represents an abnormality in a JIRA Story. It is typically described in a technical format, such as “The login screen is not displaying correctly on mobile devices.”
Relationship between JIRA Issues and Git Branches
Our practice was to create a separate Git branch for each JIRA Story and Bug. This allows developers to work on multiple issues in parallel without interfering with each other’s work.
Once a JIRA Story or Bug is complete, the developer will merge the respective Git branch back into the main feature branch. This ensures that all changes are integrated and tested before they are released to pre-production(UAT), followed by production release.
Example
Imagine that a team is working on a new feature for their web application that allows users to log in with their Google account.
- The team would first create a JIRA Epic for the new feature.
- This Epic would contain all of the individual JIRA Stories and Tasks that need to be completed in order to implement the feature.
- For each JIRA Story, the team would create a separate Git branch. This would allow each developer to work on their assigned Story without interfering with each other’s work.
- Once a Story is complete, the developer would merge the Git branch back into the main development branch. The team would then review the changes and test the new feature.
- Once the team is satisfied with the new feature, they would release it to pre-production, and follow it up with production.
Benefits of using separate Git branches for each JIRA Issue
- Isolates work, enabling parallel development and minimizing disruption.
- Supports experimentation and collaboration.
- Provides a detailed audit trail.
Using separate Git branches for each JIRA Issue proved to be valuable in terms of clarity of responsibilities for individual roles, thus leading to increased collaboration and improved quality of code.
Code Review Guidelines
Code reviews, also known as peer reviews, are a software development process in which one or more people review the source code of a program to identify and fix defects, performance issues, stylistic problems, and other issues. Code reviews can be done at any stage of the development process, but they are typically performed after a developer has completed a new feature or bug fix.
One of the key challenges we faced was the lack of common practices for implementing code changes. This was a major contributor to our slower development time and higher rate of production issues. Therefore, we developed a set of coding guidelines to address this issue. The following is a summary of these guidelines
Code Review Guidelines
- Correctness: Ensure the code does what it is intended to do and handles all possible cases correctly. Use unit tests to verify correctness.
- Completeness: Handle all possible cases, including edge and corner cases. Use error handling to gracefully handle unexpected situations.
- Efficiency: Avoid unnecessary loops and recursion, use efficient data structures and algorithms, and profile performance to identify and fix bottlenecks.
- Readability: Use clear and concise variable and function names, comments to explain complex code, and the team’s code style guide.
- Security: Check for common security vulnerabilities and use security best practices and a scanner to scan the code.
- Style: Follow the team’s code style guide and use consistent formatting, indentation, and whitespace to write code that is easy to read and maintain.
In addition to this, we reduced the size of changes and started using smaller batch sizes when deploying code to production. Smaller changes are easier to review and deploy, which lead to reduction in lead time to change.
By following the code review guidelines, we reduced our time to production from first commit to PR to less than a week, compared to 2 weeks before the guidelines were implemented. However, our rework increased from less than an hour to about 6 hours on average, which suggests that:
- The code review guidelines are helping us to identify and fix errors earlier in the development process.
- The rework is necessary to ensure that the code meets our high standards of quality.
UI Automation
UI automation is the process of automating the testing of a software application’s user interface (UI). This is done by writing scripts that interact with the UI elements of the application in the same way that a user would. The scripts can be run repeatedly to ensure that the UI continues to function as expected.
Benefits of UI Automation
- Increased speed and efficiency: Scripts can run repeatedly without human input, reducing time and effort.
- Improved accuracy: Deterministic scripts eliminate human error, improving accuracy.
- Increased coverage: Scripts can automate complex UI interactions, increasing test coverage.
One of the improvements we made was to implement a CI/CD pipeline with an automation test suite. We implemented UI automation for our critical flows. This led to reduction of regression testing time from 5 days to 1.5 days, and increased confidence levels in feature release.
Conclusion
By monitoring our lead time to change (LTC) over time and implementing best practices, we have been able to reduce our LTC from about a month to less than a week. This has also led to an increase in our deployment frequency (DF) from 3 deployments per week to 8 deployments per week. As a result, we can now release new features and bug fixes to our customers more quickly and frequently.
This improvement has resulted in a number of benefits, including:
- Faster time to market for new features and bug fixes
- Improved customer satisfaction
- Increased ability to respond to market changes and customer feedback
- Reduced risk of errors and regressions
- More efficient and streamlined development process
We are committed to continuing to invest in our development process and to finding new ways to increase our speed of delivery.
Any suggestions or thoughts, let me know:
LinkedIn + Insta + Twitter + Medium
