Hotfix pipelines are like an extended warranty; they make sense in principle, but in reality, they aren’t necessary. And just like an extended warranty, you spend a lot of money (and time) building a hotfix pipeline that you should use only rarely. This article will first walk through the fundamental flaws of hotfix pipelines. Then it will explain why it is better to focus on improving the standard Production deployment pipeline.
Defining hotfix pipelines
Hotfix pipelines are a "fast lane" to the normal Production deployment pipeline. Their purpose is to skip specific steps to get changes into Production as soon as possible. Typically, they are:
- Production deployment pipeline: Development → Test → Staging → Production
- Hotfix pipeline: Staging → Production
They exist because of this scenario:
- A critical bug appears in Production hours or days after a deployment.
- A rollback is impossible, as that will cause more problems than it solves.
- There is work in flight in lower environments.
- The hotfix pipeline skips the lower environments.
The core idea behind hotfix pipelines is that they allow normal work to continue while providing a path to fix a critical bug as soon as possible. Someone, either within the development team or the business, determines that a bug is important enough to fix and can skip use the hotfix pipeline.
Hotfixes in the real world
From experience, hotfix pipelines are created in response to a suboptimal Production deployment pipeline. There is a step, or several steps, that either add little value or take so much time that everyone is fine skipping them when certain conditions are met.
I experienced a suboptimal process before joining Octopus Deploy. Before Octopus Deploy, I was a software engineer for almost 15 years. This was a deployment process at a company I worked for before we fully automated it (using Octopus Deploy, of course!).
- Four environments: Development, QA, Integration, and Production.
- Shared database for the same application for all in-flight work.
- The build server only monitored the main branch.
- The build server automatically deployed to Development and QA.
- Deploying to Integration and Production used different tooling and was a mix of manual and automated.
As you can see, deploying to Integration and Production was completely different from Development and QA. Some notable new steps include placing code build artifacts in "hot folders" to automatically deploy code, placing database delta scripts in designated network folders, and hand-typing instructions for the Web Admins and DBAs.
Deployments to Production took hours due to missing database and configuration changes, and occasional failures to update a server with the latest code. Because deployments to Production took hours and were error-prone, the application I was responsible for deployed new functionality and most bug fixes to Production once a quarter. Those deployments were always off-hours.
Integration was slightly better; we could deploy to that in the middle of the day, but it was still just as error-prone. We didn’t want to deal with that headache. Integration was often updated only days before a Production deployment. As a result, Integration and Production were almost always running the same version. This created a clear hotfix path for us.
Generally, but not always, hotfixes could be placed into one of four buckets. In each case, we could run the proposed fix in Integration before Production.
- Fix the issue by changing a record in the database to "trick" the code into the right path.
- Solve the problem by making a change to the database schema or a stored procedure. Sometimes it was as simple as adding an index.
- Make a configuration change, either in a web.config file or the operating system host.
- If it required a code fix, we could configure the build server to create the build artifacts but not deploy to Development and QA. We’d copy the artifacts to the "hot folders" for a deployment to Integration.
The common workflow when a bug was reported was for the on-call engineer to triage it. Once they isolated the root cause, they’d pair with a DBA or Web Admin until the issue was resolved. While issues were resolved, it led to other problems.
- Database schema hotfix changes were rarely "copied down" to lower environments. The delta in the database schema between Dev/QA and Integration/Production was significant.
- Configuration changes sometimes didn’t make it into version control. The next deployment to Integration and Production would often overwrite the hotfix, requiring another hotfix.
- Because of the once-a-quarter releases, finding an appropriate point in time in version control that just had what was in Integration/Production took a lot of effort.
Impact of automation on hotfix pipelines
Automating a terrible process doesn’t make it any less terrible. The core problems must be addressed. Our core problem was two different processes to deploy software from Development through to Production. Those two processes were required because the build server was used for deployments and shared application database. The build server lacked the appropriate RBAC and configuration controls to satisfy security and audit requirements. The shared database encouraged the merging of unfinished code.
Using a combination of local database development, git, Redgate's tooling, Octopus Deploy, and Azure DevOps (or Visual Studio Team Services as it was called back then), we rebuilt the entire deployment pipeline.
- All changes, code, and database are made in a branch.
- Changes are merged to the main using a Pull Request.
- Azure DevOps / VSTS builds the code and packages database changes.
- Octopus + Redgate deploys those build artifacts to Development.
- Automated Verification, then Octopus + Redgate, promotes those artifacts to QA.
- QA verifies.
- Promote those artifacts to Integration using Octopus + Redgate.
- Final sign-off.
- Promote those artifacts to Production using Octopus + Redgate.
Rebuilding the deployment pipeline had numerous benefits.
- Consistent deployments: No more surprises when deploying to Integration and Production, as it was the same process for Development and QA
- Eliminated post-deployment emergency fixes: Automation stopped the dumb mistakes that led to many hotfixes.
- Reduced manual work: No one had to copy files to specific network folders and write up instructions.
- Increased release cadence: Production deployments increased from once a quarter to once a week.
- Smaller changeset: Instead of dozens or 100s of changes, each deployment had fewer than a dozen. That meant there was a lower chance that something could go wrong.
The most surprising thing to us was the more tolerant users. When it took two or three months to get a bug fix into production, users would classify most bugs as high/must-fix because they couldn’t wait that long. With the new pipeline, unless the bug was critical, most users were willing to wait a day or two for a fix. With smaller change sets, the number of critical bugs dropped significantly.
While we were deploying once a week, we still had major features that would take over a month to develop. All that work would occur in a branch. When we merged in those major changes, it would sometimes take a week or two to fully test it (our actual average was deploying every 10 days). During that time, there was "no clear path to production" for hotfixes.
- For major features, we’d often do a few deployments to Integration as a final test with Production-like data.
- Once major features were pushed to Integration, there was no easy way to roll back those changes.
- We used a Service-Oriented Architecture (SOA) to achieve loose coupling, but there was still some coupling. Whenever anyone rolled back Integration to match Production, other applications would break because they were expecting a specific version.
Because we knew there was a chance of bugs after deploying a major feature to Production, we would implement a merge freeze, except for critical bugs, for a week after the Production deployment.
The merge freeze was the best solution we could come up with at the time. We looked into creating a hotfix pipeline, but with deployments to Production occurring once a week (or so), we kept running into the same roadblocks:
- How do we qualify a bug to justify using the hotfix pipeline?
- What if a pending change is in Integration?
- How do we configure the build server to tell Octopus to push to Integration instead of Development?
- How do we know which branch would be in Production when another bug occurred with a hotfix pipeline?
The decision tree of if/then/elseif/else for when to use a hotfix pipeline became extremely confusing. In addition, fewer critical bugs were released. The work to create a hotfix pipeline for my team was de-prioritized. Until I left, I never saw that becoming a priority. Every once in a while, when we had a merge freeze, someone would bring it up. But it was more of a passing comment. Not a dictate to make a change.
Hotfixes don’t fit in modern software delivery
The story from above took place over 10 years ago. It involved deploying .NET applications to static environments hosted by Windows Servers. There were many Continuous Delivery techniques we didn’t implement at the time. In addition, many technologies and techniques have been introduced.
Below are the core principles every software delivery pipeline should follow. See Continuous Delivery and Achieving Continuous Delivery with TPF, and Trunk-Based Development for more details.
- The main branch must always be in a deployable state.
- Create the build artifacts from the main branch once and promote them through the necessary environments to production.
- How you deploy to Production should be exactly the same as how you deploy to Development, Testing/QA, and Staging/Integration.
- Automate as much testing as possible, including unit testing, integration testing, and post-deployment smoke and soak tests.
- Store environment configuration in version control and use that to keep all environments similar. Compute resources and external access can differ by environment.
- Separate deploying new code from releasing new functionality by using feature flags.
With those core principles in mind, the delivery pipeline is the following:
- Make any changes in a short-lived branch.
- Checking in changes to a branch creates a pre-release artifact and deploys to an ephemeral environment or a static Development environment.
- Merge those changes into the main branch via a pull request.
- The pull request should also be verified on an ephemeral environment or a static PR environment.
- After merging into the main branch, create the release artifacts.
- Promote those release artifacts through any static testing environments (Test, QA, Staging, Pre-Production, etc.) to Production.
- Once a feature is ready, enable the new functionality for a subset of users in Production via feature flags. Start with internal teams, then slowly add users until you have enabled it for all users.
:::figure
:img{ src="/blog/img/hotfix-myth/branching-diagram-with-ephemeral-environments.png" alt="Diagram demonstrating when ephemeral environments will be used in a trunk-based or GitHub Flow based branching strategy" loading="lazy" }
:::
With the appropriate guardrails around the main Production deployment pipeline, a hotfix pipeline for just Staging → Production raises a lot of questions.
- Branching and Deploying
- Will the hotfix branch use an ephemeral environment for testing before going to Staging?
- How will the build server know to skip the Test environment and move to Staging → Production?
- What will the version number be for the hotfix release? If Production is 2026.8.1, does that mean the hotfix is 2026.8.1-Hotfix, 2026.8.1.1, or 2026.8.1.1-hotfix?
- The main branch is supposed to represent production. How will the appropriate hotfix be communicated to the rest of the engineering team?
- When will the hotfix changes merge into the main branch? How much of a delta is there between what is in the main branch and production? Can the fix even be merged into the main branch without serious modifications?
- Testing and Risk
- What is preventing the main branch from being deployed to Production?
- If there is new functionality that’ll likely have many edge cases and potentially show-stopping bugs, why wasn’t it behind a feature toggle?
- Were there changes already in Staging that were overwritten by the hotfix? Will that impact other teams or applications?
- What steps and tests are being skipped in the Test environment?
- How much time is really being saved by skipping the Test environment?
- What if the hotfix requires a hotfix? How long is it acceptable to block the normal pipeline from deploying to Staging → Production?
- How often is the hotfix pipeline tested and verified?
Hotfix pipelines no longer make sense
All of the challenges listed above are solvable with enough time and money. But to quote Ian Malcolm from Jurassic Park, "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should."
To put it bluntly:
- Is it worth spending time to work through all those issues to create a hotfix pipeline?
- How often do you need to push a hotfix?
- If it is a regular occurrence, is the hotfix process masking a suboptimal process like the one I described earlier?
The time spent creating a hotfix pipeline is better spent making the primary software delivery pipeline as efficient as possible. A good goal is to take less than an hour from pull request acceptance to being ready to deploy to Production. That includes builds, testing, linting, scanning, deploying to lower environments, and verification.
Achieving that requires addressing some hard problems:
- If the main branch is regularly in an undeployable state, what testing and verification should be moved earlier in the pipeline to ensure it is always in a deployable state?
- Is the branching strategy Trunk Based Development or GitHub Flow (not to be confused with GitFlow)? If not, why not? Even the creator of GitFlow has said not to use it for most applications.
- Showstopping bugs, ones that typically require a hotfix, are typically the result of new features and functionality. How can Feature Flags be introduced into the pipeline to separate deploying new code from releasing new features?
- If manual review and approval processes are the primary bottleneck, which steps in that process are prime candidates for automation to speed up approvals?
The primary advantage of focusing on the items above is that they have a net positive for any change. New features, security patches, and bug fixes will be deployable faster.
**Disclaimer:*- The one hour is a goal to aim for. It isn’t a hard rule. One hour for some applications, like monoliths, is impossible. That doesn’t mean you shouldn’t try. Small improvements add up over time. Improving a monolith's pipeline from one day to two hours is a huge accomplishment.
Configuring Octopus Deploy
By this point, you might be asking yourself, how does this impact my configuration of Octopus Deploy?
Unfortunately, lifecycles (and channels) are among the most misconfigured constructs within Octopus Deploy. To make onboarding easier, the default lifecycle is built using conventions. If you were to create Development, Test, Staging, and Production environments, the default lifecycle automatically becomes:
- Development → Test → Staging → Production
That default lifecycle encourages bad behavior, necessitating a hotfix lifecycle.
- It doesn’t represent how developers work in branches. Because it includes Production, it has to represent the main branch. But it also includes Development. To get feedback, developers are forced to use their local machine or merge unfinished changes into main.
- With unfinished changes in the main branch, it is currently undeployable. It could remain there for days and sometimes weeks.
- Enforcing SemVer versioning rules becomes nearly impossible. For example, releases that are not ready for Production typically receive a pre-release tag because the lifecycle includes Development and Production; that rule cannot exist.
- There isn’t a clear path to push a fix to Production. To prevent that from happening, teams will implement merge freezes for a period of time after a major release (like I did at a previous job). Or create a hotfix lifecycle that skips Development and Test.
The root cause of that is having the static Development environment included in the same lifecycle as Production. Development is for unfinished changes, Production is for finished changes. A Production lifecycle must never include Development.
If a static Development environment is required, my recommended lifecycles are:
- Default: Development
- Release: Test → Staging → Production
:::figure
:img{ src="/blog/img/hotfix-myth/recommended-octopus-lifecycles.png" alt="Screenshot of Octopus Deploy interface showing the recommended lifecycles of default and release." loading="lazy" }
:::
**Disclaimer:*- The release lifecycle should include all static testing environments required to reach Production. You might only need Test → Production, or Staging → Production. I included Test → Staging → Production because, as an industry, we have coalesced around four environments.
The subsequent Project Channels are:
- Default (uses the default lifecycle or an ephemeral environment): build artifacts require a pre-release tag and can only be created from non-main branches.
- Release (uses release lifecycle): build artifacts cannot have a pre-release tag and can only come from the main branch.
:::figure
:img{ src="/blog/img/hotfix-myth/recommended-octopus-channels.png" alt="Screenshot of Octopus Deploy interface showing the recommended default and release channels for a specific project." loading="lazy" }
:::
GitHub Actions (or really any build server) doesn’t make dynamically selecting channels based on branches any easier. They require using a hard-to-decipher if/then/else command in the build definition. For example, ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_DEFAULT_CHANNEL }}.
Running multiple versions in Production
Occasionally, REST APIs and other backend services must run multiple versions in Production for backward compatibility. For the recommendations below, my example application has three versions: v1.x, v2.x, and v3.x.
- The main branch represents the latest version (v3.x)
- Separate branches for each version (v1.x and v2.x)
- Each version branch is treated like a "trunk"
- Changes are made in short-lived branches that were branched off the version branch.
- Merging into those version branches requires a pull request.
Within Octopus, you’ll only need one lifecycle:
- Release: Test → Staging → Production
But the Project will have four Channels:
- Default
- Uses an ephemeral environment
- Build artifacts require a pre-release tag and can only come from non-version or main branches.
- vCurrent
- Uses release lifecycle
- Build artifacts cannot have a pre-release tag
- Build artifacts must come from the main branch
- Build artifacts version must be <= 3.x
- V2
- Uses release lifecycle
- Build artifacts cannot have a pre-release tag
- Build artifacts must come from the v2 branch
- Build artifacts version must be between 2 and 2.999999
- V1
- Uses release lifecycle
- Build artifacts cannot have a pre-release tag
- Build artifacts must come from the v1 branch
- Build artifacts version must be between 1 and 1.999999
Ephemeral environments make this significantly easier, as you can spin up a sandbox for a change for any version and verify it before merging into the appropriate branch. If you cannot use ephemeral environments, I’d recommend setting up a couple of static development environments and configuring a lifecycle that lets you deploy to any of them. The downside is that a person must determine which static development environment to use.
Conclusion
A Production incident is not the time to improvise a deployment pipeline. Steps shouldn’t be skipped to "go faster." All too often, a "simple change" that isn’t properly vetted causes a bigger issue. But that is essentially what a hotfix pipeline is designed to do. It skips important steps in the normal Production deployment pipeline to save time. To resolve a Production incident, you want a well-tested and well-used pipeline, so you know you aren’t introducing even more risk. The time required to create and improve a hotfix pipeline is better spent improving the Production deployment pipeline. Once the Production deployment pipeline takes less than an hour to be ready for deployment to Production, the need for a hotfix pipeline will be all but eliminated.








