X-Flow: A Powerful Cross-Breed Between Version Management Workflows

Welcome to the working document for X-Flow, the modern DevOps workflow. In encountering various version management workflows I have found many trends as the DevOps environment progresses. Many progressive teams have adopted certain hybrid workflows and some new workflows benefit from deployment tools that didn’t exist in previous generations. This is really just a collection of some of these practices with additional explicit guidance for various teams in the development lifecycle. Much credit is due GitHub Flow & Trunk Based Development as the core of what I expand on here with a more holistic view of the engineering workflow (not just the Version Management piece). So if someone else has already named & codified what I delineate here, let me know.

Photo by Firdouss Ross on Unsplash

What Is X-Flow

X-Flow or “Cross Flow” is a form of the Trunk Based Development version control management practice and is essentially GitHub Flow but extended with some extra guidance around deployment. This is less a purely “version management practice” and more of a holistic approach towards the engineering lifecycle from when code hits the remote repository on. It is adjusted to allow for different stages of the development journey including a version which will feel a bit more comfortable for devs coming from Git Flow. “Mid-Flow” is the true Crossover but X-Flow would not be complete without the other evolutions of the process & tooling.

There are three versions of X-Flow: Start-Flow, Mid-Flow, and Full-Flow. Start-Flow is the base for the other two flows, so it is important to become familiar with it. From there we just have targeted modifications: Mid-Flow for “slow approval” organizations & organizations coming from Git Flow, and Full-Flow for organizations who are capable of implementing complex Infrastructure as Code (IaC) which dynamically creates and destroys cloud resources. Get to know X-Flow with Start-Flow and then look into either of the modifications to see if it better fits your needs & abilities.

Start-Flow

Start-Flow looks basically like GitHub Flow. It starts with the “main” branch which comes with repository creation. This is called the trunk (you can even name it to trunk if you like). This branch should NEVER have changes made or merged directly to it, except through a Pull Request (PR). Work is done in clearly named feature branches with an agreed upon convention (my personal convention can be found below). These feature branches should be relatively small and incremental to avoid large amounts of un-synchronized & un-deployed code.

When a feature is considered ready for deployment the changes should be proposed through a PR against the trunk, just like GitHub Flow. To be approved, the code should be reviewed for quality & tested in a live deployed environment (in X-Flow, changes should NEVER be approved & merged if they have not been shown as stable in a live environment which is reasonably representative of production — though it need not be resourced to the same level as production). This PR deployment process is core to the functional innovation of X-Flow.

In X-Flow ALL changes which are PRd against the trunk are automatically deployed to a live development (dev) environment. Start-Flow assumes that highly sophisticated (IaC) is not implemented. With this assumption it might also be assumed that there are a limited number of developers making changes to a given code base at a time. Because of this, Start-Flow all but REQUIRES that only one PR is open at a given time. This is important for ensuring that there is no confusion on what is in your singular dev environment and always keeps development ahead of staging & production. A team may choose to open multiple PRs at a time without the additions in Mid-Flow or Full-Flow, but they are then responsible for ensuring their process creates clarity on what is in the dev environment, and in general this practice is advised against (and why the other X-Flows exist).

For Start-Flow the creation of a PR and the associated code review is an “andon” event: a work stoppage event. If another developer is available to review the PR, their first priority is to get the change through to trunk. Process-wise, this ensures that only one PR is ever open. Another key aspect is the importance of Acceptance Testing in closing a PR. X-Flow ensures that a live environment is available for a product owner (PO) to review before a change even makes it to trunk. So, particularly when the product owner is positioned to keep a feature from production approval, they should also treat these PR events as their highest priority. This ensures that features continue flowing to trunk and that nothing sitting in trunk blocks other features from production deployment (in the event unapproved features made it to trunk). If this is a concern and the PO is not available, consider using the below “On-Hold PR State” strategy to ensure “ready to go” features make it through but only approved changes are in Trunk.

Once a PR has been approved, it should be merged to trunk. Another automated deploy should be set to trigger on this merge to the trunk. The environment associated with trunk is meant to represent the potential production environment, often called the staging (stg) environment. If there is any automated load testing, it should also be run on this automation trigger. At this point the code is double verified and ready to go live. However, particularly in customer facing apps, just because a feature is ready for prime time doesn’t mean the business is ready for users to have access to it. This is why the production (prd) environment is triggered through a tagged release (you may choose to have production deploy in this step following any load testing, though generally this won’t be done without a comprehensive feature flagging system). X-Flow does not require or give guidance on the use of feature flagging, but it is a strategy which also enhances throughput of features while providing targeted delivery to users. For best results of a wholistic DevOps workflow, the practice is encouraged.

The final step of X-Flow is the production deploy. Deployment to the prd environment is done using a release tag which should follow Semantic Versioning conventions (though SemVer may be most beneficial for public libraries, it is a standard worth proliferating across all software releases). Depending on the team structure releases may be restricted to a subset of individuals, the product owner only, or even be driven by an API release connected to some external management process. Regardless, all traditional releases will only be made against the trunk. The exception, which will be covered further down, is for a Hot Fix.

As mentioned, an important feature of X-Flow is the use of PRs to trigger the dev deployments. This is empowered by modern CI triggers which will run every time an updated commit is added to the open PR. This allows for adjustments to be made & reviewed on the fly in the proposed feature branches. Once the changes in the PR are cleared they will be merged into the trunk. This PR driven dev deployment also allows versatility to adjust for various project management workflows, while using the same deployment infrastructure and general repository management rules. That is where Mid-Flow stands out.

Mid-Flow

Mid-flow is for a few team archetypes. Those who have reached a velocity & team size which requires multiple open PRs but without the “Full-Flow infrastructure” to back it up. Teams who are used to Git Flow and interacting primarily with a “dev” branch. Finally, teams who have a very slow feature approval process, or want to have a clear grouping of their application “stages” such as grouping features by sprint.

The key difference of mid flow is the use of a “collector” or “mid” branch in-between feature branches and the trunk. That is to say, this mid-branch is created off of trunk which is also the only branch to PR against trunk. All feature changes are branched off of this mid-branch and PRs for features are made to merge back into this mid-branch. Once these features are merged in, it will trigger the automated deploy to the dev environment. This allows for various developers to be working on features which are opened and awaiting approval, but instead of against trunk it is against the mid-branch (keeping them from impacting deployments). IT IS IMPORTANT TO NOTE: in this situation the feature PRs are not being put into a live environment, as the build is only triggered on changes to the mid-branch (which has the only PR against trunk). This works fine for code reviews where a dev can run a local environment but DOES NOT allow for live acceptance testing of individual feature changes. Instead acceptance testing must happen on the mix of whatever made it into the mid-branch, creating the opportunity for features to convolute another feature’s progress to deployment. However, approval still happens before trunk, ensuring it remains stable & accepted.

For those familiar with Git Flow, this mid-branch can be a long-standing dev branch, mimicking the commit structure of Git Flow. If you are looking for the discrete collection of features into “stages”, the PRs themselves may represent these stages (a new PR created for each new sprint or feature set). Alternatively, a completely new branch may be created for each new stage or sprint, cycling in and out regularly. Mid-flow provides the greatest level of flexibility to accommodate project management workflows while still using sophisticated deployment strategies. Nothing needs to be changed from the Start-Flow CI/CD and a team could switch back and forth between the two ad nauseam without it impacting any other operational infrastructure, so long as there is an agreement between participating members before the process change is made.

Full-Flow

Full-Flow is the apex of X-Flow. It is built for large teams with large projects and modern cloud infrastructure that can be programmatically built & destroyed. Just like Mid-Flow, Full-Flow is only changed in the way you interact with the commits PRd against trunk (dev commits). In Start-Flow we expressly avoid running multiple PRs against trunk, but in high velocity and large team environments this may just be impossible (even if it is ideal). Unlike Mid-Flow, teams that choose Full-Flow are committing to a level of IaC & automated cloud resource creation that goes beyond the base infrastructure which is set up with Start-Flow.

In Full-Flow instead of trunk PRs triggering deployment to the same dev resources, a whole new set of the necessary coupled resources get created for each PR. For instance, if the codebase is for a backend which is coupled to a particular ORM performing Database Migrations, this would require a new container instance hosting the application as well as a new database added to your dev database server instance and the appropriate migrations run. It would not require a dedicated frontend (assuming the discrete apps are decoupled and associated by a URL in configuration variables), but a team may choose to do so for their own purposes.

Because each PR gets its own set of resources, there is no longer uncertainty about “what is in the dev environment”. Instead a “PR unique URL” is provided to testers to see the live version of whatever they are testing. This sophistication is powered by IaC like Terraform & build triggers like GCP’s which offer the PR number to the build tool for creating associated resources. In this situation (for a PR numbered 2) we may then use the likes of Terraform to provision a database called pr-2-postgres, create a Google Cloud Run instance called pr-2-run, connect the two, and return the unique URL “dev-api-pr-2.example.com” for the run instance to be referenced by Postman or a frontend (which you may also create on the fly as dev-app-api-pr-2.example.com).

Once a PR has been tested it is merged into the trunk and all else operates as usual. However, in Full-Flow be extra vigilant to “back-merge” trunk onto the given PR branch and do some quick testing before the final merge. This ensures that no side-effects result from closely timed merges of various feature branches.

Another important trigger change that comes with Full-Flow is what happens on the PR merge. After something has been merged to staging, the PR environment is no longer needed for testing. At this point our build triggers should run infrastructure automations to delete the resources created for that given PR.

This of course leaves the question of “when there are no PRs, does this leave us with no dev environment? What if one app in the dev environment needs to reference another and doesn’t have a specific PR?” I encourage keeping a “core dev” environment which is deployed on the same trigger as stg (meaning the code is in line with staging). Basically if you switch to Full-Flow from the others, you would update the dev build event’s trigger from “on PR” to “on merge”, then create a new “on PR’ trigger which runs your IaC to handle all the PR specific environments.

Obviously Full-Flow is considerably more involved. However, by using automated infrastructure we are able to mitigate concerns around environment confusion which come with the day-to-day realities of using PR driven deploys, while sticking to the “feature direct to trunk” approach which eliminates friction in high velocity projects. It is also the single best way to keep features from blocking each other’s path to production. And at the end of the day, the core process is still the same as the other evolutions of X-Flow: trunk is the fundamental source of truth, PRs are used to merge changes into trunk and deploy dev environments, changes to trunk trigger stg deployments, and prd is driven by release tags. X-flow is all about this clear coupling between version management & deployment flow. But what about HotFixes?

HotFixes In X-Flow

For all versions of X-flow, the process is basically the same for HotFixes. A HotFix assumes that the standard flow of features can’t address a certain issue: For instance production is behind the trunk so trunk can’t be used to base a hotfix from for deploying back into production (otherwise a hotfix could just follow the traditional feature deployment process, being escalated above all other features). Though feature flagging can remove many of these situations, it is still important to consider a separate strategy for deploying fixes which helps keep your repository & branches clean.

The first step in deploying a HotFix is to determine the commit which is currently deployed to production by looking at the release tag’s commit version. A branch should then be created directly from that commit, following a HotFix naming convention (see mine below). Once changes are made and pushed into version control they should be PRd against trunk to deploy to dev. As part of this, unless you are using Full-Flow, if there is an existing PR it should be labeled “on-hold” and closed, only reopened after the fix resolution (see On-Hold PR State). Now that we have our fix applied to the same code which is in production and PRd, it can be tested in the dev environment.

Following successful testing a semantically versioned release tag should be made directly against the HotFix branch triggering deployment for prd. By using tagging in this way we are not tied to all production deploys going out from trunk, freeing us to use the same build trigger for hotfixes.

Once a hotfix is deployed we can follow our traditional flow, merging trunk onto our branch, ensuring changes work properly then merge back into the trunk, triggering stg deploy. Finally make sure to re-open and update any PRd branches (of course verifying that the hotfix didn’t create side-effects when mixed back in). And that’s it. Hot fixes don’t need a special deployment trigger or to be treated all that different from any other changes. The only difference is that the prd deployment happens directly from the hotfix branch instead of trunk, ensuring the only code added to prd is the fix.

Prerequisite & Assumptions of X-Flow

It is assumed that anyone seeking to implement X-Flow is doing so as part of efforts to create streamlined CI/CD. As with any Trunk Based Development, the version control is only half of a whole. Without automatic deployments to the dev & stg environments, TBD falls on its face. That is because it is built on small feature additions requiring a tight feedback cycle. Though the mid-flow evolution creates a layer of buffer, it is still expected that automated deployments create tight synchronization.

It is also assumed that individuals implementing X-Flow are using the Git versioning system. While other systems may provide similar capabilities, the flow was developed in context of Git. It is also assumed that the specific version management tool your team uses has the ability to connect to a deployment system which can watch for the various DevOps triggers we use for the rest of the development lifecycle.

On-Hold PR State

For Start-Flow & Mid-Flow it may be important to use the following strategy to create alignment in your dev environment & code base. Because any branch PRd against trunk triggers deployment to dev, it is important to control what PRs are open and allowed to have changes trigger. If we don’t tightly control PRs we can easily get into a situation where changes were made in one branch, PRd & deployed, but before testing got deployed overtop of by another set of PRd changes, potentially resulting in inaccurate approval.

In situations where multiple changes are in process we want to use an “on-hold” state to push changes through if they can’t follow a proper FIFO queuing (in the event of HotFixes or slow approvals on a given PR). The process is as follows:

1. Tag the current open PR as On-Hold
2. Close the On-Hold PR (an unmerged closed PR should be reopenable)
3. Only then create the new PR (if accidentally already created, make sure to trigger a new deploy)
4. Run & test the open PR until satisfied, then merge for deploy, closing the PR
5. Re-open the on-hold PR, un-tag & make sure to trigger a new deploy

By following this pattern you can ensure that the dev environment is clearly associated with your codebase while flexibly shifting between work that may take longer to get through the testing phase. More importantly, we are able to keep features in the testing phase for as long as necessary to make sure they don’t make it into trunk prematurely and pollute the production ready codebase. Obviously this is not necessary in Full-flow with discrete environments, but it is always an option if programmatic infrastructure fails you for some reason.

The CI/CD of X-Flow

X-Flow pivots on the quality of deployment automation. These are the general base suggested triggers to implement if you are starting out:

PR Against Trunk:

A trigger should be created to start an automated deployment to your development environment on every commit to a branch which has a PR against Trunk. This is your dev environment.

Merge To Trunk:

Create a trigger which runs on every merge to Trunk. This will deploy to your stg environment from Trunk and should represent your production ready code, used for load & user testing.

Release Tag:

Production (prd) should be deployed using a trigger on creation of a release tag. These tags can follow any convention and be filtered in various ways, though semantic version is advised.

Branch Naming:

Everyone has their opinions about naming and conventions, this is just a personal preference for naming that you are welcome to adopt or follow your own.

My suggested branch naming

Other Considerations & Good Hygiene

X-Flow is encouraged to be used in decoupled systems that generally follow the 12-factor app. When developing in this environment it is important that changes are backwards compatible across the various levels/apps of the stack, or that discrete environments are spun up which pair with the given changes being tested. This generally anticipates backend changes being done ahead of frontend changes, testing backend with a tool like Postman.

When implementing “Full-Flow” with discrete apps dynamically spun up and deployed per PR, you might consider using an “environment injection” strategy for the frontend when testing backend changes. That is, allowing for the fronted to receive from the tester, the specific URL to access the backend from. In a web-app the simplest way to do this is with a query string as such: dev-frontend.example.com?api_url=dev-api-pr-2.examplehost.com which would then update the stored value used for the API url, ideally as part of pre-routing.

As with any flow, it is important to do regular pulls between merging branches to ensure they are in sync. That is to say, before a PR is merged to trunk you will need to pull trunk onto the merging branch to ensure code synchronization (particularly in hot-fix situations). Particularly if the trunk is “ahead” (has new changes) of the PR branch, it is also important to test those changes in the dev environment before blindly merging them back to trunk, just in case the combination creates unexpected side-effects. As important: before creating a new PR, make sure that trunk is merged onto the branch to keep dev from getting behind stg and to avoid retesting before closing a PR.

Also encouraged, though dependent on team culture, is the practice of “paying it forward” when it comes to PRs. If you have successfully closed a PR and a team-member is awaiting team-member review on their PR, help them close out their PR before forging ahead on something new. This type of team process can help velocity by promoting shared responsibility of reviews assuming your team composition allows for it. Obviously your mileage will vary.

FAQ:

What happens when a branch sits unmerged for several cycles of other changes being brought into trunk?

• This is a common problem across all development workflows. Inevitably this will require merging all the deployed changes back onto the feature branch. It is the responsibility of the developer making the changes to do this work. But, hopefully by working in small increments these situations are mitigated.

When working in “Start-Flow” with multiple features in queue to be PRd, how do you choose the order which PRs are created and reviewed since only one PR is allowed at a time?

• In this situation it is suggested to lean on your feature workflow, using a feature board a column can be created for “Awaiting PR”, where the top feature is currently in PR, and the queue for what PR comes next is indicated by the subsequent features in the column.

What do you do about configuration data within an environment?

• Generally configuration data should be driven by the environment. In these situations if a configuration change is part of the proposed changes, extra diligence should be taken in switching between PRs (as it may be easy to forget the configuration update step). A “config changes” tag may also be used on PRs with accompanying configuration changes to remind developers of the importance of verifying environment before & after deploys. In Full-flow you may also decide to use an “environment configuration table” which your automations reference for determining various configurations for the various environments.

Final Thoughts:

As is software more generally, DevOps is an evolving beast. My previous agency spent a lot of time working to implement the best new practices and also innovate in strategic ways. While some of our approaches may have been cutting edge, the true bleeding edge may have eluded our grasp. This is a working document intended to grow from the thoughts & opinions of the greater development community. Yes, opinions are welcome, I truly want to help evolve the way we approach engineering process & tooling… opinions are part of that. Please share any resources you may think to be relevant, items that may need clarity, and let me know of any edge cases or “software standards” X-Flow may have missed. As we continue to evolve the field we must keep an open mind and look to what could be best, not what was best. Hopefully I can help some of you in this endeavor, as I have been and am sure to be helped in this endeavor moving forward.