GitHub merge queue is generally available

Merge queues are, as the name implies, queues for pull requests/merges. They're kinda useless if your commit traffic is low (e.g. <10 per day), but become necessary once it grows past your daily CI time budget, roughly (which can happen on large monorepos).

As a very simple example, if your CI takes 10 minutes, your CI time budget is 6 merges per hour.

This is because if you merge two things in parallel without validating CI for the combined changes, your main branch could end up in a broken state.

Merge queues run CI for groups of PRs. If the group passes, all the PRs in the group land simultaneously. If it does not, the group is discarded while other group permutations are still running in parallel.

This way you can run more "sequential" CI validation runs than your CI time budget allows.

In our monorepo, we get a volume of 200-300 commits per day with CI SLO of 20 mins.

Without a queue, our best case scenario would be getting capped at ~72 commits per day before seeing regressions on main despite fully green CI (in real life, you'd see regressions a lot earlier though because throughput of PRs is spiky in nature)

It's awful.

I scrolled down to the how does it work section where the first sentence is:

> Merge queue is designed for high-performance teams where multiple users regularly commit to a single branch

Half of the how does it work section is buzzwordy fluff.

Collaborative coding is powerful. But to be at your team’s most optimized state, you need automated branch management that enables multiple developers to commit code on a daily basis, without frustration. This can happen if your team’s branch is busy with many team members onboarding the same commit onramp. This can be frustrating for your team, but, more importantly, it gets in the way of shipping velocity. We don’t want that journey for you!

This is why we built merge queue. We’ve reduced the tension between branch stability and velocity. Merge queue takes care of making sure your pull request is compatible with other changes ahead of it and alerting you if something goes wrong. The result: your team can focus on the good stuff—write, submit, and commit. No tool sprawls here. This flow is still in the same place with the enablement of a modified merge button because GitHub remains your one-stop-shop for an integrated, enterprise-ready platform with the industry’s best collaboration tools.

I read it yesterday and couldn't figure out exactly what they were talking about and upon re-reading, yeah, it's bad marketing copy.

The problem is probably whoever wrote the blog post (who is likely not even the named author, depending on how their marketing team does things) tried to add a lot of high-level stuff to make it make sense to them without really needing to understand the details, and then dolled it up with a bunch of useless vapid quotes from customers and what not, because that is what marketing people think matters. Maybe it does make sense to have mealy-mouthed corporate speak for the overall product, since some executive is probably deciding whether to use GitHub as a whole and they might care if a big company uses it. I don't know that it makes much sense for specific features like this, especially in a fairly technical product like GitHub.

It's totally unreadable. There's 5% meat in it almost at the very end, the rest is about selling the feature.

The problem is that if you have multiple branches going into the same (mono)-repo, then they might all pass a localized CI-check, but fail if they are all merged. This is because the branches have an interaction between them. It can lead to a stall in commits and because everything hinges on the repo, work is going to stall as well.

So you serialize the branches, and impose an order on them: [x_1, x_2, x_3, ...]. Now, when running CI on one of these, x_j say, you do so in a temporary branch containing every branch x_i with i < j. This will avoid a stall up to branch x_j, if you started to merge the branches in order. If CI fails on branch x_j, you remove it from the list (queue) of branches to be merged and continue.

I feel like since the Microsoft acquisition almost all their communications at all levels have gone from detailed info about features to fluff marketing pieces.

Beyond that, their API docs prior to the acquisition were some of the best in the industry, readable and concise. Now they are just a complicated mess.

> Does anyone else find this article unreadable? It sounds more like a marketing piece than an explanation of what merge queue is.

Yes. The first useful line in the article is

"With GitHub’s merge queue, a temporary branch is created that contains"

and to reach there you have to skip fluff paragraphs halfway down the article.

I also have a hard time understanding what it really is.

What I think it is: instead of you trying to merge into the main branch, you try to merge into a branch where all pull requests before you are already merged in.

That way any pull request before you can't cause any merge conflicts, because they are already taken into account.

At least that's what I deduct from all the marketing fluff. Maybe I'm completely wrong.

Same here, the experience has been generally favorable.

If you do squash merges, how does the queue handle the creation of the final commit message? We already have a problem with trash-filled message bodies with GitHub, and I don't want this to make it worse.

same ... it was pretty flaky early on, got stuck, double commits, not configurable merge-group size ... but it's pretty good now

Yes. GitHub reserving features for organizations has kind of kludged up the whole service. Draft PRs for instance really should be available to everyone. I suspect they’re trying to do it as some sort of value add, but none of the features they are reserving are worth paying for on their own. It’s just an annoyance.

Because this is the kind of feature large orgs need which makes it a perfect upsell.

Fair question.

Though FWIW, you have to have a fairly large scale for merge queues to be relevant in practice.

I was surprised by this weird limitation as well. However I found its easy to workaround -- I made my CI checks provide the skipped status when they run on the PR, and then provide the success/failure status when they run on the merge-queue branch.

In a GitHub Action, this is was very easy:

  on:
    pull_request:
    merge_group:
  jobs:
    build:
      name: Whatever...
      if: github.event_name == 'merge_group'

Where do you see that limitation?

The docs say there's a separate event for merge_group which I assume means you can configure different checks.

https://docs.github.com/en/repositories/configuring-branches...

If you can cache heavily enough, the cost of a no-change re-run is effectively zero.

Actually getting to that point can be quite hard. But it's definitely possible. But also, if there's any change then you want to re-run anything that depends on the change. Limiting the rebuild to only things that have changed also take effort. But it's really worth it for the benefits you reap, both in CI and in development -- if your edit-compile-test cycle is long enough that an extra CI run is annoying, it's long enough to be detrimental in every day development.

Another option to workaround this could be to skip/noop checks based on the branch name prefix

> Was this written by ChatGPT

Don't be ridiculous... Copilot did it.

That's what your test suite is for.

I worked in a place with 50-70 programmers, and we had couple dozens rebases during the day (I don't know why anyone would want to use merge). This was a project in its initial stage, so a lot of code needed to be written rather than revised, also, after some bad experience with trying to manage the project in multiple repositories, everything was brought under one roof. So, all teams were working with the same repository.

In this situation, we usually had a significant chance of waiting for another commit to go through. We also had a system that's similar to what Github offers to manage queued rebases. This was amplified by even the sanity test being about 30 minutes long and the requirement that each commit be tested (so, if you try to rebase a branch with 10 commits, you'd have to wait 5 hours for it to go through). However, such cases would be very rare and anyone who needed to rebase such a branch would typically send an email asking for a particular day to get their changes in. To my memory, there were two instances of something like this happening.

From what I could gather from the article and the comments, the system worked differently though. First of all, ours could try to rebase in parallel, so that if one rebase fails the test, another set of changes could be incorporated immediately. However, no attempt was made to see if the change set being submitted agrees with the changes in the pipeline. I think we might have tried that once, but later ruled this to be both confusing and in most cases catching very few issues anyways (it is in general rare that two developers work on the same exact files). The confusion part comes from detecting conflicts with the branch that doesn't get rebased due to the failed test, and then the developer who's notified about a conflict is left wondering about the real reason, because, usually, by the time they get to investigate it, the failed branch had been already changed by the branch owner.

I also worked in larger companies, but usually they try to split the repository along administrative boundaries, and the number of people pushing to the same repository at the same time is not so big. They pay for it by having to invest a lot into release management, version synchronization across multiple teams, much more integration testing, worse understanding of the product by individual developers / teams, and, in general, lower quality of the product. But, off-the-shelf VCSs don't allow for sharing large repositories easily, and, of course, the problem this merge queue is trying to address would grow more severe with the size.

One of the huge advantages of this sort of workflows is it avoid devs rebasing then pushing without having run the test suite on the rebase PR.

It’s most important if the test suite is pretty slow though (e.g. minutes) as that makes it more likely you’ll get push races.

The problem it solves is that the vast number of PRs on a busy team don't conflict, both at the source level and the unit test/CI level, so you can increase the throughput of PRs by automating the merging process and merging multiple PRs at once.

It helps prevent incompatible PRs from shipping together. If two people merge incompatible changes at the same time (because both of them have a big green button), you might end up with a broken application. The way Merge Queue does its checks allows you to detect when these kinds of changes are being attempted and return one of the PRs to its author.

> I thought that was the problem it was trying to solve.

Of course not since it’s not a problem it can solve.

The problem it tries to solve is having to rebase / merge by hand, racing for CI, and the risk of going “fuck it” and merging a change with does not conflict but is semantically incompatible with a previous change.

Most of the time your colleagues’ changes and yours don’t interfere, but on repos with lots of traffic losing the CI race and having to rebase, wait for CI again, rinse and repeat, gets old quick, when the test suite takes more than a few seconds.

There are comments upthread which mention hitting this issue and say that things got smoother and they’re happy now.

> Pull request merge queues are available in any public repository owned by an organization, or in private repositories owned by organizations using GitHub Enterprise Cloud.

Source: https://docs.github.com/en/repositories/configuring-branches...