Trusted Merge Requests

Motivation

The projects for OSDU hosted here on community are tested by deploying and executing in several different cloud environments, and they also are developed through the efforts of the open source community. These two properties come into conflict when trying to accept contributions from the community into the master development line. Before the merge can occur, we need to run the proposed code through the full battery of integration tests, which includes deploying to the supported cloud platforms and executing tests against the resulting code — probably including some new tests that were written to validate the new code feature. In order to run this, the GitLab Runner needs access to several deployment keys, which are stored as protected variables. Code coming from unknown sources cannot be trusted to manage these keys securely, nor can the code contributed be consider safe to execute prereview — in fact, the merge request itself could be coming from an attacker. Therefore, the branches created by contributors and their resulting pipelines are not marked as protected, and do not execute these parts of the pipeline.

However, once the merge request has been created, and a trusted maintainer (we call them "committers") has reviewed the code, that committer can mark the code as safe for execution — effectively lending their permissions to the merge request. Since this pattern is not directly supported by GitLab, we introduce a mechanism via a separate Git branch, which is marked as a trusted execution. Via some GitLab CI magic, the two branches are connected together so that the merge request page shows the status of the other branch, so that committers can make determinations about the readiness for merging the new code.

Branch Types

The merge request process is different depending on the type of branch being merged. The two kinds of branches are referred to as "Contributor Branches" and "Committer Branches", which implies the level of trust and permissions that go with them. However, while only committers will be able to submit commits to Committer Branches, anybody can develop on a Contributor Branch. Committers of a project may find that their work is best done on a contributor branch, since this does not continually execute the integration tests. Committer branches are most useful for development related to the deployment and integration testing itself. Other development can run the integration tests as part of the merge request review process, similar to the contributor branch style, saving on pipeline time during development. It also opens the branch for collaboration among a wider audience, rather than restricting the branch to be used only by committers.

Contributor Branches

Contributor Branches are branches within a project repository that are not marked as protected, as well as branches coming in from remote repositories (aka forks). The pipelines for these branches will have executed the basic tests and scans, but will not have deployed the resulting services to any cloud providers or run integration tests against them. Once the Merge Request has been created, the project committers will work with the requestor to enable these tests and coordinate correcting any errors the tests show as well as the creation of any new tests that demonstrate proper execution of the new code feature.

Step 1: Develop Code

The fun part! If you are listed as a "Developer" on the project members list, this is as simple as creating a new branch and starting to push commits to it. If not, begin by making a fork, then create a branch on your fork to do your work.

Note that if you do make a fork, you will be able to create protected branches, since you now own the resulting project (and the namespace you copied it to). However, that isn’t sufficient to run the integration tests — the protected variables from the original project do not carry over through the fork. You will need to develop on a branch following the contributor model, or set up your own integration test environment (which is outside the scope of this document).

Step 2: Create a Merge Request

Once the code is developed sufficiently that you’d like to start sharing it, open a merge request with the original project. It’s ok if the work isn’t complete — you can even add a "WIP" flag to indicate that it isn’t fully ready to merge, yet. Creating the merge request early gives a way to show off your work and start collecting feedback earlier in the process.

The merge request will fire a special CI/CD pipeline (more accurately, a specific subset of the jobs in the standard pipeline configuration). It will detect that the merge request is a Contributor Branch (because the pipeline is not protected), and will attempt to trigger a trusted version of the pipeline.

Since you’ve just added your merge request, no committer will have had the chance to review it. As a result, the pipeline will fail immediately on the first stage. This will cause your merge request to look like this:

Initial Contributor Merge Request

If you look into the details of the failing job, you will notice (among other output) the lines:

$ trigger-trusted-tests.js
CI_PROJECT_ID: 5
CI_SERVER_URL: https://community.opengroup.org
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME: example-contributor-branch
Preparing packages...
Could not launch trusted pipeline (a branch named trusted-example-contributor-branch)
GitLab returned: HTTPError: Bad Request
Please wait until a maintainer can review your request

At this point, you need to wait for a Committer to review what you’ve included and move it along in the process. Obviously, if you are a Committer, you can execute the next step yourself.

Step 3: A Committer Authorizes Protected Execution

Once a Committer sees a merge request in this state, they first need to verify that the code is not malicious or reckless. That can be done by reviewing the code changes that the request includes, with extra attention for the parts that execute in the integration tests and changes to the pipeline logic itself. The Committer needs to ensure that the cloud secrets and other protected variables, if provided to this code via environment variables, would be safe from intentional or accidental exposure.

Once that Committer is convinced in the safety of the code, they should create a new branch and name it similar to the branch being review, but with a "trusted-" prefix. This can be done locally using a Git client and pushed to the server, or directly on the GitLab UI using the Repository | Branches page, as shown here:

New Branch Creation

Once the trusted branch exists, we need to manually execute a new pipeline in the merge request.

Running a Pipeline

Now, when we return to the Merge Request Overview, we will see the merge request pipeline has a child pipeline. The triggering job from the parent pipeline is configured to wait for the conclusion of the child (trusted) pipeline, then mirror its status.

Pipeline with Trusted Execution

Step 4: Review Iteration

Now that a Committer has executed the merge request in a protected context, there may be additional review comments. This might include fixes to the code of the requesting branch, fixes to the existing tests whose behavior is altered, or new integration tests to operate on the new code. Any commits to refine the merge request should be made on the original branch, whether those commits are by the original requester or someone else.

However, once a commit is pushed to GitLab on the MR branch, the trusted branch will no longer point to the same SHA. This will cause the pipeline to fail immediately (without executing any tests on the trusted branch). That failure will include the following output in the job log:

Checking for matching commit SHAs between trusted branch and review branch
f08d52e455d08c328a6692fb27429e5221e61bda trusted-example-contributor-branch
4ea0ba74122746619a0a699f06e8379d74ae7ec8 example-contributor-branch
Running after_script
Uploading artifacts for failed job
ERROR: Job failed: exit code 1

The original branch’s pipeline will still execute to help the author catch simple problems (like unit test failures) early, and without the assistance of a Committer. A Committer must re-mark the branch as trusted — this will likely the same Committer that initially marked the branch trusted, but does not need to be. To do this, the Committer should "move" the trusted branch to point to the same SHA that the requesting branch is now pointing to. The easiest way to accomplish this is a git checkout trusted-example-contributor-branch; git merge example-contributor-branch; git push command. This will create a fast-forward merge, updating the branch without creating any new commits. Deleting the trusted branch and recreating it also works, and is fairly easy.

It is likely this process will iterate several times, as various comment is the merge request are corrected, and then tested again.

Step 5: Post-Merge Cleanup

Once the merge is complete, a Committer will need to delete the trusted branch, since the "Delete Source Branch" option in the GitLab merge request only deletes the contributor’s branch. If this step is forgotten, no real harm befalls the project. It merely clutters the branch namespace.

These trusted branches will be direct ancestors of master (aka a "merged" branch), and can therefore be delete in one click under the Branches page by clicking "Delete merged branches".

Committer Branches

Committer branches are branches that are marked as protected during development. This is sometimes necessary if the purpose of the branch is to extend or fix the integration tests or other protected operations themselves, and the developer would like to see the status of these tests on every push — not just during merge request review. These types of branches can only be created by Committers. Moreover, only Committers can push changes to the branch or trigger a pipeline.

Contributor Branches are Preferred over Committer Branches

Contributor Branches can be created and predominantly developed by anyone. Committers should utilize Contributor Branches by default, and only resort to using Committer Branches when the development effort intimately depends upon the integration test environment. This reduces the strain on the CI/CD environments, but more importantly encourages a greater pool of developers to collaborate on the feature.

Step 1: Develop Code

After creating a branch, Committers should use the project Settings | Repository page, under the Protected Branches heading to specify that the new branch is to be protected. Remember that these pipelines will have access to the protected variables, limit those that can merge or push to the branch to Maintainers only.

As an optimization on the trusted-* branch execution, pipelines are disabled for normal commits if the branch name starts with "trusted-". Because of this, Committers should not name their branches with a "trusted-" prefix, or it will not create any pipelines and undercut the purpose.

Now that you have a protected branch, develop as usual. The pipelines resulting from pushing new commits will include all aspects of the CI/CD chain — not just the building / scanning parts.

Step 2: Create a Merge Request

When creating a merge request, add a label to the request named "no-detached-pipeline". If you are the first Committer branch in this particular project, you may need to create the label first.

This label is recognized within the standard ci-cd-pipelines templates, and causes the detached pipeline (the one the runs on the merged results) to be suppressed. Without a detached pipeline, the merge request will show the last executed pipeline for the branch, which will include all the jobs of pipeline because this branch is already marked as a protected branch.

If the merge request is created without the "no-detached-pipeline" label, then the merge request will trigger a child pipeline to execute the tests in a protected context. This does not harm anything, but it creates duplicate work — all the commits made during the review will be executed twice. The label can be added after creation, and will take effect the next time a commit is pushed.

Step 3: Review Iteration

Reviews are straightforward for Committer branches. Since the branch is already protected, changes can be pushed directly to the branch and new pipelines will execute on the change immediately. Obviously, only Committers will be able to add content to the review — other contributors will need to submit their suggestions as comments.

Step 4: Post-Merge Cleanup

Since the branch was individually marked as a protected branch, a Committer will need to go back into the Settings | Repository page and remove the protected entry. The entry will show the branchname and indicate that it was removed. Failing to do this has no ill effects, it only serves to clutter the protected branch list.