Experimenting with GitHub Actions: PoC to Solution

October 20th 2023

The Infrastructure Engineering team is responsible for developer tooling at NerdWallet. We like to run short, one or two week experiments (Proof of Concept or PoC) to prove out options for new tooling for our teams. We also work closely with our security team to make sure the solutions we develop for our teams are staying within our security guardrails.

As part of NerdWallet’s migration from our bespoke Continuous Integration (CI) system to community supported CI tools, we’ve been experimenting with how to effectively implement secure access to private Go modules from GitHub Actions.

This post examines how we use experimentation to find the most secure access solution for private modules from GitHub Actions. Our goal is to quickly iterate over options found in community sources to find the best way to access the private repositories while still staying within the security guardrails. We will also describe how we set up our selected solution.

Background

One of the languages used at NerdWallet to build our backend microservices is GoLang. We have a number of common library modules that are stored in private GitHub repos and are used by most of our Go microservices. Because our Go microservices need access to these private modules in order to build, we need a way for a GitHub Action to securely access our NerdWallet private modules. By design, the GitHub Action is only allowed access to the repo where the workflow is executing.

Setting up the GitHub Action Workflow

GitHub has some examples of where to start for a Go workflow, but the examples are a bit lacking.

For CI, we know we want to:

Build and test every time we make a commit on a pull request
Only run those commands if a go file has changed (or the action has changed)
Use our internal self-hosted runners to avoid 💵 overage costs

NOTE: For a full CI action, we will want some additional steps (format, lint, generate docs, publish, etc.). Also we may want caching, parallelization and some cleanup. Those additional steps are not covered in this blog post—the focus of our experiment is finding the most secure access solution for the private modules, so we just need to build the microservice..

Here’s the GitHub Action workflow to build and test our microservice:

name: Go-CI

on:
  pull_request:
    types:
      - opened
      - synchronize
  paths:
    - "go.mod"
    - "go.sum"
    - "**/*.go"
    - ".github/workflow/ci.yaml" # This workflow file
      	
jobs:
  build:
    Runs-on: ["self-hosted", "linux"]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v3
        with:
          go-version: ">=1.19"

      - name: Build
        shell: bash
        run: |
          go get ./...
          go build -v ./...

      - name: Test
        shell: bash
        run: go test -v ./..

Set GOPRIVATE ✅

My Go microservice is a pretty simple API, but it uses the common library modules that are used by most of our NerdWallet Go microservices. So in order to build, we need to set GOPRIVATE in the workflow. Setting GOPRIVATE causes Go to bypass the public servers when accessing dependencies and use our internal NerdWallet repos to find the private modules. This tutorial has a great explanation of how it works.

You can set environment variables at many levels in GitHub Actions workflows. We need this one to be set all the time, so we can put it at a global level, right before jobs:

env:
  GOPRIVATE: "github.com/NerdWallet/*"

Run the new Workflow 🔴

We committed the “ci.yaml” workflow file in a .github/workflows directory in our microservice repo, then made a Pull Request (PR). This PR triggers the run because we asked the workflow to run run on: pull_request:. Right away, we see errors right on the PR:

Clicking on the Details link shows errors when trying to go get one of our private modules: fatal: could not read Username for 'https://github.com': terminal prompts disabled

💡 This makes sense – we’re trying to access those module files from another repo, which is not allowed in a GitHub Action without additional authentication.

🤔 What to do? Turn to the community! This is a benefit of using a community-sourced tool like GitHub Actions. We’re looking for a solution we can experiment with quickly that will still meet our security requirements.

Web search results recommend making a repo-scope Personal Access Token (PAT) and calling git configure to allow the action to access the remote repos. This will work, but we don’t want a PAT for CI. PATs are specific to a user, so it’s not an ideal situation for CI. For example, if we make a PAT in a NerdWallet engineer’s account and that person leaves the company, everyone loses access to the CI workflows that use the PAT. We could use a bot GitHub user, but there are issues with that as well

Bot user needs to be able to authenticate the token using SSO, which means the bot user needs to be in the NerdWallet GitHub organization and have an Okta account
- There are overhead costs (~$300/year) of having a bot user that can have these accounts
A PAT can can have wide scope, and bot users tend to have more privileges than regular users, so it can be a security risk if the PAT is not rotated on a regular basis
- We could automate this token rotation to reduce the risk, but GitHub already provides an alternative with smaller scope available – use a GitHub App instead

Use a GitHub App to provide an Installation Access Token

Infrastructure Engineering has solved this authentication issue recently by using GitHub Apps, which can help scope access and provide short-lived “installation access tokens” for authentication. There are a few steps involved:

Create GitHub App in the NerdWallet org
Generate a private key for the App
Use the App’s private key to mint a JSON Web Token (JWT)
Use the GitHub API to exchange the JTW for an installation access token (expires in 1 hour)
Use the installation access token instead of a PAT to configure git so when go get executes, the GitHub Action is authorized to access the remote repositories

NOTE: You will need admin privileges to create and install the GitHub App

Create the GitHub App

In GitHub, navigate to the org Settings page

Select Developer settings > GitHub Apps

Click New GitHub App

Give the App a name
Add a description
Put in a Homepage URL, which is required (can default this to point back to the organization GitHub page)
Uncheck the Active checkbox under the Webhook section. Otherwise, you need to put in a webhook URL (and we don’t need one for this App)
Set Permissions. There are A LOT of permissions options. A repo scope PAT provides wide access to the repo:

But do we need to replicate that? Probably not. We only need to read the private repos, not make any changes to them. Following security best practices, we’ll start with least privileges and select only Read access to Contents. We’re experimenting – if it doesn’t work the first time, we can add more permissions later.

Click Create GitHub App all the way at the bottom of the page
After creating, note the App ID at the top of the page. (you will need this later)

Generate a private key

Scroll all the way down to the bottom and click the Generate a private key button. You will get a PEM file. Store the contents in 1Password (you will need this later).

Install the app

Scroll back up to the top and click the Install App link in the sidebar and select the organization/account and repositories to grant access to. Instead of granting access to all NerdWallet private repos, we can continue to reduce scope by only adding the private modules needed for my API microservice.

Use an action to get an Installation Access Token

One benefit of GitHub Actions is that they are composable, and there are A LOT of community-based actions available. We’re using a third-party action to do these two steps

Use the App’s private key to mint a JSON Web Token (JWT)
Use the GitHub API to exchange the JTW for an installation access token (expires in 1 hour)

In order to make this work, we need to store the App ID and private key as encrypted GitHub secrets. Then we can use those secrets when we call our third party action as a step in our job:

- name: Get token for private repo
  id: generate_token
  uses: getsentry/action-github-app-token@v2
  with:
    app_id: ${{ secrets.APP_ID }}
    private_key: ${{ secrets.PRIVATE_KEY }}

NOTE: It’s important to give this step an id, because we will need to reference that to retrieve the output of this step (which is our installation access token).

Configure Git Credentials

Now that we have the GitHub App and the step to provide the installation access token, we can configure git to use the username and token instead of the default credential URL.

Add this as a step before your build step, and after the generate_token step:

- name: Configure git for private modules
  env:
    TOKEN: ${{ steps.generate_token.outputs.token }}
  shell: bash
  run: git config --global url."https://${{ github.actor }}:${TOKEN}@github.com".insteadOf "https://github.com"

❗While experimenting, it took a few tries to figure out that because we are using an installation access token instead of a PAT, we need to provide both the username AND the token in this configure call. A PAT has the username encoded within it, and you can just specify that without the username. But when we tried configuring only with the installation access token without the username, we get errors like could not read Password for 'https://***@github.com'.

Successful CI Build 🎉

This is the workflow with all of the pieces in place:

name: Go-CI

on:
  pull_request:
    types:
      - opened
      - synchronize
  paths:
    - “go.mod”
    - “go.sum”
    - “**/*.go”
    - “.github/workflow/ci.yaml” # This workflow file
      	
env:
  GOPRIVATE: "github.com/NerdWallet/*"

jobs:
  build:
    Runs-on: [“self-hosted”, “linux”]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v3
        with:
          go-version: ">=1.19"

      - name: Get token for private repo
        id: generate_token
        uses: getsentry/action-github-app-token@v2
        with:
          app_id: ${{ secrets.APP_ID }}
          private_key: ${{ secrets.PRIVATE_KEY }}

      - name: Configure git for private modules
        env:
          TOKEN: ${{ steps.generate_token.outputs.token }}
        shell: bash
        run: git config --global url."https://${{ github.actor }}:${TOKEN}@github.com".insteadOf "https://github.com"

      - name: Build
        shell: bash
        run: |
          go get ./...
          go build -v ./...

      - name: Test
        shell: bash
        run: go test -v ./...

Committing those updates triggered a new run. Now we see no more errors from the build step, and we get status right on the PR!

GitHub Apps Can Help Reduce Security Risk

This post demonstrates how experimentation can help you find the best way to access the private repositories while still staying within the security guardrails, and how to set up a GitHub App to provide the CI authentication needed while reducing the security risks.

You’ll need to balance a lot of requirements when building the tooling for your developers. While we could have used a PAT, an SSH token, or deploy keys to achieve the same access results, those options all have cost and security implications that are not ideal for our CI pipelines. You should use experiments and proof of concept work as a first priority, which allows you to focus your design work on solutions that you know will deliver what your teams need.

Would you like more detailed insight into how we built some of the workflows mentioned above? Feel free to comment below, and if you see yourself experimenting and building tooling at NerdWallet, we encourage you to apply.