Using GitHub Actions and Tailscale to build and deploy applications securely

Naman Sood on
Photo of Naman Sood

Recently, I’ve started blogging, and to serve the raw Markdown into delicious HTML and CSS, I wrote a basic web server in Go that compiles Markdown and then injects it into a template and serves it over HTTP. The biggest annoyance in this server is deployment; every change I make needs to be pulled on the server-side, potentially recompiled if the Go code changes, and restarted.

Automating all this should be DevOps 101 — which is convenient for me since being a university student is about the time I figure I should be doing DevOps 101. Armed with this confidence, I devised a plan — a GitHub Action that builds the binary, packages it along with the pages and static files, then pushes the package and restarts the server automatically. The only problem is that my server’s SSH is accessible only over Tailscale.

It sounds like a great idea security-wise; expose SSH only over Tailscale, so the only devices that can access your server at all are the devices on your trusted network. But that means that for the GitHub Action to be able to access my server, it too must join my Tailscale network. That works for me, but there are a couple of reasons why you can’t just tailscale up.

Firstly, when you do tailscale up, you’re given a URL on your terminal that you need to open and sign into Tailscale on in order to authorize this node to join your network. It would be very inconvenient to have to manually approve your CI every time it runs. This is a solved problem, though — with authentication keys, you can tailscale up without any user interaction.

Secondly, the moment you sign onto your Tailscale network, your device is registered there permanently, at least until you go to the admin panel and remove it. This sounds like it could be fine, until you realize that Docker containers are supposed to be stateless. The container itself forgets its identity with every fresh boot, but your Tailscale network doesn’t. So you end up with a new machine registered on your network every time a container spins up.

Three containers in the Tailscale admin panel, all with randomly generated hostnames and all shown as offline.

Not pictured: 36 more of these that I deleted by hand.

This issue can be somewhat resolved with a very clever hack, like the one Tavian Barnes used with his FreeBSD CI GitHub Action. He generated the machine-specific key once, stored it as a GitHub Actions secret, and then piped it into the container before starting Tailscale. So every time it starts up, the container uses the same identity as last time. That’s great, that’s exactly what we want. Right? Well, almost.

The thing about this machine-specific key is that it’s supposed to be unique — only one machine is supposed to have it at any given time. So while the GitHub Action would work great as long as there was only one instance of it running, as soon as another Action starts concurrently, the two containers would start battling each other as the One True Machine represented by the key. While that may be acceptable in a lot of cases, it’s not ideal.

So we devised another solution: ephemeral authentication keys. The idea is that since multiple container instances can be running concurrently, it makes sense to treat them all as different machines. So we generate a new machine key, and sign these containers into Tailscale as normal. But when signed up with an ephemeral key, Tailscale will assume that these machines themselves are “ephemeral”, or temporary. This means that if such machines go offline for a long period of time, they are expected to be gone forever, and Tailscale will automatically remove them from your network. Also, since there can be a lot of containers running at any given time, we decided to make these “ephemeral machines” IPv6-only. This allows them to use a vast /64 pool of addresses that does not conflict with any non-ephemeral nodes, and makes them easy to identify on the network, thanks to Tailscale’s unique ULA prefix fd7a:115c:a1e0 (a reference to tailscale0, the default network interface name for Tailscale on Linux).

Generating an ephemeral authentication key will be simple, once the feature is available to use. You simply create it in the admin panel, on the Keys page, like any other key:

The Keys page on the Tailscale admin panel, with options for different kinds of keys. The option for Ephemeral Keys is selected, and below the options, there is a big blue button that generates a new key of the selected kind.

Side note: the best part about creating a new feature is clicking the big blue button and seeing it work.

And then, you tell your containers to tailscale up using this key:

A container running the tailscaled server, bringing tailscale up with the newly generated key, and then showing all the machines on the network. The container is named as a random string of hexadecimal numbers, and the other machines are named hafnium, hydrogen, and magnesium.

I like to name my devices after periodic table elements. Remind me again, what’s the atomic number of 8c7a2c1c8938?

Now we have our key, and we can sign onto the network with it. Time to make some GitHub Actions!

To use GitHub Actions to accomplish our task, we need to create a job, which is a list of steps that the job needs to do. To quickly recap, my workflow needs to:

  1. Build my blog binary
  2. Package it up along with static resources
  3. Push it to my server over Tailscale

Breaking it down further, we get:

  1. Build my blog binary

    1. Install Go 1.16
    2. Build the blog binary
  2. Package it up along with static resources

    1. Run tar with the right magic incantations
  3. Push it to my server over Tailscale

    1. Connect to Tailscale
    2. Send the tarball to my server
    3. Remotely extract the tarball and restart the blog process

Each of these can be a distinct step. For most of them, I can quickly write a shell script that will do the thing I need it to. For the Go install step and the Tailscale connect step, however, my job essentially boils down to “fetch a binary and set it up”. Here, it would be really nice if the people who provided those binaries could do this for me.

For Go, GitHub provides an Action called actions/setup-go, which you can include as a step, and have it take care of everything. All I need to do is provide the Go version I need.

For Tailscale, no such Action existed. Fortunately, I work at Tailscale so I can justify spending the time to make it happen! Now, using tailscale/github-action, you can include “connect to Tailscale” as a step, after which your CI will have access to your Tailscale network. All you need to do is include the action and provide it an ephemeral authentication key. (A regular authentication key will also work, but then you have to do all that removing-nodes business.)

- name: Setup Tailscale
  uses: tailscale/github-action@main
  with:
    authkey: ${{ secrets.TAILSCALE_AUTHKEY }}

And with this, we’re done! My workflow is saved, and all is well. This is how it looks today:

on: [push]

jobs:
  deploy:
    runs-on: ubuntu-latest
    name: Deploy blog to hydrogen
    if: github.ref == 'refs/heads/main'
    env:
      MACHINE: hydrogen
    steps:
    - name: Setup Go 1.16
      id: go
      uses: actions/setup-go@v2
      with:
        go-version: '^1.16'
    - name: Setup Tailscale
      id: tailscale
      uses: tailscale/tailscale-deploy-github@main
      with:
        authkey: ${{ secrets.TAILSCALE_AUTHKEY }}
        version: 1.7.359
    - name: Add SSH key
      id: ssh
      env:
        SSH_KEY: ${{ secrets.SSH_KEY }}
      run: |
        mkdir -p ~/.ssh
        MACHINE_IP="$(tailscale ip -6 $MACHINE)"
        ssh-keyscan $MACHINE_IP >> ~/.ssh/known_hosts
        printf "%s" "$SSH_KEY" > ~/.ssh/key
        chmod 600 ~/.ssh/key
    - name: Fetch code
      id: fetch
      uses: actions/checkout@v1

    - name: Compile blog binary
      id: compile
      run: go build -o prose .

    - name: Build tarball and ship it
      id: tarball
      run: |
        TIME=$(date +%Y%m%d-%H%M%S)
        FILENAME=prose-${TIME}.tar.gz
        mkdir -p static/css
        tar -czf $FILENAME prose static/ styles/ templates/ posts/
        MACHINE_IP="$(tailscale ip -6 $MACHINE)"
        echo $FILENAME "github@[$MACHINE_IP]:/home/github/"
        scp -i ~/.ssh/key $FILENAME "github@[$MACHINE_IP]:/home/github/"
        ssh -i ~/.ssh/key "github@$MACHINE_IP" "tar -C /var/www/blog -xzf ~/$FILENAME"
        ssh -i ~/.ssh/key "github@$MACHINE_IP" "sudo systemctl restart prose"

This was just one of the many possible use cases of ephemeral authentication keys. These keys, along with the recently added support for userspace networking, will allow Tailscale to run in a wide range of container-based cloud environments, like Heroku, AWS Lambda, Google Cloud Run, and more. My time at Tailscale as an intern is ending, but I’d love to see what you build with this. Tweet them to me @tendstofortytwo; you receive a free like and I receive the satisfaction of having built something useful to people. It’s a win-win. :)

Subscribe for monthly updates

Product updates, blog posts, company news, and more.

Too much email? RSS Twitter