Kubernetes API request events with Tailscale

With the Tailscale Kubernetes operator, you can record all Kubernetes API requests that pass through the Kubernetes API server proxy. This lets you create a detailed audit trail of cluster interactions for security and compliance purposes.

This feature builds on the existing kubectl session recording feature. When you enable API request events, kubectl session recording is also enabled.

Kubernetes API request events are sent to a configured tsrecorder instance in your tailnet. The tsrecorder supports storing events and recorded sessions in an S3-compatible storage backend or on disk.

What request events contain

A Kubernetes API request event provides a comprehensive record of each API request, including:

• Timestamp of when the event occurred.
• Detailed information about the Kubernetes API request, including:
  • The verb such as get, list, create.
  • The targeted resource such as pods and services.
  • The namespace, name, API group, and version.
• Full HTTP request details, including the method, path, request body and query parameters.
• User agent of the client that made the request (for example, kubectl/v1.33.3).
• Source Tailscale machine that the request came from.
• Tailscale identity that initiated the API call, including:
  • Tailscale hostname of the source device.
  • If the source device is tagged and the tags of the device.
  • If the source device is not tagged, the Tailscale user ID and login name.

This is all encapsulated in a general tsrecorder event of type kubernetes-api-request.

The API server proxy sends API request events to tsrecorder over your tailnet, so the traffic is end-to-end encrypted using WireGuard, just like any other tailnet traffic.

You can also choose whether API requests should be allowed ('fail open') or forbidden ('fail closed') if recording fails (for example, a connection to tsrecorder cannot be established).

Prerequisites

Before you begin, you'll need to:

• Follow the steps outlined for configuring session recording on the operator.
• Deploy a tagged tsrecorder instance. You can also deploy tsrecorder on Kubernetes using the operator).

Configure the server proxy to send events

Because Kubernetes API request events is an alpha feature, you will need to set the TS_EXPERIMENTAL_KUBE_API_EVENTS environment variable on your Kubernetes API server proxy deployment. This will involve different configuration depending on how you have deployed your proxy.

Configure operator in-process proxy

If you are running the API server proxy that is integrated into the operator deployment, you will need to modify the deployment so that the environment variable is set on the operator's container.

If you're deploying the operator using helm, you will need to add the following configuration to your values file:

operatorConfig:
  extraEnv:
    - name: "TS_EXPERIMENTAL_KUBE_API_EVENTS"
      value: "true"

If you're deploying the operator using static manifests, you will need to add the following configuration to the container spec of the operator container in the operator deployment:

env:
- name: "TS_EXPERIMENTAL_KUBE_API_EVENTS"
	value: "true"

High Availability API server proxy

If you are running the API server proxy in a high availability configuration using a ProxyGroup, you will need to either create or modify a ProxyClass resource to configure the containers of that ProxyGroup to have the environment variable:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: api-proxy-events
spec:
  statefulSet:
    pod:
      tailscaleContainer:
        env:
          - name: TS_EXPERIMENTAL_KUBE_API_EVENTS
            value: "true"

The ProxyClass can then be consumed by the ProxyGroup responsible for running the Kubernetes API server proxy by setting it in the ProxyGroup's spec:

apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
  name: ts-api-server
spec:
  type: kube-apiserver
  proxyClass: api-proxy-events

Review events

Once the API server proxy has been configured as specified above, events will start getting sent to tsrecorder.

If you want to visualize your information in a web browser, you can enable and access the tsrecorder web UI. If tsrecorder was created using the operator, you can enable the web UI by configuring the spec.enableUI field for the recorder resource that manages it:

apiVersion: tailscale.com/v1alpha1
kind: Recorder
metadata:
  name: recorder
spec:
  enableUI: true

The web UI can then be accessed by going to the MagicDNS name assigned to the recorder that was created. This can be found by running kubectl get recorders against the cluster that has the recorder resource on it:

$ kubectl get recorders
NAME       STATUS            URL                                                AGE
recorder   RecorderCreated   https://recorder.<your-tailnet-name>.ts.net        14d