Kubernetes API server proxy 'kubectl exec' session recording

In Tailscale 1.70 and later, you can record kubectl exec session contents when using the Kubernetes API server proxy.

Recorded sessions are sent to a configured tsrecorder instance in your tailnet. The tsrecorder supports storing recorded sessions in an S3-compatible storage or on disk.

A recording contains:

• Command sent in the kubectl exec call (for example, kubectl exec -- cat /myfile.txt will show up as cat /myfile.txt).
• The contents of stdout and stderr from the attached terminal, but not stdin. This prevents the recording of sensitive input, such as passwords.
• Session start timestamp.
• Name and namespace of the Pod and name of the container that was exec-ed.
• Tailscale hostname of the API server proxy.
• Tailscale identity that initiated the session:
  • Tailscale hostname of the source device from which the kubectl exec session was inititated.
  • If the source device is tagged, tags of the device.
  • If the source device is not tagged, Tailscale user ID and login name.

The API server proxy sends session contents to tsrecorder over the tailnet, so the traffic is end-to-end encrypted using WireGuard, like any other tailnet traffic.

You can choose whether a kubectl exec session should be allowed ('fail open') or be forbidden ('fail closed') if the recording fails (for example, if a connection to tsrecorder cannot be established or fails).

Prerequisites

• Set up the Kubernetes operator with the API server proxy enabled.
• Deploy a tagged tsrecorder instance. Note that you can also deploy tsrecorder on Kubernetes using the operator.
• Ensure that your ACLs allow the operator to access port 80 of the tsrecorder.

Configure the session recording

You can configure session recording options using grants.

You must specify the source (the tailnet identity that initiates the kubectl exec session), the destination (the API server proxy), and the tsrecorder instance. You can also specify the failure policy. The default failure policy is to fail open.

The following example:

• Allows requests from group:engineering to connect to the API server proxy on all operator instances tagged with tag:k8s-operator on port 443.
• Enforces session recording for kubectl exec sessions from tailnet group group:engineering to all API server proxy instances tagged with tag:k8s-operator.
• Configures the recorded sessions to be sent to a tsrecorder instance tagged with tag:tsrecorder.
• Sets failure policy to "fail closed" (if the recording fails, forbid the session).

"acls": [
	{"action": "accept", "src": ["group:engineering"], "dst": ["tag:k8s-operator:443"]}
],
"grants": [
	{
		"src": ["group:engineering"],
		"dst": ["tag:k8s-operator"],
		"app": {
			"tailscale.com/cap/kubernetes": [{
				"recorder":        ["tag:tsrecorder"],
				"enforceRecorder": true,
			}],
		},
	},
]

Combine session recording and group impersonation in grant rules

A grant can also combine session recording rules and API server proxy impersonation rules.

The following example:

• Allows requests from group:engineering to connect to the API server proxy on all operator instances tagged with tag:k8s-operator on port 443.

• Configures session recording for kubectl exec sessions from group group:engineering to all operator instances tagged with tag:k8s-operator.

• Configures the recorded sessions to be sent to a tsrecorder instance tagged with tag:tsrecorder.

• Configures that requests from group:engineering should be impersonated as from Kubernetes group system:masters.

"acls": [
	{"action": "accept", "src": ["group:engineering"], "dst": ["tag:k8s-operator:443"]}
],
"grants": [
	{
		"src": ["group:engineering"],
		"dst": ["tag:k8s-operator"],
		"app": {
			"tailscale.com/cap/kubernetes": [{
				"impersonate": {
					"groups": ["system:masters"],
				},
				"recorder": ["tag:tsrecorder"],
			}],
		},
	},
]

Enforce recording of all sessions

To ensure the recording of all kubectl exec sessions, use the wildcard syntax (*) as the grant's source and/or destination.

The following example:

• Enforces recording of all kubectl exec sessions from any source to any API server proxy instance.

• Configures the recording to be sent to tsrecorder instance tagged with tag:tsrecorder.

"grants": [
	{
		"src": ["*"],
		"dst": ["*"],
		"app": {
			"tailscale.com/cap/kubernetes": [{
				"recorder":        ["tag:tsrecorder"],
				"enforceRecorder": true,
			}],
		},
	},
]

Known issues and limitations

• The contents of shell sessions of ephemeral debug containers as well as sessions initiated via 'kubectl attach' and 'kubectl run' are currently not recorded.

• In some cases, if an established connection to tsrecorder fails midway (because the tsrecorder instance fails), the session will be allowed to continue even if the policy is to fail closed. We are working on fixing this.