Customize the Kubernetes operator and resources it manages

Cluster resource customization using `ProxyClass`

Tailscale operator v1.60 and later provides the ability to customize the configuration of cluster resources created by the operator using ProxyClass Custom Resource Definition.

You can specify cluster resource configuration for custom labels and resource requests using a ProxyClass Custom Resource.

You can then:

Apply configuration from a particular ProxyClass to cluster resources created for a tailscale Ingress or Service using a tailscale.com/proxy-class=<proxy-class-name> annotation on the Ingress or Service.
Apply configuration from a particular ProxyClass to cluster resources created for a Connector using connector.spec.proxyClass field.

The following example shows how to use a ProxyClass that specifies custom labels and node selectors. These are applied to Pods for a tailscale Ingress, a cluster egress proxy, a Connector, and a ProxyGroup:

Create a ProxyClass resource:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  statefulSet:
    pod:
      labels:
        team: eng
        environment: prod
      nodeSelector:
        beta.kubernetes.io/os: "linux"

Create a tailscale Ingress with tailscale.com/proxy-class=prod annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    tailscale.com/proxy-class: "prod"
spec:
  rules:
  ...
  ingressClassName: tailscale

Create a cluster egress Service with a tailscale.com/proxy-class=prod annotation:

apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-ip: <tailnet-ip>
    tailscale.com/proxy-class: "prod"
  name: my-tailnet-service
spec:

Create a Connector that refers to the 'prod' ProxyClass:

apiVersion: tailscale.com/v1alpha1
kind: Connector
metadata:
  name: prod
spec:
  proxyClass: prod
  ...

Create a ProxyGroup that refers to the 'prod' ProxyClass:

apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
  name: egress
spec:
  proxyClass: prod
  ...

You can find all available ProxyClass configuration options on GitHub →

Default `ProxyClass`

Tailscale v1.74 and later allows specifying a default ProxyClass. Configuration from a default ProxyClass is applied to cluster ingress, cluster egress and ProxyGroup proxies that don't have a ProxyClass explicitly set. You can set a default ProxyClass for the cluster via the proxyConfig.defaultProxyClass Helm value if installing using Helm or via the PROXY_DEFAULT_CLASS environment variable if installing using static manifests.

Proxy configuration

You can specify a ProxyClass for Connector resources, and egress and ingress proxies.

The API server proxy currently runs as part of the same process as the Kubernetes operator. You can use the available operator configuration options to configure the API server proxy parameters.

Customizing tags

All the proxies that the operator creates are Tailscale devices tagged by one or more tags.

The Tailscale operator must be a tag owner of all the proxy tags: if you want to tag a proxy device with tag:prod,tag:emea, the tagOwners section of the tailnet policy file must list tag:k8s-operator as one of the owners of both tag:prod and tag:emea.

Currently, tags can not be modified after a proxy has been created.

Default tags

By default, a proxy device joins your tailnet tagged with the tag tag:k8s. You can modify the default tag or tags when installing the operator.

If you install the operator with Helm, you can use .proxyConfig.defaultTags in the Helm values file.

If you install the operator with static manifests, you can set the PROXY_TAGS environment variable in the deployment manifest.

Multiple tags must be passed as a comma separated string, that is, tag:prod,tag:emea.

Tags for individual proxies

To override the default tags for an individual proxy created for a Tailscale Service or Ingress, you can set the tailscale.com/tags annotation on the Service or Ingress resource to a comma separated list of the desired tags. For example, setting tailscale.com/tags: "tag:prod,tag:emea" annotation will result in the proxy device having the tags tag:prod and tag:emea.

To override the default tags for the proxy created for a Connector custom resource, you can set tags via the spec.tags field.

See also Common patterns for tag names for best practices around tag names.

Static endpoints

This functionality is available in Tailscale v1.86 and later, and is only available to proxies within ProxyGroups.

Tailscale uses various NAT traversal techniques to securely connect to other Tailscale nodes without manual intervention. Most of the time, you do not need to open any firewall ports for Tailscale. However, in some scenarios where NAT Traversal is unsuccessful, Tailscale proxies deployed by the operator may have to rely on a relayed connection, resulting in lower throughput and performance compared to direct connections. For example, when using AWS NAT Gateways, which are hard NATs).

In these scenarios, ProxyClass provides the configuration that allows users to leverage Kubernetes NodePort Services as extra endpoints to facilitate direct connections to ProxyGroups.

Once configured, these endpoints will only work for tailnet devices on the same network as that Kubernetes Node. If the Kubernetes Node has a public IP address, the configured static endpoint will be reachable by all Tailscale devices.

Configure firewall rules

A maximum of two static endpoints are now allowed per ProxyGroup. To specify which nodes the operator uses for NodePort Services, configure the spec.staticEndpoints.selector field in the ProxyClass resource.

To leverage static endpoints, firewall rules will likely need to be configured to allow inbound traffic to be sent to each relevant Node's IP address.

Given the horizontally scalable nature of ProxyGroups, we recommend configuring a suitable number of ports on the firewall to reduce the chance of running out of ports when scaling the number of ProxyGroup replicas.

Configure static endpoints

Create a ProxyClass. The configuration for static endpoints is exposed as part of the ProxyClass custom resource, under spec.staticEndpoints:
```
apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
 name: prod
spec:
 staticEndpoints:
  nodePort:
	 ports:
	 - "31667-31680"
	 selector:
	  kubernetes.io/os: linux
```
In example above, ProxyClass is configured to use Kubernetes NodePort services for the static endpoints. Inside the nodePortConfig, a list of ports can be configured. It's also important to ensure that the specified ranges have the necessary firewall rules configured so that the endpoints are reachable from other tailnet devices. The selector field can also be used to select which Kubernetes node's ExternalIPs should be used for the static endpoints.

Once created, this ProxyClass can be referenced on a ProxyGroup to configure static endpoints for all its replicas:

Create a ProxyGroup.

apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
	name: ingress
spec:
	type: ingress
	proxyClass: prod
	replicas: 3

The example above (once applied) will create three proxy replicas, but given the configuration in the ProxyClass will also create three Kubernetes NodePort services to facilitate connections to the ProxyGroup from other tailnet devices:

NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
prod-0-nodeport        NodePort    172.20.123.3     <none>        22918:31669/UDP   84m
prod-1-nodeport        NodePort    172.20.163.230   <none>        22918:31668/UDP   84m
prod-2-nodeport        NodePort    172.20.84.83     <none>        22918:31667/UDP   84m

Test the connection.

Once the ProxyGroup has been created, a test Pod can be created and exposed using an HA cluster ingress to test the static endpoint:

apiVersion: v1
kind: Pod
metadata:
	name: test
	namespace: default
	labels:
		app: test
spec:
	containers:
	- name: nginx
		image: nginx:latest
		ports:
		- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
	name: test
	namespace: default
	annotations:
		tailscale.com/proxy-group: prod
spec:
	type: LoadBalancer
	loadBalancerClass: tailscale
	selector:
		app: test
	ports:
		- port: 80
			targetPort: 80
			protocol: TCP

When applied, you can view the Tailscale Service IP address used to expose the Pod to the tailnet using the command kubectl get service -n default.

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                           PORT(S)           AGE
test         LoadBalancer   172.20.50.164    100.112.194.186                                       80:31109/TCP      2m17s

Test for a direct connection to the Pod in your tailnet by using the Tailscale Service IP address:

tailscale ping <EXTERNAL-IP>

If a direct connection is successful, you should see a log output similar to:

pong from prod-0 (100.80.195.111) via 44.213.106.32:31670 in 91ms

If unsuccessful, you might see a log output similar to:

pong from prod-0 (100.80.195.111) via DERP(nyc) in 90ms

If this log output is observed, ensure that:

Inbound firewall rules are configured appropriately for traffic to flow to the correct Kubernetes nodes and port ranges, as configured in the ProxyClass.
Selected nodes using spec.staticEndpoints.nodePort.selector have ExternalIPs in status.addresses.
The staticEndpoints field on the ProxyGroup's status.devices is populated with the correct address using a valid ExternalIP and the correct NodePort for that replica's Service.

Expose metrics

When specified for a resource, the following ProxyClass definition will enable client metrics at the path /metrics on a container port named "metrics":

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true

Additionally the operator will also create a metrics Service for the proxy in the operator's namespace that will also expose metrics on the path /metrics on a port named "metrics".

Prometheus `ServiceMonitor`

The Kubernetes Operator can also create a Prometheus ServiceMonitor for proxy resources.

To enable it:

Ensure that Prometheus operator including its Custom Resource Definitions is installed

Apply ProxyClass that enables metrics and ServiceMonitor creation:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true
    serviceMonitor:
      enable: true

The ingested metrics will have labels that identify the proxy to which the ProxyClass was applied:

ts_proxy_type: type of the proxy. Values can be ingress_service, ingress_resource, connector or proxygroup.
ts_proxy_parent_name: name of the proxy's parent resource. That is, name of the Ingress resource, Tailscale Service, Connector or ProxyGroup.
ts_proxy_parent_namespace: namespace of the proxy's parent resource. This only applies to namespaced resources.

Currently it is not possible to customize the created ServiceMonitor. We welcome your feedback as to what customization options would be useful for you.

Debug endpoints

Debug endpoints are unstable, may change without notice, and are not recommended for production use. If you rely on the debug metrics (at /debug/metrics), you must explicitly enable the following debug option before upgrading to 1.82 which will always default debug to disabled.

If enabled, the debug endpoints will be available on a container port named "debug". The endpoints include /debug/metrics and /debug/pprof/ paths from Go's net/http/pprof library.

To maintain backward compatibility, debug endpoints default to enabled if .spec.metrics.enable is set to true. If .spec.metrics.enable is set to false, the debug endpoints default to disabled.

In Tailscale v1.82 and later, the debug endpoints always default to disabled. You can override the default for debug endpoints using ProxyClass:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true
  statefulSet:
    pod:
      tailscaleContainer:
        debug:
          enable: false

Using custom machine names

Cluster ingress and egress proxies support overriding the hostname they announce while registering with Tailscale. For Services, you can set a custom hostname using a tailscale.com/hostname annotation. For Ingresses, you can set a custom hostname using the .spec.tls.hosts field (only the first value will be used).

Note that this only sets a custom operating system (OS) hostname reported by the device. The actual machine name will differ if a device is on the network with the same name.

Machine names are subject to the constraints of DNS: they can be up to 63 characters long, must start and end with a letter, and consist of only letters, numbers, and -.

Customize permissions

Pods for proxies created for cluster ingress using Service, cluster egress, Connector and ProxyGroup contain a main tailscale container and an init container.

Since Tailscale v1.78, both containers run as privileged containers. The main tailscale container requires privileged permissions to create a /dev/net/tun device. As an alternative, you can restrict the main container's permissions and delegate the device creation to a Kubernetes device plugin:

Install the generic device plugin to your cluster. Pass a --device flag that configures the plugin to create /dev/net/tun devices:
```
--device='{"name": "tun", "groups": [{"paths": [{"path": "/dev/net/tun"}]}, "count": 1000]}'
```

Apply a ProxyClass that restricts the tailscale container's permissions to NET_ADMIN and NET_RAW capabilities, and tells the device plugin to create a /dev/net/tun device:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: tailscale-tun
spec:
  statefulSet:
    pod:
      tailscaleContainer:
        resources:
          limits:
            squat.ai/tun: "1"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
            add:
              - NET_ADMIN
              - NET_RAW

Ensure that the ProxyClass is applied to all proxies created for Tailscale ingress Services, Tailscale egress Services, Connectors and ProxyGroups.

Refer to the detailed instructions and discussion on GitHub →

Limitations

You cannot enable metrics for egress proxies that do not use a ProxyGroup.