Customize the Kubernetes operator and resources it manages

Cluster resource customization using ProxyClass

Tailscale operator v1.60 and later provides the ability to customize the configuration of cluster resources created by the operator using ProxyClass Custom Resource Definition.

You can specify cluster resource configuration for custom labels and resource requests using a ProxyClass Custom Resource.

You can then:

• Apply configuration from a particular ProxyClass to cluster resources created for a tailscale Ingress or Service using a tailscale.com/proxy-class=<proxy-class-name> annotation on the Ingress or Service.

• Apply configuration from a particular ProxyClass to cluster resources created for a Connector using connector.spec.proxyClass field.

The following example shows how to use a ProxyClass that specifies custom labels and node selectors. These are applied to Pods for a tailscale Ingress, a cluster egress proxy, a Connector, and a ProxyGroup:

1. Create a ProxyClass resource:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
     name: prod
   spec:
     statefulSet:
       pod:
         labels:
           team: eng
           environment: prod
         nodeSelector:
           beta.kubernetes.io/os: "linux"
   

2. Create a tailscale Ingress with tailscale.com/proxy-class=prod annotation:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-app
     annotations:
       tailscale.com/proxy-class: "prod"
   spec:
     rules:
     ...
     ingressClassName: tailscale
   

3. Create a cluster egress Service with a tailscale.com/proxy-class=prod annotation:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       tailscale.com/tailnet-ip: <tailnet-ip>
       tailscale.com/proxy-class: "prod"
     name: my-tailnet-service
   spec:
   

4. Create a Connector that refers to the 'prod' ProxyClass:

   apiVersion: tailscale.com/v1alpha1
   kind: Connector
   metadata:
     name: prod
   spec:
     proxyClass: prod
     ...
   

5. Create a ProxyGroup that refers to the 'prod' ProxyClass:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyGroup
   metadata:
     name: egress
   spec:
     proxyClass: prod
     ...
   

You can find all available ProxyClass configuration options on GitHub →

Default ProxyClass

Tailscale v1.74 and later allows specifying a default ProxyClass. Configuration from a default ProxyClass is applied to cluster ingress, cluster egress and ProxyGroup proxies that don't have a ProxyClass explicitly set. You can set a default ProxyClass for the cluster via the proxyConfig.defaultProxyClass Helm value if installing using Helm or via the PROXY_DEFAULT_CLASS environment variable if installing using static manifests.

Proxy configuration

You can specify a ProxyClass for Connector resources, and egress and ingress proxies.

Customizing tags

All the proxies that the operator creates are Tailscale devices tagged by one or more tags.

The Tailscale operator must be a tag owner of all the proxy tags: if you want to tag a proxy device with tag:prod,tag:emea, the tagOwners section of the tailnet policy file must list tag:k8s-operator as one of the owners of both tag:prod and tag:emea.

Default tags

By default, a proxy device joins your tailnet tagged with the tag tag:k8s. You can modify the default tag or tags when installing the operator.

If you install the operator with Helm, you can use .proxyConfig.defaultTags in the Helm values file.

If you install the operator with static manifests, you can set the PROXY_TAGS environment variable in the deployment manifest.

Multiple tags must be passed as a comma separated string, that is, tag:prod,tag:emea.

Tags for individual proxies

To override the default tags for an individual proxy created for a Tailscale Service or Ingress, you can set the tailscale.com/tags annotation on the Service or Ingress resource to a comma separated list of the desired tags. For example, setting tailscale.com/tags: "tag:prod,tag:emea" annotation will result in the proxy device having the tags tag:prod and tag:emea.

To override the default tags for the proxy created for a Connector custom resource, you can set tags via the spec.tags field.

See also Common patterns for tag names for best practices around tag names.

Static endpoints

Tailscale uses various NAT traversal techniques to securely connect to other Tailscale nodes without manual intervention. Most of the time, you do not need to open any firewall ports for Tailscale. However, in some scenarios where NAT Traversal is unsuccessful, Tailscale proxies deployed by the operator may have to rely on a relayed connection, resulting in lower throughput and performance compared to direct connections. For example, when using AWS NAT Gateways, which are hard NATs).

In these scenarios, ProxyClass provides the configuration that allows users to leverage Kubernetes NodePort Services as extra endpoints to facilitate direct connections to ProxyGroups.

Configure firewall rules

To leverage static endpoints, firewall rules will likely need to be configured to allow inbound traffic to be sent to each relevant Node's IP address.

Given the horizontally scalable nature of ProxyGroups, we recommend configuring a suitable number of ports on the firewall to reduce the chance of running out of ports when scaling the number of ProxyGroup replicas.

Configure static endpoints

1. Create a ProxyClass. The configuration for static endpoints is exposed as part of the ProxyClass custom resource, under spec.staticEndpoints:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
    name: prod
   spec:
    staticEndpoints:
     nodePort:
   	 ports:
   	 - "31667-31680"
   	 selector:
   	  kubernetes.io/os: linux
   

   In example above, ProxyClass is configured to use Kubernetes NodePort services for the static endpoints. Inside the nodePortConfig, a list of ports can be configured. It's also important to ensure that the specified ranges have the necessary firewall rules configured so that the endpoints are reachable from other tailnet devices. The selector field can also be used to select which Kubernetes node's ExternalIPs should be used for the static endpoints.

   Once created, this ProxyClass can be referenced on a ProxyGroup to configure static endpoints for all its replicas:

2. Create a ProxyGroup.

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyGroup
   metadata:
   	name: ingress
   spec:
   	type: ingress
   	proxyClass: prod
   	replicas: 3
   

   The example above (once applied) will create three proxy replicas, but given the configuration in the ProxyClass will also create three Kubernetes NodePort services to facilitate connections to the ProxyGroup from other tailnet devices:

   NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
   prod-0-nodeport        NodePort    172.20.123.3     <none>        22918:31669/UDP   84m
   prod-1-nodeport        NodePort    172.20.163.230   <none>        22918:31668/UDP   84m
   prod-2-nodeport        NodePort    172.20.84.83     <none>        22918:31667/UDP   84m
   

3. Test the connection.

   Once the ProxyGroup has been created, a test Pod can be created and exposed using an HA cluster ingress to test the static endpoint:

   apiVersion: v1
   kind: Pod
   metadata:
   	name: test
   	namespace: default
   	labels:
   		app: test
   spec:
   	containers:
   	- name: nginx
   		image: nginx:latest
   		ports:
   		- containerPort: 80
   ---
   apiVersion: v1
   kind: Service
   metadata:
   	name: test
   	namespace: default
   	annotations:
   		tailscale.com/proxy-group: prod
   spec:
   	type: LoadBalancer
   	loadBalancerClass: tailscale
   	selector:
   		app: test
   	ports:
   		- port: 80
   			targetPort: 80
   			protocol: TCP
   

   When applied, you can view the Tailscale Service IP address used to expose the Pod to the tailnet using the command kubectl get service -n default.

   NAME         TYPE           CLUSTER-IP       EXTERNAL-IP                                           PORT(S)           AGE
   test         LoadBalancer   172.20.50.164    100.112.194.186                                       80:31109/TCP      2m17s
   

   Test for a direct connection to the Pod in your tailnet by using the Tailscale Service IP address:

   tailscale ping <EXTERNAL-IP>
   

   If a direct connection is successful, you should see a log output similar to:

   pong from prod-0 (100.80.195.111) via 44.213.106.32:31670 in 91ms
   

   If unsuccessful, you might see a log output similar to:

   pong from prod-0 (100.80.195.111) via DERP(nyc) in 90ms
   

   If this log output is observed, ensure that:

   • Inbound firewall rules are configured appropriately for traffic to flow to the correct Kubernetes nodes and port ranges, as configured in the ProxyClass.
   • Selected nodes using spec.staticEndpoints.nodePort.selector have ExternalIPs in status.addresses.
   • The staticEndpoints field on the ProxyGroup's status.devices is populated with the correct address using a valid ExternalIP and the correct NodePort for that replica's Service.

Expose metrics

When specified for a resource, the following ProxyClass definition will enable client metrics at the path /metrics on a container port named "metrics":

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true

Additionally the operator will also create a metrics Service for the proxy in the operator's namespace that will also expose metrics on the path /metrics on a port named "metrics".

Prometheus ServiceMonitor

The Kubernetes Operator can also create a Prometheus ServiceMonitor for proxy resources.

To enable it:

1. Ensure that Prometheus operator including its Custom Resource Definitions is installed

2. Apply ProxyClass that enables metrics and ServiceMonitor creation:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
     name: prod
   spec:
     metrics:
       enable: true
       serviceMonitor:
         enable: true
   

The ingested metrics will have labels that identify the proxy to which the ProxyClass was applied:

• ts_proxy_type: type of the proxy. Values can be ingress_service, ingress_resource, connector or proxygroup.

• ts_proxy_parent_name: name of the proxy's parent resource. That is, name of the Ingress resource, Tailscale Service, Connector or ProxyGroup.

• ts_proxy_parent_namespace: namespace of the proxy's parent resource. This only applies to namespaced resources.

Debug endpoints

If enabled, the debug endpoints will be available on a container port named "debug". The endpoints include /debug/metrics and /debug/pprof/ paths from Go's net/http/pprof library.

To maintain backward compatibility, debug endpoints default to enabled if .spec.metrics.enable is set to true. If .spec.metrics.enable is set to false, the debug endpoints default to disabled.

In Tailscale v1.82 and later, the debug endpoints always default to disabled. You can override the default for debug endpoints using ProxyClass:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true
  statefulSet:
    pod:
      tailscaleContainer:
        debug:
          enable: false

Using custom machine names

Cluster ingress and egress proxies support overriding the hostname they announce while registering with Tailscale. For Services, you can set a custom hostname using a tailscale.com/hostname annotation. For Ingresses, you can set a custom hostname using the .spec.tls.hosts field (only the first value will be used).

Note that this only sets a custom operating system (OS) hostname reported by the device. The actual machine name will differ if a device is on the network with the same name.

Machine names are subject to the constraints of DNS: they can be up to 63 characters long, must start and end with a letter, and consist of only letters, numbers, and -.

Customize permissions

Pods for proxies created for cluster ingress using Service, cluster egress, Connector and ProxyGroup contain a main tailscale container and an init container.

Since Tailscale v1.78, both containers run as privileged containers. The main tailscale container requires privileged permissions to create a /dev/net/tun device. As an alternative, you can restrict the main container's permissions and delegate the device creation to a Kubernetes device plugin:

1. Install the generic device plugin to your cluster. Pass a --device flag that configures the plugin to create /dev/net/tun devices:

   --device='{"name": "tun", "groups": [{"paths": [{"path": "/dev/net/tun"}]}, "count": 1000]}'
   

2. Apply a ProxyClass that restricts the tailscale container's permissions to NET_ADMIN and NET_RAW capabilities, and tells the device plugin to create a /dev/net/tun device:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
     name: tailscale-tun
   spec:
     statefulSet:
       pod:
         tailscaleContainer:
           resources:
             limits:
               squat.ai/tun: "1"
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:
               drop:
                 - ALL
               add:
                 - NET_ADMIN
                 - NET_RAW
   

3. Ensure that the ProxyClass is applied to all proxies created for Tailscale ingress Services, Tailscale egress Services, Connectors and ProxyGroups.

Refer to the detailed instructions and discussion on GitHub →

Limitations

• You cannot enable metrics for egress proxies that do not use a ProxyGroup.