Customize the Kubernetes operator and resources it manages

Cluster resource customization using ProxyClass Custom Resource

Tailscale operator v1.60 and later provides the ability to customize the configuration of cluster resources created by the operator using ProxyClass Custom Resource Definition.

You can specify cluster resource configuration for custom labels and resource requests using a ProxyClass Custom Resource.

You can then:

• Apply configuration from a particular ProxyClass to cluster resources created for a tailscale Ingress or Service using a tailscale.com/proxy-class=<proxy-class-name> label on the Ingress or Service.

• Apply configuration from a particular ProxyClass to cluster resources created for a Connector using connector.spec.proxyClass field.

The following example shows how to use a ProxyClass that specifies custom labels and node selectors. These are applied to Pods for a tailscale Ingress, a cluster egress proxy, a Connector, and a ProxyGroup:

1. Create a ProxyClass resource:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
     name: prod
   spec:
     statefulSet:
       pod:
         labels:
           team: eng
           environment: prod
         nodeSelector:
           beta.kubernetes.io/os: "linux"
   

2. Create a tailscale Ingress with tailscale.com/proxy-class=prod label:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-app
     labels:
       tailscale.com/proxy-class: "prod"
   spec:
     rules:
     ...
     ingressClassName: tailscale
   

3. Create a cluster egress Service with a tailscale.com/proxy-class=prod label:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       tailscale.com/tailnet-ip: <tailnet-ip>
     labels:
       tailscale.com/proxy-class: "prod"
     name: my-tailnet-service
   spec:
   

4. Create a Connector that refers to the 'prod' ProxyClass:

   apiVersion: tailscale.com/v1alpha1
   kind: Connector
   metadata:
     name: prod
   spec:
     proxyClass: prod
     ...
   

5. Create a ProxyGroup that refers to the 'prod' ProxyClass:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyGroup
   metadata:
     name: egress
   spec:
     proxyClass: prod
     ...
   

You can find all available ProxyClass configuration options on GitHub →

Default ProxyClass

Tailscale v1.74 and later allows specifying a default ProxyClass. Configuration from a default ProxyClass is applied to cluster ingress, cluster egress and ProxyGroup proxies that don't have a ProxyClass explicitly set. You can set a default ProxyClass for the cluster via the proxyConfig.defaultProxyClass Helm value if installing using Helm or via the PROXY_DEFAULT_CLASS environment variable if installing using static manifests.

Proxy configuration

You can specify a ProxyClass for Connector resources, and egress and ingress proxies.

Customizing tags

All the proxies that the operator creates are Tailscale devices tagged by one or more tags.

The Tailscale operator must be a tag owner of all the proxy tags: if you want to tag a proxy device with tag:prod,tag:emea, the tagOwners section of the tailnet policy file must list tag:k8s-operator as one of the owners of both tag:prod and tag:emea.

Default tags

By default, a proxy device joins your tailnet tagged with the tag tag:k8s. You can modify the default tag or tags when installing the operator.

If you install the operator with Helm, you can use .proxyConfig.defaultTags in the Helm values file.

If you install the operator with static manifests, you can set the PROXY_TAGS environment variable in the deployment manifest.

Multiple tags must be passed as a comma separated string, that is, tag:prod,tag:emea.

Tags for individual proxies

To override the default tags for an individual proxy created for a Tailscale Service or Ingress, you can set the tailscale.com/tags annotation on the Service or Ingress resource to a comma separated list of the desired tags. For example, setting tailscale.com/tags: "tag:prod,tag:emea" annotation will result in the proxy device having the tags tag:prod and tag:emea.

To override the default tags for the proxy created for a Connector custom resource, you can set tags via the spec.tags field.

See also Common patterns for tag names for best practices around tag names.

Exposing metrics

When specified for a resource, the following ProxyClass definition will enable client metrics at the path /metrics on a container port named "metrics":

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true

Additionally the operator will also create a metrics Service for the proxy in the operator's namespace that will also expose metrics on the path /metrics on a port named "metrics".

Prometheus ServiceMonitor

The Kubernetes Operator can also create a Prometheus ServiceMonitor for proxy resources.

To enable it:

1. Ensure that Prometheus operator including its Custom Resource Definitions is installed

2. Apply ProxyClass that enables metrics and ServiceMonitor creation:

   apiVersion: tailscale.com/v1alpha1
   kind: ProxyClass
   metadata:
     name: prod
   spec:
     metrics:
       enable: true
       serviceMonitor:
         enable: true
   

The ingested metrics will have labels that identify the proxy to which the ProxyClass was applied:

• ts_proxy_type: type of the proxy. Values can be ingress_service, ingress_resource, connector or proxygroup.

• ts_proxy_parent_name: name of the proxy's parent resource. That is, name of the Ingress resource, Tailscale Service, Connector or ProxyGroup.

• ts_proxy_parent_namespace: namespace of the proxy's parent resource. This only applies to namespaced resources.

Debug endpoints

If enabled, the debug endpoints will be available on a container port named "debug". The endpoints include /debug/metrics and /debug/pprof/ paths from Go's net/http/pprof library.

To maintain backward compatibility, debug endpoints default to enabled if .spec.metrics.enable is set to true. If .spec.metrics.enable is set to false, the debug endpoints default to disabled.

In Tailscale v1.82 and later, the debug endpoints always default to disabled. You can override the default for debug endpoints using ProxyClass:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  metrics:
    enable: true
  statefulSet:
    pod:
      tailscaleContainer:
        debug:
          enable: false

Using custom machine names

Cluster ingress and egress proxies support overriding the hostname they announce while registering with Tailscale. For Services, you can set a custom hostname using a tailscale.com/hostname annotation. For Ingresses, you can set a custom hostname using the .spec.tls.hosts field (only the first value will be used).

Note that this only sets a custom operating system (OS) hostname reported by the device. The actual machine name will differ if a device is on the network with the same name.

Machine names are subject to the constraints of DNS: they can be up to 63 characters long, must start and end with a letter, and consist of only letters, numbers, and -.

Limitations

• You cannot enable metrics for egress proxies that do not use a ProxyGroup.
• You cannot enable metrics for ingress proxies that use the annotation tailscale.com/experimental-forward-cluster-traffic-via-ingress.