Kubernetes operator
The Tailscale Kubernetes operator allows you to:
- Expose services in your Kubernetes cluster to your Tailscale network
- Securely connect to the Kubernetes control plane (kube-apiserver) via an API server proxy, with or without authentication
- Egress from a Kubernetes cluster to an external service on your Tailscale network
Setting up the Kubernetes operator
-
In your tailnet policy file, create the ACL tags
tag:k8s-operator
andtag:k8s
, and maketag:k8s-operator
an owner oftag:k8s
. If you want your services to be exposed with tags other than the defaulttag:k8s
, create those as well and maketag:k8s-operator
an owner."tagOwners": { "tag:k8s-operator": [], "tag:k8s": ["tag:k8s-operator"], }
-
Create an OAuth client in the OAuth clients page of the admin console. Create the client with
Devices
write scope and the tagtag:k8s-operator
. -
Download the Tailscale Kubernetes operator manifest file from the tailscale/tailscale repo.
-
Edit your version of the manifest file:
- Find
# SET CLIENT ID HERE
and replace it with your OAuth client ID. - Find
# SET CLIENT SECRET HERE
and replace it with your OAuth client secret.
For both the client ID and secret, quote the value, to avoid any potential yaml misinterpretation of unquoted strings. For example, use:
client_id: "k123456CNTRL" client_secret: "tskey-client-k123456CNTRL-abcdef"
instead of:
client_id: k123456CNTRL client_secret: tskey-client-k123456CNTRL-abcdef
- Find
-
Apply the edited file to your Kubernetes cluster:
kubectl apply -f manifest.yaml
This creates the “tailscale” namespace in your cluster, and deploys the Tailscale operator within it.
-
Verify that the Tailscale operator has joined your tailnet. Open the Machines page of the admin console and look for a node named tailscale-operator, tagged with the
tag:k8s-operator
tag. It may take a minute or two for the operator to join your tailnet, due to the time required to download and start the container image in Kubernetes.
Exposing a service to your tailnet (cluster ingress)
You can use the Tailscale Kubernetes operator to expose a Kubernetes service to your Tailscale network in three ways: by making it a LoadBalancer
type with the tailscale
loadBalancerClass
, by annotating an existing service, or by creating an ingress resource fronting a service.
Exposing a service using loadBalancerClass
Edit the service you want to expose and make it a load balancer:
- Set
spec.type
toLoadBalancer
. - Set
spec.loadBalancerClass
totailscale
.
Once provisioning is complete, the service’s status will show the
fully-qualified domain name of the service in your tailnet. You can view the
service’s status by running kubectl get service <service name>
.
You should also see a new node with that name appear in the Machines page of the admin console.
Exposing a service using annotations
If the service you want to expose already exists, you can expose it to Tailscale using object annotations.
Edit the service and under metadata.annotations
, add the annotation
tailscale.com/expose
with the value "true"
. Note that "true"
is quoted
because annotation values are strings, and an unquoted true
will be
incorrectly interpreted as a boolean.
In this mode, Kubernetes doesn’t tell you the Tailscale machine name. You can look
up the node in the Machines of the admin console to learn
its machine name. By default, the machine name of an exposed service is
<k8s-namespace>-<k8s-servicename>
.
Using a custom machine name
If you want the service to have a machine name other than the default
<k8s-namespace>-<k8s-servicename>
, you can provide your own machine name by
setting the tailscale.com/hostname
annotation on the service, with your
desired machine name as the value.
Machine names are subject to the constraints of DNS: they can be up to 63 characters
long, must start and end with a letter, and consist of only letters, numbers,
and -
.
Customizing ACL tags
By default, services join your tailnet tagged with the ACL tag tag:k8s
. You can use a
different tag or tags by setting the tailscale.com/tags
annotation on the
service, with a comma-separated list of the desired tags.
For example, setting tailscale.com/tags = tag:foo,tag:bar
will result in the
tailnet node having the tags tag:foo
and tag:bar
.
The Tailscale operator must be a tag owner of all the specified tags: if you want
to expose a service with tag:foo,tag:bar
, the tagOwners
section of the
tailnet policy file must list tag:k8s-operator
as one
of the owners of both tag:foo
and tag:bar
.
Exposing a service using ingress
You can use the Tailscale Kubernetes operator to expose an ingress resource in your Kubernetes cluster to your tailnet.
Ingress resources only support TLS, and are only exposed over HTTPS. You must enable HTTPS on your tailnet.
Edit the ingress resource you want to expose to use the ingress class tailscale
:
- Set
spec.ingressClassName
totailscale
. - Set
tls.hosts
to the desired host name of the Tailscale node. Only the first label is used.
For example, to expose an ingress resource nginx
to your tailnet:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
spec:
defaultBackend:
service:
name: nginx
port:
number: 80
ingressClassName: tailscale
tls:
- hosts:
- nginx
The backend is HTTP by default. To use HTTPS on the backend, either set the port name to https
or the port number to 443
:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
spec:
defaultBackend:
service:
name: nginx
port:
name: https
ingressClassName: tailscale
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
name: https
port: 443
targetPort: 443
type: ClusterIP
A single ingress resource can be used to front multiple backend services:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress
spec:
ingressClassName: tailscale
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ui-svc
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: api-svc
port:
number: 80
Exposing a service to the public internet using ingress and Tailscale Funnel
You can also use the Tailscale Kubernetes operator to expose an ingress resource in your Kubernetes cluster to the public internet using Taiscale Funnel. To do so, add a tailscale.com/funnel: "true"
annotation:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: funnel
annotations:
tailscale.com/funnel: "true"
spec:
defaultBackend:
service:
name: funnel
port:
number: 80
ingressClassName: tailscale
tls:
- hosts:
- funnel
Removing a service
Any of the following actions remove a Kubernetes service you exposed from your tailnet:
- Delete the service entirely.
- If you are using
type=LoadBalancer
, removeloadBalancerClass=tailscale
or settype
toClusterIP
. - If you are using the
tailscale.com/expose
annotation, remove the annotation. - If you are using an ingress resource, remove the ingress resource or remove the service from the ingress resource.
Accessing the Kubernetes control plane using an API server proxy
You can use the Tailscale Kubernetes operator to expose and access the Kubernetes control plane (kube-apiserver) over Tailscale.
You can use the API server proxy with or without authentication headers. With authentication headers, when a user tries to access the Kubernetes control plane over Tailscale using an API server proxy, they will hit the kube-apiserver with the same user identity that they have in Tailscale. This is done by injecting an authentication header in the request. For example, alice@example.com
will have the user alice@example.com
in an API server proxy.
If you do not want to use Tailscale for authentication, but use an existing authentication mechanism instead, you can disable the use of authentication headers. This allows you to access the Kubernetes control plane over Tailscale, without using Tailscale for authentication.
Configuring the API server proxy
To use a Tailscale Kubernetes API server proxy, you need to enable HTTPS for your tailnet.
To configure the API server proxy:
1. In your Tailscale Kubernetes operator’s manifest file, add the following lines to the env
section:
name: APISERVER_PROXY
value: "true"
The kube-apiserver is automatically discovered by the operator.
2. Apply the changes from the example to your operator’s manifest file.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tailscale-auth-proxy
rules:
- apiGroups: [""]
resources: ["users"]
verbs: ["impersonate"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tailscale-auth-proxy
subjects:
- kind: ServiceAccount
name: operator
namespace: tailscale
roleRef:
kind: ClusterRole
name: tailscale-auth-proxy
apiGroup: rbac.authorization.k8s.io
3. Add an access rule in your tailnet’s policy file to grant access to the API server proxy over Tailscale.
4. Run the following command to grant a user the Kubernetes cluster-admin role in your cluster.
kubectl create clusterrolebinding --clusterrole cluster-admin --user alice@example alice@example
5. Use the tailscale configure kubeconfig <hostname-or-fqdn>
CLI command to configure your local kubeconfig
file to manage how to authenticate to kubectl as the Tailscale Kubernetes API server proxy.
To validate that API server proxy allows you to access the kube-apiserver over Tailscale:
6. Run kubectl config current-context
to verify that kubectl commands will
now use Tailscale context.
7. Run the kubectl get pods -A
command to run a test and verify that you have authorization.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-6b2x8 1/1 Running 0 28d
kube-system cilium-operator-759999b555-qbsrk 1/1 Running 3 (21d ago) 29d
kube-system coredns-7697897646-4vh2l 1/1 Running 0 29d
kube-system coredns-7697897646-rshwm 1/1 Running 0 29d
kube-system cpc-bridge-proxy-xksns 1/1 Running 0 29d
kube-system csi-do-node-k5snn 2/2 Running 0 29d
kube-system do-node-agent-n8nrr 1/1 Running 0 29d
kube-system konnectivity-agent-k846g 1/1 Running 0 29d
kube-system kube-proxy-lgzr9 1/1 Running 0 29d
tailscale operator-6b94c54478-n6tmc 1/1 Running 0 14d
Enabling tagged nodes to authenticate using the API server proxy
The API server proxy allows users to use their Tailscale identities to authenticate to the Kubernetes control plane.
To enable tagged nodes to authenticate to the Kubernetes control plane, create a Kubernetes RoleBinding for a group, and give the group cluster access.
For example, create a RoleBinding for the group tag:ci
:
subjects:
- kind: Group
name: "tag:ci"
apiGroup: rbac.authorization.k8s.io
Then, grant the group tag:ci
a ClusterRole
using a Kubernetes ClusterRoleBinding:
kubectl create clusterrolebinding --clusterrole cluster-admin --group "tag:ci" tag-ci
Disabling authentication headers
To use the API server proxy without authentication headers, in the env
section of your Kubernetes operator.yml file, set the value "noauth"
for the APISERVER_PROXY
:
name: APISERVER_PROXY
value: "noauth" # instead of true
Exposing a service to your cluster (cluster egress)
You can use the Tailscale Kubernetes operator to advertise a service external to your cluster which is on your Tailscale network.
This is done by deploying a proxy in the cluster, setting the service’s spec.externalName
to point to the proxy, and setting iptables rules for the proxy to direct incoming traffic to the DNS entry for the tailnet service. When a cluster workload attempts to reach the service, it is first directed to the proxy, which then redirects the traffic to the external service.
Exposing a tailnet service using annotations
You can expose a tailnet service to your cluster workloads using annotations and an external name. You can currently only expose tailnet services that use HTTP.
To expose a tailnet service to your cluster workloads:
- Create a Kubernetes Service of type ExternalName annotated with the Tailscale IP address of the service you want to make available:
apiVersion: v1
kind: Service
metadata:
annotations:
tailscale.com/tailnet-ip: 100.68.29.93 // Tailscale IP address
name: nginx // service name
spec:
externalName: unused // any value - will be overwritten by operator
type: ExternalName
Under metadata.annotations
, add the annotation
tailscale.com/tailnet-ip
with the Tailscale IP address for the tailnet service.
This can be either an IPv4 or IPv6 address, for either a Tailscale node or a route in a Tailscale subnet. This does not support IP ranges or Tailscale node names.
Under spec.externalName
, add any value. This needs to be set to pass Kubernetes validation, but can hold any placeholder value. It will be overwritten by the Tailscale Kubernetes operator with a DNS name that cluster workloads can use to reach the proxy.
- Wait for the Tailscale Kubernetes operator to deploy the proxy in the
tailscale
namespace, and to update thespec.externalName
of the Kubernetes Service. It will get set to the DNS name of the egress proxy that Tailscale operator creates.
Any cluster workload can now access the exposed Tailscale service using the annotated Kubernetes Service — calls to it will be routed to the proxy and forwarded to the right Tailscale node.
Validate the proxy is properly deployed
The proxy pod is deployed in the tailscale
namespace, and will have a name of the form ts-<annotated-service-name>-<random-string>
.
If there are issues reaching the external service, verify the proxy pod is properly deployed:
- Review the logs of the proxy pod.
- Review the logs of the operator. You can do this by running
kubectl logs deploy/operator --namespace tailscale
. The log level can be configured using theOPERATOR_LOGGING
environment variable in the operator’s manifest file. - Verify that the cluster workload is able to send traffic to the proxy pod in the
tailscale
namespace.
Exposing a service in one cluster to another cluster (cross-cluster connectivity)
You can use the Tailscale Kubernetes operator to expose a service in one cluster to another cluster. This is done by exposing the service on destination cluster A to the tailnet (cluster ingress), and connecting from a source service in cluster B to the tailnet (cluster egress) in order to access the service running in cluster A.
This will need to be configured for each ingress and egress pair of services. To set this up for access via ingress to a service in cluster A and routing via egrees from a service in cluster B:
- Set up ingress in cluster A for the service you wish to access.
- Expose the external service (running in cluster A) using its Tailscale IP address in cluster B with an annotation on the external service
Limitations
- Only development (“unstable”) builds are usable for now, as the operator depends on some changes that happened after the release of Tailscale v1.36.
- There are no deployment options other than applying the manifest file.
- There are no automated updates. The operator and proxy pods will not update automatically to newer Tailscale releases as they become available.
- There are no dashboards or metrics.
Cluster ingress
- Tags are only considered during initial provisioning. That is, editing
tailscale.com/tags
on an already exposed service doesn’t update the tags until you clean up and re-expose the service. - The requested machine name is only considered during initial provisioning. That
is, editing
tailscale.com/hostname
on an already exposed service doesn’t update the machine name until you clean up and re-expose the service. - Ingress to cluster services currently only supports hosts where netfilter can be configured via iptables.
API server proxy
- The API server proxy runs inside of the cluster. If your cluster is non-functional or is unable to schedule pods, you may lose access to the API server proxy.
Cluster egress
- When exposing a service to your cluster, any associated MagicDNS name will not resolve in-cluster. Instead, the name of the ExternalName service should be used to connect to the resource. As a result of this, if you use Tailscale to provision certificates you may see certificate name mismatch errors. We are working on this.
- Egress to external services supports using an IPv4 or IPv6 address for a single route in the
tailscale.com/tailnet-ip
annotation, but not IP ranges or node names. - Egress to external services currently only supports hosts where netfilter can be configured via iptables.
- Egress to external services currently only supports clusters where privileged pods are permitted (i.e., GKE Autopilot is not supported).