Kubernetes operator
The Tailscale Kubernetes operator lets you:
- Expose
Services
in your Kubernetes cluster to your Tailscale network (known as a tailnet) - Securely connect to the Kubernetes control plane (kube-apiserver) via an API server proxy, with or without authentication
- Egress from a Kubernetes cluster to an external service on your tailnet
Setting up the Kubernetes operator
-
In your tailnet policy file, create the ACL tags
tag:k8s-operator
andtag:k8s
, and maketag:k8s-operator
an owner oftag:k8s
. If you want yourServices
to be exposed with tags other than the defaulttag:k8s
, create those as well and maketag:k8s-operator
an owner."tagOwners": { "tag:k8s-operator": [], "tag:k8s": ["tag:k8s-operator"], }
-
Create an OAuth client in the OAuth clients page of the admin console. Create the client with
Devices
write scope and the tagtag:k8s-operator
. -
Download the Tailscale Kubernetes operator manifest file from the tailscale/tailscale repo.
-
Edit your version of the manifest file:
- Find
# SET CLIENT ID HERE
and replace it with your OAuth client ID. - Find
# SET CLIENT SECRET HERE
and replace it with your OAuth client secret. The OAuth client secret is case-sensitive.
For both the client ID and secret, quote the value, to avoid any potential yaml misinterpretation of unquoted strings. For example, use:
client_id: "k123456CNTRL" client_secret: "tskey-client-k123456CNTRL-abcdef"
instead of:
client_id: k123456CNTRL client_secret: tskey-client-k123456CNTRL-abcdef
- Find
-
Apply the edited file to your Kubernetes cluster:
kubectl apply -f manifest.yaml
This creates the “tailscale” namespace in your cluster, and deploys the Tailscale operator within it.
-
Verify that the Tailscale operator has joined your tailnet. Open the Machines page of the admin console and look for a node named tailscale-operator, tagged with the
tag:k8s-operator
tag. It may take a minute or two for the operator to join your tailnet, due to the time required to download and start the container image in Kubernetes.
Deploying the operator using a Helm chart
You can deploy the Tailscale Kubernetes operator by using a Helm chart. The Tailscale Kubernetes operator Helm chart is available from the tailscale/tailscale repo.
-
In your tailnet policy file, create the ACL tags
tag:k8s-operator
andtag:k8s
, and maketag:k8s-operator
an owner oftag:k8s
. If you want your services to be exposed with tags other than the defaulttag:k8s
, create those as well and maketag:k8s-operator
an owner."tagOwners": { "tag:k8s-operator": [], "tag:k8s": ["tag:k8s-operator"], }
-
Create an OAuth client in the OAuth clients page of the admin console. Create the client with
Devices
write scope and the tagtag:k8s-operator
. -
Download the Tailscale Kubernetes operator Helm chart from the tailscale/tailscale repo.
-
Edit the
oauth-secret.yaml
template and set your OAuth client ID and client secret.The OAuth client secret is case-sensitive.
apiVersion: v1 kind: Secret metadata: name: operator-oauth namespace: tailscale stringData: client_id: "k123456CNTRL" client_secret: "tskey-client-k123456CNTRL-abcdef"
-
Set any other desired configurations in
values.yaml
. -
Package and run the Helm chart:
helm install operator ./cmd/k8s-operator/deploy/chart -n tailscale -f <path-to-values-file>
Exposing a Service
to your tailnet (cluster ingress)
You can use the Tailscale Kubernetes operator to expose a Kubernetes Service
to your Tailscale network in three ways:
- Make it a
LoadBalancer
type with thetailscale
loadBalancerClass
- Annotate an existing service
- Create an
Ingress
resource fronting aService
Create a new Kubernetes Service
of type Load Balancer
Create a new Kubernetes Service
of type load balancer:
- Set
spec.type
toLoadBalancer
. - Set
spec.loadBalancerClass
totailscale
.
Once provisioning is complete, the Service
status will show the
fully-qualified domain name of the Service
in your tailnet. You can view the
Service
status by running kubectl get service <service name>
.
You should also see a new node with that name appear in the Machines page of the admin console.
Exposing a Service
using annotations
If the Service
you want to expose already exists, you can
expose it to Tailscale using object annotations.
Edit the Service
and under metadata.annotations
, add the annotation
tailscale.com/expose
with the value "true"
. Note that "true"
is quoted
because annotation values are strings, and an unquoted true
will be
incorrectly interpreted as a boolean.
In this mode, Kubernetes doesn’t tell you the Tailscale machine name. You can look
up the node in the Machines of the admin console to learn
its machine name. By default, the machine name of an exposed Service
is
<k8s-namespace>-<k8s-servicename>
.
Using a custom machine name
If you want the Service
to have a machine name other than the default
<k8s-namespace>-<k8s-servicename>
, you can provide your own machine name by
setting the tailscale.com/hostname
annotation on the Service, with your
desired machine name as the value.
Machine names are subject to the constraints of DNS: they can be up to 63 characters
long, must start and end with a letter, and consist of only letters, numbers,
and -
.
Customizing ACL tags
By default, Services
join your tailnet tagged with the ACL tag tag:k8s
. You can use a
different tag or tags by setting the tailscale.com/tags
annotation on the
Service, with a comma-separated list of the desired tags.
For example, setting tailscale.com/tags = tag:foo,tag:bar
will result in the
tailnet node having the tags tag:foo
and tag:bar
.
The Tailscale operator must be a tag owner of all the specified tags: if you want
to expose a Service
with tag:foo,tag:bar
, the tagOwners
section of the
tailnet policy file must list tag:k8s-operator
as one
of the owners of both tag:foo
and tag:bar
.
Exposing a Service
using Ingress
You can use the Tailscale Kubernetes operator to expose an Ingress
resource in your Kubernetes cluster to your tailnet.
When configured using an Ingress
resource, you also get the ability to identify callers using HTTP headers
injected by the Ingress
proxy.
Ingress
resources only support TLS, and are only exposed over HTTPS. You must enable HTTPS on your tailnet.
Edit the Ingress
resource you want to expose to use the Ingress
class tailscale
:
- Set
spec.ingressClassName
totailscale
. - Set
tls.hosts
to the desired host name of the Tailscale node. Only the first label is used.
For example, to expose an Ingress
resource nginx
to your tailnet:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
spec:
defaultBackend:
service:
name: nginx
port:
number: 80
ingressClassName: tailscale
tls:
- hosts:
- nginx
The backend is HTTP by default. To use HTTPS on the backend, either set the port name to https
or the port number to 443
:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
spec:
defaultBackend:
service:
name: nginx
port:
name: https
ingressClassName: tailscale
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- name: https
port: 443
targetPort: 443
type: ClusterIP
A single Ingress
resource can be used to front multiple backend Services
:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress
spec:
ingressClassName: tailscale
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ui-svc
port:
number: 80
- path: /api
pathType: Prefix
backend:
service:
name: api-svc
port:
number: 80
Exposing a Service
to the public internet using Ingress
and Tailscale Funnel
You can also use the Tailscale Kubernetes operator to expose an Ingress
resource in your Kubernetes cluster to the public internet using Tailscale Funnel. To do so, add a tailscale.com/funnel: "true"
annotation:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: funnel
annotations:
tailscale.com/funnel: "true"
spec:
defaultBackend:
service:
name: funnel
port:
number: 80
ingressClassName: tailscale
tls:
- hosts:
- funnel
Removing a Service
Any of the following actions remove a Kubernetes Service
you exposed from your tailnet:
- Delete the
Service
entirely - If you are using the
tailscale.com/expose
annotation, remove the annotation - If you are using an
Ingress
resource, delete it or change or unsetspec.ingressClassName
Service
’s Tailscale node
in the admin console does not clean up the Kubernetes state
associated with that Service
.Accessing the Kubernetes control plane using an API server proxy
You can use the Tailscale Kubernetes operator to expose and access the Kubernetes control plane (kube-apiserver) over Tailscale.
You can use the API server proxy with or without authentication headers. With authentication headers, when a user tries to access the Kubernetes control plane over Tailscale using an API server proxy, they will hit the kube-apiserver with the same user identity that they have in Tailscale. This is done by injecting an authentication header in the request. For example, alice@example.com
will have the user alice@example.com
in an API server proxy.
If you do not want to use Tailscale for authentication, but use an existing authentication mechanism instead, you can disable the use of authentication headers. This allows you to access the Kubernetes control plane over Tailscale, without using Tailscale for authentication.
Configuring the API server proxy
To use a Tailscale Kubernetes API server proxy, you need to enable HTTPS for your tailnet.
To configure the API server proxy:
1. In your Tailscale Kubernetes operator’s manifest file, add the following lines to the env
section:
name: APISERVER_PROXY
value: "true"
The kube-apiserver is automatically discovered by the operator.
2. Apply the changes from the example to your operator’s manifest file.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tailscale-auth-proxy
rules:
- apiGroups: [""]
resources: ["users"]
verbs: ["impersonate"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tailscale-auth-proxy
subjects:
- kind: ServiceAccount
name: operator
namespace: tailscale
roleRef:
kind: ClusterRole
name: tailscale-auth-proxy
apiGroup: rbac.authorization.k8s.io
3. Add an access rule in your tailnet’s policy file to grant access to the API server proxy over Tailscale.
4. Run the following command to grant a user the Kubernetes cluster-admin role in your cluster.
kubectl create clusterrolebinding --clusterrole cluster-admin --user alice@example alice@example
5. Run the following CLI command to configure your kubeconfig
for authentication with kubectl
via the Tailscale Kubernetes API server proxy: tailscale configure kubeconfig <operator-hostname>
. By default, the hostname for the operator node is tailscale-operator.
To validate that API server proxy allows you to access the kube-apiserver over Tailscale:
6. Run kubectl config current-context
to verify that kubectl commands will
now use Tailscale context.
7. Run the kubectl get pods -A
command to run a test and verify that you have authorization.
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-6b2x8 1/1 Running 0 28d
kube-system cilium-operator-759999b555-qbsrk 1/1 Running 3 (21d ago) 29d
kube-system coredns-7697897646-4vh2l 1/1 Running 0 29d
kube-system coredns-7697897646-rshwm 1/1 Running 0 29d
kube-system cpc-bridge-proxy-xksns 1/1 Running 0 29d
kube-system csi-do-node-k5snn 2/2 Running 0 29d
kube-system do-node-agent-n8nrr 1/1 Running 0 29d
kube-system konnectivity-agent-k846g 1/1 Running 0 29d
kube-system kube-proxy-lgzr9 1/1 Running 0 29d
tailscale operator-6b94c54478-n6tmc 1/1 Running 0 14d
Enabling tagged nodes to authenticate using the API server proxy
The API server proxy allows users to use their Tailscale identities to authenticate to the Kubernetes control plane.
To enable tagged nodes to authenticate to the Kubernetes control plane, create a Kubernetes RoleBinding for a group, and give the group cluster access.
For example, create a RoleBinding for the group tag:ci
:
subjects:
- kind: Group
name: "tag:ci"
apiGroup: rbac.authorization.k8s.io
Then, grant the group tag:ci
a ClusterRole
using a Kubernetes ClusterRoleBinding:
kubectl create clusterrolebinding --clusterrole cluster-admin --group "tag:ci" tag-ci
Disabling authentication headers
To use the API server proxy without authentication headers, in the env
section of your Kubernetes operator.yml file, set the value "noauth"
for the APISERVER_PROXY
:
name: APISERVER_PROXY
value: "noauth" # instead of true
Exposing a Service
to your cluster (cluster egress)
You can use the Tailscale Kubernetes operator to advertise a Service
external to your cluster which is on your Tailscale network.
This is done by deploying a proxy in the cluster, setting the Service
’s spec.externalName
to point to the proxy, and setting iptables rules for the proxy to direct incoming traffic to the DNS entry for the tailnet service. When a cluster workload attempts to reach the Service, it is first directed to the proxy, which then redirects the traffic to the external Service.
Exposing a tailnet service using annotations
You can expose a tailnet service to your cluster workloads using annotations and an external name. You can currently only expose tailnet services that use HTTP.
To expose a tailnet service to your cluster workloads:
-
Create a Kubernetes
Service
of type ExternalName annotated with the Tailscale IP address of theService
you want to make available:apiVersion: v1 kind: Service metadata: annotations: tailscale.com/tailnet-ip: 100.68.29.93 // Tailscale IP address name: nginx // service name spec: externalName: unused // any value - will be overwritten by operator type: ExternalName
Under
metadata.annotations
, add the annotationtailscale.com/tailnet-ip
with the Tailscale IP address for the tailnet service. This can be either an IPv4 or IPv6 address, for either a Tailscale node or a route in a Tailscale subnet. This does not support IP ranges or Tailscale node names.Under
spec.externalName
, add any value. This needs to be set to pass Kubernetes validation, but can hold any placeholder value. It will be overwritten by the Tailscale Kubernetes operator with a DNS name that cluster workloads can use to reach the proxy. -
Wait for the Tailscale Kubernetes operator to deploy the proxy in the
tailscale
namespace, and to update thespec.externalName
of the Kubernetes Service. It will get set to the DNS name of theEgress
proxy that the Tailscale operator creates.
Any cluster workload can now access the exposed Tailscale service using the annotated Kubernetes Service
— calls to it will be routed to the proxy and forwarded to the right Tailscale node.
Validate the proxy is properly deployed
The proxy pod is deployed in the tailscale
namespace, and will have a name of the form ts-<annotated-service-name>-<random-string>
.
If there are issues reaching the external service, verify the proxy pod is properly deployed:
- Review the logs of the proxy pod
- Review the logs of the operator. You can do this by running
kubectl logs deploy/operator --namespace tailscale
. The log level can be configured using theOPERATOR_LOGGING
environment variable in the operator’s manifest file. - Verify that the cluster workload is able to send traffic to the proxy pod in the
tailscale
namespace
Exposing a Service
in one cluster to another cluster (cross-cluster connectivity)
You can use the Tailscale Kubernetes operator to expose a Service
in one cluster to another cluster. This is done by exposing the Service
on destination cluster A to the tailnet (cluster ingress), and connecting from a source Service
in cluster B to the tailnet (cluster egress) in order to access the Service
running in cluster A.
This will need to be configured for each Ingress
and Egress
pair of Services
. To set this up for access via ingress to a Service
in cluster A and routing via egress from a Service
in cluster B:
- Set up
Ingress
in cluster A for theService
you wish to access. - Expose the external
Service
(running in cluster A) using its Tailscale IP address in cluster B with an annotation on the externalService
Limitations
- The Helm chart is not currently hosted. We are working on this.
- There are no deployment options other than applying the manifest file
- There are no automated updates. The operator and proxy pods will not update automatically to newer Tailscale releases as they become available.
- There are no dashboards or metrics
Cluster ingress
- Tags are only considered during initial provisioning. That is, editing
tailscale.com/tags
on an already exposedService
doesn’t update the tags until you clean up and re-expose the Service. - The requested machine name is only considered during initial provisioning. That
is, editing
tailscale.com/hostname
on an already exposedService
doesn’t update the machine name until you clean up and re-expose the Service.
API server proxy
- The API server proxy runs inside of the cluster. If your cluster is non-functional or is unable to schedule pods, you may lose access to the API server proxy.
Cluster egress
- When exposing a service to your cluster, any associated MagicDNS name will not resolve in-cluster. Instead, the name of the ExternalName service should be used to connect to the resource. As a result of this, if you use Tailscale to provision certificates you may see certificate name mismatch errors. We are working on this.
- Egress to external services supports using an IPv4 or IPv6 address for a single route in the
tailscale.com/tailnet-ip
annotation, but not IP ranges or node names. - Egress to external services currently only supports clusters where privileged pods are permitted (i.e., GKE Autopilot is not supported).