Multi-cluster Ingress
The following example shows the process for exposing an application, deployed across two different clusters, to your tailnet. A single MagicDNS name routes each Tailscale client to their closest cluster.
- A
ProxyGroup
in each cluster will manage a set of highly available proxies. - A Tailscale
Ingress
in each cluster will configure proxies to route tailnet traffic to the in-cluster app instance. - A single Tailscale Service ensures that tailnet traffic for the app is routed to a proxy in the closest cluster.
Multi-cluster Ingress uses a new Tailscale feature, Tailscale Services, currently in alpha.
Contact support to enable it for your tailnet and start using multi-cluster Ingress
.
This tutorial covers exposing multi-cluster applications using Tailscale's application layer ingress via Ingress
resource.
You can use the same approach to expose multi-cluster applications using Tailscale's network layer ingress via Service
resource.
Setup
-
Ensure you have two clusters available, preferably in different geographical regions, so you can test clients in different regions being routed to the closest cluster.
-
Update
tagOwners
in your tailnet policy file to let the Kubernetes Operator to createProxyGroup
proxies with the tagtag:ingress-proxies
and Tailscale Services with the tagtag:internal-apps
:"tagOwners": { "tag:k8s-operator": [], "tag:internal-apps": ["tag:k8s-operator"], "tag:ingress-proxies": ["tag:k8s-operator"], ... },
-
Update
autoApprovers
in your tailnet policy file to let theProxyGroup
proxies to advertise Tailscale Services with the tagtag:internal-apps
:"autoApprovers": { "services": { "tag:ingress-proxies": ["tag:internal-apps"], }, }
-
Update grants to allow tailnet clients access to the cluster apps you want to expose. There will be a Tailscale Service created for each app. The Service's ACL tag can be used to grant permissions to access the app. For example, to grant the Tailscale user group
group:engineering
access to apps tagged withtag:internal-apps
:"grants": [ { "src": ["group:engineering"], "dst": ["tag:internal-apps:80","tag:internal-apps:443"], "ip": ["tcp:80","tcp:443"], }, { "src": ["group:engineering"], "ip": ["icmp:22"], "dst": ["tag:ingress-proxies"], }, ... ]
The requirement to grant access to the devices to access a Tailscale Service is a temporary limitation.
-
Ensure your client accepts routes. Clients other than Linux accept routes by default.
-
Ensure your tailnet has regional routing enabled.
In each cluster
-
Create OAuth client credentials for the Kubernetes Operator with write scopes for
VIP Services
,Devices Core
, andAuth Keys
. -
Add
https://pkgs.tailscale.com/helmcharts
to your local Helm repositories:helm repo add tailscale https://pkgs.tailscale.com/helmcharts
-
Update your local Helm cache:
helm repo update
-
Install the operator:
helm upgrade --install operator tailscale/tailscale-operator \ -n tailscale --create-namespace \ --set oauth.clientId=<id> \ --set oauth.clientSecret=<key> \ --wait
-
Apply the following
ProxyGroup
andProxyClass
resources to create a set of Tailscale devices that will act as proxies:While initially deploying
Ingress
resources, we highly recommend you use Let's Encrypt's staging environment to avoid production's tighter rate limits. The following example uses Let's Encrypt's staging environment, but you can unsetuseLetsEncryptStagingEnvironment
once you are ready to provision production HTTPS certificates.apiVersion: tailscale.com/v1alpha1 kind: ProxyGroup metadata: name: ingress-proxies spec: type: ingress hostnamePrefix: eu-west replicas: 2 tags: ["tag:ingress-proxies"] proxyClass: letsencrypt-staging --- apiVersion: tailscale.com/v1alpha1 kind: ProxyClass metadata: name: letsencrypt-staging spec: useLetsEncryptStagingEnvironment: true
-
(Optional) Wait for the ProxyGroup to become ready:
kubectl wait proxygroup ingress-proxies --for=condition=ProxyGroupReady=true
For the above
ProxyGroup
the operator creates aStatefulSet
with two replicas. Each replicaPod
runs a Tailscale device with a tagtag:ingress-proxies
and hostname with prefixeu-west-
-
(Optional) if you don't have an existing workload to route traffic to, deploy
nginx
as a sample application:apiVersion: v1 kind: Pod metadata: labels: run: nginx name: nginx spec: containers: - name: nginx image: nginx --- apiVersion: v1 kind: Service metadata: labels: run: nginx name: nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: run: nginx
-
Apply the following
Ingress
resource to expose nginx with the DNS namenginx.<your-tailnet-domain>
:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx annotations: tailscale.com/proxy-group: ingress-proxies tailscale.com/tags: "tag:internal-apps" spec: tls: - hosts: - nginx rules: - http: paths: - backend: service: name: nginx port: number: 80 pathType: Prefix path: / ingressClassName: tailscale
-
(Optional) Wait for the HTTPS endpoints to be ready:
kubectl wait --timeout=80s ingress nginx --for=jsonpath='{.status.loadBalancer.ingress[0].ports[0].port}'=443
For the above
Ingress
resource, the Kubernetes Operator ensures that a Tailscale Service namedsvc:nginx
exists for the tailnet and that proxies route tailnet traffic for the Tailscale Service to the KubernetesService
nginx
. The Tailscale Service's name is determined fromingress.spec.tls[0].hosts[0]
field. The Tailscale Service will be created if it does not already exist.Ingress
resources in multiple clusters can define backends for a single Tailscale Service by using the sameingress.spec.tls[0].hosts[0]
field. -
Repeat the steps from this section for your second cluster (optionally changing
proxyGroup.spec.hostnamePrefix
field value).
You can expose any number of Ingress
resources on the same ProxyGroup
(limited by resource consumption).
You can not create multiple Ingress
resources for the same Tailscale Service in the same cluster.
Test
-
Check the MagicDNS name for the created Ingress:
kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE nginx tailscale * nginx.tailxyz.ts.net 443 3h18m
-
Test that traffic flows for the MagicDNS name:
curl -ksS https://nginx.tailxyz.ts.net <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ...
(Optional) Cluster-specific DNS names
When using a single DNS name for Ingress
resources deployed across multiple clusters, clients will automatically route to their closest cluster.
If you want to also create cluster-specific DNS names, you can deploy additional Ingress
resources that are specific to each cluster.
Apply a cluster-specific Ingress
such as:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-eu
annotations:
tailscale.com/proxy-group: ingress-proxies
tailscale.com/tags: "tag:internal-apps"
spec:
tls:
- hosts:
- nginx-eu
rules:
- http:
paths:
- backend:
service:
name: nginx
port:
number: 80
pathType: Prefix
path: /
ingressClassName: tailscale
TLS certificates
The cluster app behind the Ingress
is exposed to tailnet over HTTPS with a MagicDNS name constructed using the value of ingress.spec.tls[0].hosts[0]
field and the tailnet domain.
A single TLS certificate is issued for the Ingress
and shared amongst the ProxyGroup
replicas.
TLS certificates are issued from Let's Encrypt's production environment by default.
LetsEncrypt imposes rate limits to certificate issuance.
Tailscale's ts.net
domain is in public suffix list.
This means that Let'sEncrypt considers a tailnet's top level domain (for example, tailxyz.ts.net
) a registered domain and rate limits are applied to each tailnet individually.
For example:
-
50 certificates per week per domain. This means no more than 50 unique Ingresses per week per tailnet. Note that any other certs issued for tailnet devices also count towards this limit.
-
5 certificates for the same hostname per week. This means that the number of clusters that can expose the same app with the same DNS name is limited to 5 per week.
We highly recommend testing using Let's Encrypt's staging environment to avoid tighter rate limits until you are ready to deploy to production.
Using Let's Encrypt's staging environment
You can use the staging environment with a ProxyGroup
and ProxyClass
such as the following:
apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
name: ingress-proxies
spec:
type: ingress
replicas: 2
tags: ["tag:ingress-proxies"]
proxyClass: letsencrypt-staging
---
apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
name: letsencrypt-staging
spec:
useLetsEncryptStagingEnvironment: true
For the above configuration, the operator will create a ProxyGroup
that always uses Let's Encrypt's staging endpoint to issue certificates for any Tailscale Ingress
DNS names.
Advertising an HTTP endpoint
You can optionally enable an HTTP endpoint on port 80 in addition to the HTTPS endpoint on port 443.
This may be helpful if you want the Ingress
to still be available when an HTTPS certificate cannot be issued due to rate limits or other failure cases.
To enable an HTTP endpoint, add a tailscale.com/http-endpoint: enabled
annotation to your Ingress
.
Note that:
-
If an
Ingress
does not have an HTTP endpoint enabled, the proxies that advertise thisIngress
will only be considered healthy once the certificate issuance has succeeded. For example, if you haveIngress
resources that expose appnginx
with DNS namenginx.tailxyx.ts.net
in clustersus-east
,us-west
andeu-west
and certificate issuance only succeeds inus-west
andeu-west
, tailnet traffic fornginx.tailxyz.ts.net
will never be routed tous-east
. The proxies inus-east
will retry issuance and once it succeeds, they will be considered healthy and tailnet traffic will start getting routed tous-east
. -
If an
Ingress
does have an HTTP endpoint enabled, the proxies that advertise thisIngress
will be considered healthy, even if certificate issuance has failed. So, if you haveIngress
resources that expose appnginx
in clusters inus-east
,us-west
andeu-west
, and certificate issuance only succeeds inus-west
andeu-west
, tailnet traffic will still be routed to all three clusters, but only the HTTP endpoint is guaranteed to be healthy. The proxies inus-east
will retry issuance and once it succeeds, all clients will be able to reach the app via HTTPS in any cluster.
Debugging
Debugging Ingress
resource failure
-
Take a look at operator's logs.
-
Take a look at the proxies: For a
ProxyGroup
namedingress-proxies
:kubectl get pod -n tailscale -l tailscale.com/parent-resource="ingress-proxies",tailscale.com/parent-resource-type="proxygroup" NAME READY STATUS RESTARTS AGE ingress-proxies-0 1/1 Running 0 2d22h ingress-proxies-1 1/1 Running 0 2d22h
kubectl logs ingress-proxies-0 -n tailscale -c tailscale boot: 2025/03/28 14:20:43 Starting tailscaled ...
Validate to which cluster a tailnet client's requests are routed
-
Find the MagicDNS name by which Ingress is exposed to tailnet. This will be the DNS name of the corresponding Tailscale Service.
kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE nginx tailscale * nginx.tailxyz.ts.net 443 3h18m
-
Find the tailnet IP address of the Tailscale Service:
dig nginx.tailxyz.ts.net +short 100.100.126.127
If the Tailscale Service's IP address can not be resolved, its creation might have failed. Check the operator's logs.
-
Find which proxy the tailnet client uses as a backend for the Tailscale Service. Each tailnet client's requests for a Tailscale Service will be routed to a proxy in the closest cluster. You can run the following commands on a client to find which proxy it uses as a backend for a Tailscale Service:
tailscale status --json | jq '.Peer | to_entries[] | select(.value.AllowedIPs[] | contains("100.100.126.127/32")) | .value | {DNSName, ID}' { "DNSName": "ingress-proxies-eu-0.tailxyz.ts.net.", "ID": "n9Ch5VvNug11CNTRL" }
The first part of DNS name is either
proxyGroup.spec.hostnamePrefix
or theProxyGroup
name - you can use that to identify which cluster the tailnet client's traffic will be routed to.If
tailscale status --json
does not contain any results for the given tailnet IP address then the client likely does not have permissions to access the Tailscale Service.
Best practices
-
Use Let's Encrypt's staging environment for initial testing.
-
Set a different
proxygroup.spec.hostnamePrefix
field values forProxyGroup
resources in different clusters (or simply name theProxyGroup
resources differently). This will ensure that proxies in different clusters can be easily identified by the Tailscale hostname.
Current limitations
-
Exposing a new
Ingress
resource takes up to a minute to become available. We are working to make this faster. -
The current access model requires explicitly defining access for each app in ACLs. We are working on allowing tag based access.
-
When exposing
Ingress
resources with the same DNS name concurrently, you may hit transient failures due to DNS challenge errors. We are working on fixing this. -
Currently Let's Encrypt's staging endpoint can only be enabled for the
ProxyGroup
as a whole and not for individualIngress
resources. -
Currently you must ensure that multi-cluster
Ingress
resources for the same app all have the same tag and all either expose an HTTP endpoint or not. ApplyingIngress
resources for the same app, but with different tags or different HTTP endpoint settings, might result in the operator instances in the different clusters continously attempting to reconcile the Tailscale Servcie.