Mike Stefaniak
Over the past few months, we’ve chosen to tackle some of the most stubborn issues of the Tailscale Kubernetes Operator to improve performance, connectivity, and scale:
- Cross-cluster connections being routed through DERP servers
- IPv6 connectivity
- Let’s Encrypt rate limiting
While IPv4/IPv6 connectivity and Let’s Encrypt rate limits are not completely solved problems, our optimizations and fixes contained in the 1.102 release should significantly reduce how often users are blocked from connectivity and deployment. For multi-cluster environments, the new in-cluster peer relays will reduce the configuration burden of direct multi-cluster connectivity, even in difficult network scenarios such as VPCs.
In-cluster Tailscale Peer Relays
In many cases, Kubernetes clusters are deployed into virtual private cloud (VPC) networks and are unable to communicate directly with each other without setting up explicit rules in the network firewall. To enable communication regardless of firewall rules, Tailscale uses a DERP server as a relay to establish a connection. Connection speeds are much lower than a direct connection, so you would typically set up a Tailscale Peer Relay to help facilitate a direct connection to improve performance.
Before 1.102, you could technically deploy a peer relay inside or outside your cluster to enable direct connections between clusters. However, we have heard from you that setting up a peer relay to work with Kubernetes clusters can be a headache, requiring manual configuration of a Tailscale client completely separate from the Operator.
With Tailscale Peer Relays In 1.102, you can now deploy peer relays in your clusters using our new custom resource, PeerRelay, for the Kubernetes Operator. When defined, the Kubernetes Operator will deploy and maintain each peer relay for you, with little configuration:
apiVersion: tailscale.com/v1alpha1
kind: PeerRelay
metadata:
name: my-relay
spec:
replicas: 2You can also supply additional configuration such as tags, proxyClass inheritance, and tailnet by referencing our custom resource API.
IPv6 connectivity
IPv6 connectivity, especially enabling connectivity between IPv4 clients and IPv6 workloads, can be tricky in Kubernetes. This has also been true of the Tailscale Kubernetes Operator, as we rely on iptables and nftables to perform routing within the cluster. If you are not running a dual-stack cluster with both IPv4 and IPv6, you cannot route with the IP set you don’t have, and connections from clients can never be established.
Even given those limitations, there was support missing around IPv6, which we’ve included in this release. Firstly, we added support for the ProxyGroup Egress to egress to IPv6 destinations. Previously, only an IPv4 address was a valid destination, which included Tailscale services’ IPv4 address.
However, if you needed to connect to a destination that only supported IPv6 on incoming connections, such as our 4via6 subnet routers, you would not be able to establish a connection. Most of Tailscale is compatible with both IPv4 and IPv6, but on large networks with isolated sub-networks, you may run into cases where your IPv4 addresses overlap. In this case, a 4via6 subnet router can bridge that connection by exposing an IPv6 address for an IPv4 destination. As of 1.102, we now support this functionality in the Kubernetes Operator, enabling connections where IPv4 ranges overlap.
Let’s Encrypt rate limit optimizations
When you use Tailscale, every node in your Tailscale network (tailnet) is given a MagicDNS name. In order for those MagicDNS names to be used for TLS connections, each node needs its own certificate for the hostname it was assigned. These certificates are issued by the open Certificate Authority, Let’s Encrypt.
As a free public service run by the Internet Security Research Group (ISRG), there are various rate limits placed on its usage to ensure the service remains available for everyone. As tailnets become larger, and especially as large automated deployments become more common, it’s likely you may run into some of them. The most common is a 50-certificate per registered domain per week limit that is reset only after seven days. This issue can be crippling when trying to roll Tailscale out across a large network in a short time period.
While you can request a rate limit override from Let’s Encrypt, we wanted to provide a smoother experience before you need to increase your limit, while also being better citizens of the Let’s Encrypt platform. So we’ve made a series of small optimizations to how we acquire certificates in the Kubernetes Operator:
- Certificate renewal retries now follow a backoff schedule instead of a fixed interval.
- When available,
Retry-Afterheaders are now used, which avoids tight retry loops that made rate-limit backoffs worse. - The per-attempt cert issuance timeout is increased to 30 minutes to avoid premature timeouts on ACME challenges.
- Cert issuance for multiple domains now runs in parallel instead of one at a time, preventing the provisioning of many domains from stalling each cert behind the previous one.
- During Ingress deletion, VIPServices are now prevented from acquiring new certificates, avoiding wasting Let’s Encrypt rate-limit quota. This is especially prevalent in ephemeral deployments.
We have one more optimization coming in 1.104 that will improve how certificate renewals are counted towards your rate limits. Even with all these optimizations, limits can still be encountered. We are also working on another solution that I hope to be able to share soon. In the meantime, if you are still running into rate limit issues with Let’s Encrypt, we recommend formally requesting a rate limit override from Let’s Encrypt for your tailnet.
What’s next?
Our next release cycle for the Tailscale Kubernetes Operator will be primarily focused on improving our end-to-end testing infrastructure, Let’s Encrypt account key sharing, and fixing a few stubborn bugs. In the background, we’re working on longer-term solutions for Let’s Encrypt limits, installation for OpenShift and GKE, and planning our biggest improvement since the operator’s inception.
Today, we have a few common problems that the operator struggles with, primarily due to its architecture:
- Egress to any tailnet resource requires explicit setup, per endpoint and per namespace.
- Access controls can only target the ProxyGroup rather than the workload.
- Using IPv4 and IPv6 interchangeably between clients and clusters is difficult.
- Cluster workloads bypass connectors without a steering mechanism.
To solve these problems, we need a deeper way to route traffic, beyond what iptables and nftables are capable of. This means for ingress and egress, a Container Network Interface (CNI) is required. We also need a way to assign identities to workloads within the cluster without resorting to sidecars. All of this leads us to a new architecture: something between a traditional gateway proxy and a full sidecar mesh.
One problem we need to tackle first is the requirement to create a new tag in tagOwners for every new tag created in the tailnet. Our goal is for every workload in Kubernetes to have its own identity for access controls. Today, that means tagging the workload with a Tailscale tag to grant access via the ACL. Typically for application teams, this requires a ticket to your Tailscale admin to create the new tag for you. That pattern becomes cumbersome when you have automated deployments, even outside of Kubernetes, constantly requiring new tags.
That’s why we are building pattern-based tag ownership into Tailscale’s ACL. This will allow you to specify ownership for a pattern of tags, effectively granting ownership over a predetermined scope for application teams and operators. Once granted, the owner will be able to create as many tags as needed under the pattern specified in the tagOwners section. If you have ephemeral or automated infrastructure deployment as a service within your organization, even outside of Kubernetes, this change is for you.
After pattern-based tag ownership, we’ll be building a new architecture for the operator, aimed at solving all the problems outlined above. More details on what that looks like soon!
Resources
For a complete list of changes in the Kubernetes Operator and Tailscale as a whole, see our changelog. If you’re looking to get started with Tailscale’s Kubernetes Operator, check out our installation guide or the more complete quickstart guide for a deeper look.
If you’re looking to report an issue you have, log a feature request, or get involved with helping us build Tailscale for everyone, check out our repository on GitHub.
