Tailscale is amazing. But you already knew that, right? There’s nothing more satisfying than being able to set up a secure network in seconds, almost like magic — except maybe realizing it’s Friday when you thought it was Thursday, but I digress.
Being a relatively new product, Tailscale is still adding features to make it even easier to use. One of the most requested features from both our enterprise customers as well as individual users are notifications for events happening in your tailnet, such as when new nodes are added or need to be authorized. Before Tailscale introduced the new feature I’m about to mention (shh… I know you saw it in the title, but just pretend you didn’t for a second), there wasn’t really a way for the admin of a tailnet to know if something had changed without constantly stalking the admin console for new warning badges on machines, or scrolling through the configuration audit logs for updates.
During my internship at Tailscale this past summer, I set out to fill this notification gap. (“I” meaning me, Laura the intern, not to be confused with the lovely individual of the same name who has been writing the Tailscale newsletter every month.) As a result of my (and many other peoples’) summer-long efforts, Tailscale now allows you to configure webhooks to notify you of specific kinds of events in your tailnet.
👋 Hi! I’m Laura
Before I begin, I’d like to share a little bit about myself. I’m a fourth year software engineering student at the University of Waterloo who loves designing, building, and automating systems. I love learning how to use new tools and technologies in hopes of creating something that can benefit people and the world around me. When I first learned about Tailscale, I was completely blown away by how the company was able to create a VPN that was both painless to set up and incredibly secure. It was an honor for me to be able to contribute to such an amazing product over the course of my internship, and I am so excited to be able to share my work with you and make your experience using Tailscale even better!
How Tailscale webhooks work
Now back to our regularly scheduled programming. The new feature that we’ve developed is none other than (drumroll please) … webhooks!
In general, webhooks are a method of providing push notifications that are triggered when certain events occur in a web application. They are essentially reverse APIs, requiring the user to set up a destination URL to which the webhook provider delivers requests containing event data. Afterward, when an event occurs, the webhook provider will then make a POST request to the destination URL to deliver the webhook to the user.
So how does a Tailscale webhook actually get sent to me?
I’m glad you asked; it’s really not that difficult of a process. First, create a webhook endpoint from the admin console. Next, choose the notification events you want to subscribe to and submit the form. After that, event triggers will recognize that events have to be sent to your endpoint. Event triggers are essentially functions in code on the Tailscale coordination server that get set off by notification-worthy tailnet behavior, and they determine whether to actually notify you based on your endpoint subscription preferences. When an event happens on your tailnet that your endpoint is subscribed to, the associated event trigger will create a payload with information about that event — such as the time the event occured or the type of event, and it might even contain a link to the admin console if an action needs to be taken.
After the payload is created, it gets queued up to be sent out. Queued-up events are requested in batches via a separate publisher service, which handles the actual sending of events. Once received by the service, it makes an HTTP POST request containing all the events that have to be sent to your endpoint within the batch it received. If your endpoint returns an error (non-2xx) status code, we’ll retry the request later on. In particular, we retry failing events hourly for up to 24 hours in order to give you time to fix your broken endpoint.
That’s about it! On your side, you choose what to do when you receive a webhook, like post a message to Slack or fire an alert.
To summarize and put everything what we’ve just discussed into perspective, here’s a system diagram depicting the internal process of how a webhook gets sent to a user:
Security considerations
While the process of sending a webhook seems pretty straightforward, there were certainly a bunch of design challenges and security concerns we had to be aware of when actually designing the webhook delivery system. To achieve this, we had to set up a proxy server, sign events with a shared secret key, and validate endpoints; all of which are discussed in more detail below.
The main security concern with webhooks in general is the fact that we, the webhook producer, have to make HTTPS POST requests out in order to deliver events to consumer endpoints. The issue with making these requests is that we have no idea if the consumer is actually a safe destination to connect to. We also have no way to validate that the endpoint wants to be receiving our traffic. Someone might have set up an endpoint that results in events being sent to Tailscale’s internal infrastructure, or to someone else’s infrastructure. Both of the aforementioned scenarios could result in DDoS attacks, which are attacks designed specifically to overwhelm servers to “deny service” to users.
So, in order to prevent the first issue — events being sent to Tailscale’s internal infrastructure, via server-side request forgery (SSRF) — the publisher for webhook events exists on a proxy server outside of the main coordination server. In order to prevent SSRF, we use a customized HTTP client to send webhooks, created by Tailscale’s own Andrew Dunham. It uses a net.Dialer that doesn’t allow connections to destinations that resolve to Tailscale’s internal IPs. This DNS lookup ensures that it is only possible to send webhooks to public IP addresses and eliminates the above security concern.
We also want to prevent attackers from using our webhooks to try a DDoS attack against others. To do this, we stop sending webhooks after several retries if we’re getting non-2xx responses. And, to ensure you’ve configured webhooks correctly, you can send a test event as part of setup.
Furthermore, designing the system such that the webhook dispatch service lives on a separate proxy server helps alleviate some of the load that would otherwise be placed on Tailscale’s coordination server. This server handles the main functionality Tailscale provides, so moving non-essential functionality elsewhere results in better performance.
On top of keeping our internal infrastructure safe, we also wanted to make sure that you can trust that the notifications you’re receiving via webhooks are coming from Tailscale, and haven’t been tampered with or replayed. We do this by generating a shared secret key, signing the webhook, and sending that signature and the current timestamp as part of the event list’s payload header. When you receive the events, you can then check if it’s actually from us.
In addition, we only allow you to configure HTTPS webhook endpoints and not HTTP ones — this ensures that the data sent in the requests are encrypted.
As I was the main designer and developer of the webhooks feature during my internship, I was really able to put my systems design and development experience to the test. I learned so much about how careful you have to be when designing systems to ensure that they are secure and scalable. As the system was being designed and developed, there were several tweaks that were made to the design to allow it to perform better. I am truly grateful to all of the engineers at Tailscale who provided feedback on the system during the design and development processes. Without them, the design of the system wouldn’t be where it is today, and I have definitely become a better engineer through their guidance.
I’m so excited that we are now able to share this feature with you. If you’re interested in receiving notifications for the various events that can happen in your tailnet, webhooks are now available! We’ll continue to add events to webhooks. To request something specific, file a feature request on GitHub.
Laura Florea
