Webhooks

Webhooks allow you to subscribe to certain events on your Tailscale network and process the event notifications through an integration or app. For example, you could integrate Tailscale events with a Slack channel. If subscribed to an event such as adding a node, your webhook endpoint can send a message in a Slack channel anytime a node is added to your tailnet.

Webhooks is currently in alpha.
Tailscale events in a Slack channel

The Webhooks feature is available on all plans.

How it works

You provide a webhook endpoint which can receive HTTPS POST requests for subscribed Tailscale events. The body of the request provides information about the event. It is up to you to determine how the webhook should process a notification. Tailscale typically sends an event notification to a webhook within a few seconds of the event’s occurrence.

You create and manage webhook endpoints in the webhooks page of the admin console.

Tailscale provides a digital signature for events that it sends, so you can verify whether an event was signed by a secret that is shared between you and Tailscale.

Prerequisites

  • You need a webhook endpoint which will process the Tailscale event notifications. For example, you might want to integrate with Slack, which provides a way to create incoming webhooks for Slack channels.

  • Your webhook endpoint must be able to process HTTPS POST requests.

  • You need to be an Owner, Admin, IT Admin, or Network Admin of a tailnet in order to create, modify, or delete webhooks.

Setting up a webhook endpoint

Webhooks apply to a tailnet. If one user creates a webhook, other Owner, Admin, IT Admin, or Network Admin users on the tailnet can modify or delete it. If a webhook is created by a user who is later removed or suspended from your tailnet, the webhook will still work.

  1. Open the webhooks page of the admin console.

  2. Click Add endpoint.

  3. In the Add endpoint page:

    1. For Webhook URL, provide the endpoint for your webhook. The endpoint URL must use the HTTPS protocol.

    2. Click the events you want to receive as notifications.

      The 'Add endpoint' page in the admin console
    3. Click Add endpoint.

  4. In the Webhook secret page, click Copy to copy the newly-created webhook secret. After you close the Webhook secret page, you won’t be able to copy the secret again.

    Store the webhook secret securely.
  5. Click Done.

    Your webhook endpoint is now configured.

  6. To ensure your webhook is configured correctly and can receive events from Tailscale, test your webhook.

Events

You can subscribe to the following events:

Category Event Description
Device misconfiguration EXIT_NODE_IP_FORWARDING_NOT_ENABLED Exit node has IP forwarding disabled.
Device misconfiguration SUBNET_IP_FORWARDING_NOT_ENABLED Subnet has IP forwarding disabled.
Tailnet management POLICY_UPDATE Tailnet policy file updated.
Tailnet management NODE_CREATED Node was created.
Tailnet management NODE_AUTHORIZED Node was authorized.
Tailnet management NODE_NEEDS_AUTHORIZATION Node needs authorization. Note that any pre-authorized nodes will not generate the NODE_NEEDS_AUTHORIZATION event.
Webhook testing TEST Webhook test event was generated.

Events payload

Webhook events are sent as JSON objects with the following fields:

  • timestamp: Time the event occurred, formatted as a RFC 3339 string.
  • version: Version of the event payload (currently always 1).
  • type: Type of the event (as listed in the table above).
  • tailnet: Name of the tailnet where the event occurred.
  • message: Human-readable summary of the event.
  • data: Optional per-event payload with additional data.

Multiple events may be sent in one payload to minimize overhead. The root payload object is always an array of events.

The following shows an example events payload sent to a webhook endpoint:

[
  {
    "timestamp": "2022-09-21T13:37:51.658918-04:00",
    "version": 1,
    "type": "TEST",
    "tailnet": "example.com",
    "message": "This is a test event",
    "data": null
  },
  {
    "timestamp": "2022-09-21T13:59:02.949217-04:00",
    "version": 1,
    "type": "NODE_CREATED",
    "tailnet": "example.com",
    "message": "Node alice-workstation1.example.com created",
    "data": {
      "deviceName": "alice-workstation1.example.com",
      "deviceOwner": "alice@example.com",
      "url": "https://login.tailscale.com/admin/machines/100.12.345.67"
    }
  },
  {
    "timestamp": "2022-09-21T13:59:02.949278-04:00",
    "version": 1,
    "type": "NODE_NEEDS_AUTHORIZATION",
    "tailnet": "example.com",
    "message": "Node alice-workstation1.example.com needs authorization",
    "data": {
      "deviceName": "alice-workstation1.example.com",
      "deviceOwner": "alice@example.com",
      "url": "https://login.tailscale.com/admin/machines/100.12.345.67"
    }
  },
  {
    "timestamp": "2022-09-21T13:59:15.966728-04:00",
    "version": 1,
    "type": "NODE_AUTHORIZED",
    "tailnet": "example.com",
    "message": "Node alice-workstation1.example.com authorized",
    "data": {
      "deviceName": "alice-workstation1.example.com",
      "deviceOwner": "alice@example.com",
      "url": "https://login.tailscale.com/admin/machines/100.12.345.67"
    }
  },
  {
    "timestamp": "2022-09-27T09:51:46.512946-07:00",
    "version": 1,
    "type": "POLICY_UPDATE",
    "tailnet": "example.com",
    "message": "Tailnet policy file updated",
    "data": {
      "newPolicy": "{\n\t\"acls\": [\n\t\t{\"action\": \"accept\", \"src\": [\"autogroup:members\"], \"dst\": [\"*:*\"]},\n\t],\n}",
      "oldPolicy": "{\n\t\"acls\": [\n\t\t{\"action\": \"accept\", \"src\": [\"*\"], \"dst\": [\"*:*\"]},\n\t],\n}",
      "url": "https://login.tailscale.com/admin/acls"
    }
  }
]

Testing your webhook

To ensure an endpoint is properly configured and able to receive events from Tailscale, you can send a test event:

  1. Open the webhooks page of the admin console.

  2. Find the endpoint that you want to test, click on the ellipsis icon icon to the right of the page, and select Test endpoint.

  3. Click Send test event. If your webhook is configured correctly, within a few seconds your webhook endpoint should receive an event with an ID of TEST.

Retries for events that fail to send

Tailscale typically sends an event notification to a webhook within a few seconds of the event’s occurrence. If an event notification fails to successfully send (such as when Tailscale receives a 3xx, 4xx, or 5xx error, or no response at all from your webhook endpoint), Tailscale will retry sending the event. For an event that fails to send, there is a maximum of three retries, which we’ll try to send within a few hours of the initial event.

Updating subscribed events

  1. Open the webhooks page of the admin console.

  2. Find the endpoint that you want to update, click on the ellipsis icon icon to the right of the page, and select Edit.

  3. Select the events you want to receive as notifications, and unselect those you don’t want to receive.

  4. Click Edit endpoint.

Webhook secret

The webhook secret is a signing secret shared between Tailscale and the creator of the webhook endpoint, and is unique per endpoint. If this shared secret is compromised or leaked, whomever knows the secret can send fake events. If you suspect your secret is compromised, create a new secret.

The webhook secret has no expiry.

Rotating a webhook secret

When you rotate a webhook secret, the old secret will no longer work.

  1. Open the webhooks page of the admin console.

  2. Find the endpoint that you want to have a new secret, click on the ellipsis icon icon to the right of the page, and select Rotate webhook secret.

  3. Click Rotate to confirm you want a new secret.

  4. In the Rotate webhook secret page, click Copy to copy the new secret. After you close the Rotate webhook secret page, you won’t be able to copy the secret again.

    Store the webhook secret securely.
  5. Click Done.

The new secret will be used for any new events that are sent—you need to update your signature verification routine to use the new secret.

Deleting an endpoint

  1. Open the webhooks page of the admin console.

  2. Find the endpoint that you want to delete, click on the ellipsis icon icon to the right of the page, and select Delete.

  3. Click Delete endpoint to confirm you want to delete the endpoint.

Using a new webhook for an existing endpoint

To add a new webhook (subscribed event) for an existing endpoint, edit the endpoint instead of setting up a new endpoint.

Audit logging of webhook actions

In configuration audit logging, an action will be recorded in your audit log whenever a webhook is created, deleted, or updated. The log entry will show who performed the action, and when the action occurred.

Verifying an event signature

You can verify whether an event was signed by the webhook secret that was shared between you and Tailscale. Note this doesn’t necessarily mean that an event was sent from Tailscale. Rather, it means an event was sent from an entity that has knowledge of the secret shared between you and Tailscale.

A event sent from Tailscale contains a Tailscale-Webhook-Signature header. The Tailscale-Webhook-Signature header includes a timestamp and a signature:

Tailscale-Webhook-Signature:t=1663781880,v1=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

The timestamp, prefixed by t=, is the epoch time in seconds when the event occurred. The signature, prefixed by v1=, is a hash-based message authentication code (HMAC) using SHA-256. The only supported scheme for the signature is v1.

Most modern programming languages provide libraries for computing and comparing HMACs. The following flow describes how to verify a signature.

  1. Parse the event timestamp and signature from the Tailscale-Webhook-Signature header.

    Using the , character as the separator, split the Tailscale-Webhook-Signature data into a list of elements. Then, using the = character, split each element to get two key/value pairs. The first key is t, with the event timestamp as its value. The second key is v1, with the event signature as its value.

  2. Compare the event timestamp (the value for t) with the current time. For example, if your verification process acts on events as soon as they are received, and if the event time is more than 5 minutes prior to the current time, you might consider the event as a replay attack.

  3. Create a string, string_to_sign, to sign by concatenating:

    • The timestamp (the value of t) represented as a string.
    • The . character.
    • The decoded event request body. Note this is in the request itself, not in the Tailscale-Webhook-Signature header. The request body contains the encoded events payload, you need to decode the request body for signing purposes.
  4. Compute the signature of string_to_sign.

    Create an HMAC with the SHA256 hash function. Use your webhook secret for the signing key, and use string_to_sign as the message to sign.

  5. Use an HMAC compare function to compare the signature in the Tailscale-Webhook-Signature header (the value of v1) with the signature you created in the previous step. If they are not identical, that indicates the event’s payload was not signed by your webhook’s secret, and the event should not be considered an event sent from Tailscale.

Sample verification code

The following is an example Go program that shows a technique for verifying the signature.

type event struct {
   Timestamp string            `json:"timestamp"`
   Version   int               `json:"version"`
   Type      string            `json:"type"`
   Tailnet   string            `json:"tailnet"`
   Message   string            `json:"message"`
   Data      map[string]string `json:"data"`
}

// verifyWebhookSignature checks the request's "Tailscale-Webhook-Signature"
// header to verify that the events were signed by your webhook secret.
// If the request is verified, the list of contained events is returned.
func verifyWebhookSignature(req *http.Request, secret string) (verified bool, events []event) {
   defer req.Body.Close()

   // Grab the signature sent on the request header.
   signature := req.Header.Get("Tailscale-Webhook-Signature")
   if signature == "" {
      return false, nil // missing "Tailscale-Webhook-Signature"
   }
   parts := strings.Split(signature, ",")
   if len(parts) != 2 {
      return false, nil // wrong format; should be t=<timestamp>,v1=<signature>
   }
   if !strings.HasPrefix(parts[0], "t=") {
      return false, nil // wrong format
   }
   if !strings.HasPrefix(parts[1], "v1=") {
      return false, nil // wrong format
   }
   timestamp := strings.TrimPrefix(parts[0], "t=")
   reqSignature := strings.TrimPrefix(parts[1], "v1=")

   // Verify that the timestamp is recent.
   // Here, we use a threshold of 5 minutes.
   unixSeconds, err := strconv.ParseInt(timestamp, 10, 64)
   if err != nil {
      log.Fatal(err)
   }
   if time.Unix(unixSeconds, 0).Before(time.Now().Add(-time.Minute * 5)) {
      return false, nil // timestamp older than 5 minutes
   }

   // Form the expected signature.
   if err := json.NewDecoder(req.Body).Decode(&events); err != nil {
      log.Fatal(err)
   }
   raw, err := json.Marshal(events)
   if err != nil {
      log.Fatal(err)
   }
   mac := hmac.New(sha256.New, []byte(secret))
   mac.Write([]byte(timestamp))
   mac.Write([]byte("."))
   mac.Write(raw)
   want := hex.EncodeToString(mac.Sum(nil))

   // Verify that the signatures match.
   if reqSignature != want {
      log.Printf("signature verification failed; want = %q, got = %q", want, reqSignature)
      return false, nil
   }

   // Signature was successfully verified.
   return true, events
}

Last updated

WireGuard is a registered
trademark of Jason A. Donenfeld.

© 2022 Tailscale Inc.

Privacy & Terms