Troubleshooting grants

Grants are Tailscale's unified approach to defining access controls for both network and application layers in your Tailscale network (known as a tailnet). This guide walks you through diagnosing and solving common grant-related problems, progressing from basic verification techniques to resolving complex authorization scenarios.

Diagnosing grant issues

This section covers the tools and methods available for inspecting your grant policies, verifying policy compilation, and examining how the Tailscale policy engine applies permissions to specific devices or users. Using these diagnostic techniques will help you identify the root cause of most grant-related problems.

Keep in mind that Tailscale validates the syntax of the tailnet policy file, including the grants section, when you save it. You won't be able to save it if it has a syntax error. This is true for our admin console editor, GitOps, Terraform, and our API. You also won't be able to save the tailnet policy file if it refers to a target (such as a group) that is not defined.

Verify your grants configuration

The first step in troubleshooting is to verify that your grants are correctly defined in your tailnet policy file. You can use the Preview rules tab from the Access controls page of the admin console to check how the rules apply to specific users or tags. This can help you ensure the policy configuration will work as you intend it to.

Inspect capabilities with `whois`

For diagnosing issues with application layer capabilities granted to specific devices or users, the tailscale whois command is invaluable. This command provides detailed information about a node, including all capabilities granted to it through your tailnet policy.

To inspect the capabilities of a user or device, use its Tailscale IP address:

tailscale whois 100.101.102.103

The output includes:

Basic device or user information.
Network addresses associated with the device.
A list of all capabilities granted to the device.

Pay special attention to the Capabilities section of the output, which details all application capabilities granted to the device. Each capability listing includes its full name and parameters, letting you verify that it is applying the correct permissions.

Resolve common issues

Many grant-related issues stem from basic configuration problems or misunderstandings about how grants work. This section addresses some common issues you might encounter when working with grants.

Selector matching issues

Selectors might fail to match the intended targets. This happens when the selectors in your grants don't correctly identify the devices or users you're trying to permit.

Common selector matching problems include:

Misspelling email addresses or domain names.
Using tag that have been created but not applied to any devices.
Using group names that have been created but not populated with users.
Forgetting to prefix group names with group: or tag names with tag:. This type of syntax error will be caught automatically unless there's also a host with the same name as the group or tag.

To troubleshoot selector matching issues:

Confirm that tags referenced in your grants have been applied to the appropriate devices.
Check for typos in email addresses, especially if you're granting access to specific users.
Use tailscale status to verify the tags applied to your devices.
You must run this command on each device you're checking.

If you're using autogroups (such as autogroup:member), make sure you understand the devices within each autogroup. For example, autogroup:member includes all users who are members of your tailnet, but not devices shared from other tailnets.

Less common and complex issues

Beyond basic configuration problems, you might encounter more complex issues with grants, especially when implementing sophisticated access control policies (such as posture checks, route filtering, or custom application capabilities). These issues often require a deeper understanding of how the Tailscale policy engine processes and applies grants.

Problems with `via` field routing

The via field in grants lets you segment resource access through different routers based on specific criteria. However, misconfiguration can lead to routing failures or unexpected traffic paths.

Common via routing issues include:

Specifying a via target that isn't reachable from the source.
Using a via target with subnet routers that don't have the appropriate subnet routes advertised.
Using a via target with app connectors that have not been appropriately configured.
Using a via target with exit nodes that have not been approved.
Missing via field when intending to route traffic to subnet routes, app connectors, or exit nodes.

To troubleshoot via routing issues:

Verify that the device specified in the via field exists and is online using tailscale status.
Confirm that the via device has the appropriate configuration.
For subnet routers, check that subnet routes are advertised using tailscale status --routes.
For app connectors, check that the app connector is properly configured.
For exit nodes, check that the exit node has been approved in the admin console.
Check that the via device is reachable from the source device using tailscale ping.
Verify that the destination is reachable through the via device by examining logs on the via device.

Example of a correct via configuration:

"grants": [
  {
    "src": ["group:eng"],
    "dst": ["192.0.2.0/24"],
    "ip":  ["*"],
    "via": ["tag:subnet-router"]
  }
]

In this example, make sure that a device with the tag:subnet-router tag exists, is online, and has the 192.0.2.0/24 subnet route advertised.

Device posture verification failures

The srcPosture field lets you restrict access based on device posture conditions.

Common device posture issues include:

Devices running outdated Tailscale versions that don't support posture checking.
Posture conditions that are too restrictive or impossible to meet.
Misconfigured posture conditions with syntax errors.
Devices missing required attributes for posture checking.

To troubleshoot device posture issues:

Verify that all devices are running an up-to-date version of the Tailscale client. Versions prior to 1.52.0 don't support posture checking.
Review the posture conditions in your tailnet policy file for correctness.
Use tailscale status --self to get the device attributes that are checked for posture verification.
Test with more permissive posture conditions first, then gradually tighten restrictions.

Example of device posture verification:

"postures": {
  "posture:latest": [
    "node:tsVersion >= '1.42.0'",
    "node:os == 'linux'"
  ]
},
"grants": [
  {
    "src": ["group:eng"],
    "dst": ["tag:prod"],
    "ip":  ["*"]
    "srcPosture": ["posture:latest"]
  }
]

If devices are failing posture checks, temporarily remove the srcPosture field to verify that the grant works without posture conditions, then add them back one by one to identify the problematic condition.

Application capability integration issues

Integrating application capabilities with grants can introduce issues when applications don't correctly interpret the capabilities provided by the Tailscale client.

Common application capability issues include:

Applications not querying the local Tailscale client for capabilities.
Incorrect capability names or parameter structures.
Applications misinterpreting the capabilities returned by the Tailscale client.

To troubleshoot application capability issues:

Verify that your application is correctly querying the Tailscale client for capabilities using the LocalAPI. This is only required if the application doesn't use the tsnet library.
Check that the capability name matches exactly what your application expects.
Confirm that the parameters in your grant match the schema expected by your application.
Test with simplified capability parameters first, then add complexity.

Example of a correct application capability grant:

"grants": [
  {
    "src": ["group:eng"],
    "dst": ["tag:tailsql"],
    "ip":  ["tcp:443"],
    "app": {
      "tailscale.com/cap/tailsql": [
        {
          "dataSrc": ["prod", "staging"]
        }
      ]
    }
  }
]

In this example, make sure that the application running on devices with the tag:tailsql tag is correctly checking for the tailscale.com/cap/tailsql capability and properly interpreting the dataSrc parameter.

Prevention strategies

This section outlines best practices for designing, implementing, and maintaining grants in your tailnet policy. By following these recommendations, you can minimize issues and streamline troubleshooting when problems do arise.

Design principles for effective grants

Least privilege: Grant only the minimum permissions needed for each source-destination pair. Be specific about ports and protocols in the ip field when appropriate.
Logical grouping: Organize related grants together and use comments to explain their purpose. This makes the policy file more readable and easier to maintain.
Standardized naming: Use consistent naming conventions for groups, tags, and IP sets referenced in your grants. This reduces the risk of typos and makes troubleshooting easier.
Progressive refinement: Start with broad grants and refine them over time as you better understand your access control requirements. This approach is less likely to cause unexpected denials.
Documentation: Document your grants with comments in the policy file, explaining why each grant exists and what it's intended to permit.

Following these principles will result in a more maintainable policy file that's less prone to issues and easier to troubleshoot when problems occur.

Testing strategies

Consider using testing strategies before deploying changes to your grants:

Document test cases: Create a set of tests in your tailnet policy file that cover all the permissions you need to verify.
Use the access control rules preview: The admin console's policy file editor includes a preview feature that shows what changes your edits will make to the effective policy. Review this carefully before applying changes.
Staged rollout: For significant changes, consider a staged rollout to a subset of devices or users before applying the changes to your entire tailnet.

By thoroughly testing changes before deployment and monitoring for issues afterward, you can identify and resolve problems before they affect your users.